A Tale of Two Pipelines: To DevOps or Not To DevOps

A Tale of Two Pipelines: To DevOps or Not To DevOps

Brett HoferGlobal DevOpsPractice Lead

25+ years of IT experience

Dynatrace• Recognized thought leader DevOps & Big Data• Featured speaker, presenter, writer - DevOps.com,

DevOpsSummit Journal, InfoWorld and many more• Author eBook – “The Art of DevOps”

Prior to Dynatrace• Senior app manager for 80M+ budget, staff of 70+

including onshore/offshore• Lead designer and developer of multi-platform

apps, multi-channel intranet/internet web • Senior Solutions Architect on 60M+ initiative(s)

Welcome to“A Tale of Two

(very different) Pipelines”

It’s about people,

processes and technologies!

Are you sinking, surfing, or just

surviving?

Our idea here will be to help identify bad situations by

relating to all or part of the two stories …

We’ll aim to ignite ideas of positive

change by sharing experiences through

these stories.

A story about one fictional organization struggling

with all too familiar pipeline

challenges and how it took a corporate catastrophe to

finally decide on transitioning to solid DevOps practices.

Which pipeline will look like yours?Maybe somewhere in the middle?

What can you do to influence change?

What exactly are we moving through the pipeline?

Servicing requests from business ideas, non functional needs, fixes, functionality adjustments



Not every change is

created equal!

Success equals the RIGHT change at the RIGHT time

• Precision in priority ensures all efforts are focused on what team has agreed is success!

• Getting people to agree can be tough.

• Key: invest in “best-of-breed” technology to track, collaborate, and continually monitor FACTS versus OPINIONS when prioritizing

Typical top priority factors

• Value• Cost• Risk• Impact• Time

Add continuous monitoring to deliver continuous improvements

• Find opportunities and improve value

• Identify / manage cost factors

• Reduce and mitigate risk

• Identify and manage impacts

• Find problems fast and reduce time, costs and risk

All the right people know the health status of a change and build as it moves through the pipe

• Test and monitor early and often• Find and fix problems fast and early

To DevOps or not to DevOps

Fingerscrossed…

Welcome to easyTravel

Debute latest online booking system

• Development over a year• Lots of internal jokes

“neverTravel”• Ops “hope they load tested”• Late and over budget• App manager changed twice• Tension between business

and dev stalled budget twice

Mark SmoltenChief Marketing Officer

• 2 years at easyTravel• Other successful roles driving biz up

30%• Challenged by

• channels against fierce competition• convincing exec management to

invest in new, unproven outlets• not enough info on current

platform use and state• not being tied into new system in

dev

• Mark contacts TravelVision and lines up a spot on their wildly popular live TV Show “Book Me Now!”

• Mark offers viewers an amazing deal on a live TV promo code in exchange for debuting the new easyTravel site

• Live show November 7th, 8:00 p.m.

• Yay, Mark!

TravelVision’s BookMeNow!

Tina AntonVP, eCommerce

• 8 years at easyTravel, 15 years industry• Promoted to head of eCommerce

two years ago• More experienced in the travel

business than eCommerce• Oversees the overall vision of every

easyTravel eBusiness function• Maintains online budgeting decisions• Very friendly, but long history of

battling it out with IT

Mark: Enthusiastic | Tina: Concerned

Mark speaks with Tinaabout his Live Promo Deal with TravelVision.

He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.




Tina mentions that they don’t have such a feature on the site and wishes he had consulted with her first. She asks him “when is this?”





“We air on live TV to show off the new site and the promo on November 7th at 8pm! It’s the only time until next fall that they will have this event.”





“We air on live TV to show off the new site and the promo on November 7th at 8pm! It’s the only time until next fall that they will have this event.”

Tina, very apprehensive, explains “but that’s less than two months away! We don’t even have a promo feature on the site since all our travel packages are already discounted.”


Mark states “This is the business nowadays. We have to adapt quickly to capture market opportunities. The discounts will come from the marketing budget. We need to make this happen!”



Tina, continues to voice her concerns trying to explain that there is a big change process for something like this.



Tina, continues to voice her concerns trying to explain that there is a big change process for something like this.

“Mark, things are also more complicated these days due to all the channels we support. This would need to be available on Web, Web Mobile and in our native Mobile apps.”

Rick TrentonSenior Project Manager

• Consultant with the company one year• Ex-military very structured and organized• Assigned to the project management of

easyTravel Web and mobile • Brokers budgets and costs between

business and IT for major deliverables • Challenged with producing weekly

project statuses to management in PowerPoint

• Regularly maintains high level tasks in his Microsoft Project Plan

Tim FolkDirector of Digital Services

• 5 years at easyTravel• Former Senior Solutions Architect • Inherited responsibility 4 months ago • Not a fan of his Management Admin

Tasks• Manages all the Development of Web

& mobile channels for easyTravel• Challenged with lack of time, too many

meetings, putting out fires• He manages a local smaller IT staff but

required to use at least 70% offshore staffing for his dev to keep costs down


Tina now contacts both Rick and Tim to explain this new Feature that must be ready and tested in all channels by November 7th. Less than two months away.



Rick’s PM contract is funded by the business so he is very appeasing to Tina while Tim is drowning in numerous problems. Tim explains “I’m worried about the viability of the entire new site by November never mind this new feature.”



Rick’s PM contract is funded by the business so he is very appeasing to Tina while Tim is drowning in numerous problems. Tim explains “I’m worried about the viability of the entire new site by November never mind this new feature.”

“Well you guys know the marching orders. I have confidence you will get this done. Keep me posted on the progress and take care.”

Meet Kelly RochellSenior Business Analyst easyTravel

- Works for Tina

Meet Scott DuttonDigital Services Dev LeadeasyTravel

- Works for Tim

Meet Frank BozmanDigital Services Operations LeadeasyTravel

- Works for Tim

Tim contacts three key members

critical to getting this

done.


- Works for Tina



Kelly explains her understanding of the feature

and it’s urgency.


- Works for Tina



“Ok Kelly, we are going to need to get this fully documented and exactly how it should work.”


- Works for Tina


- Works for Tim


- Works for Tim

“Hey Tim, I’m going to need to put this on a completely

separate code branch and we

will need another environment. It’s going to impact too many things to put in the Oct

release.”

To understand Tim’s concerns,let’s look at the app pipeline he owns …

Fingerscrossed…

Tale of Two Pipelinesby Brett Hofer

Senior Business Analyst easyTravel

- Works for Tina


- Works for Tim


- Works for Tim


separate branch and we will need

another environment. It’s going to impact too many things

Scott’s Traditional Reasoning on incorporating this change PROMOFEATURECHANGE

Source Control

easyTravel Master CODE Branch v2.0

easyTravel CODE Branch v2.2 (Oct)

easyTravel CODE Branch v2.5 (Jan)

easyTravel CODE Branch v3.0 (Mar)

Managed VM Infrastructure

DEV1vm

DEV2vm

DEV3vm

DEV4 (ER)vm

QA1vm

QA2vm

QA3vm

QA4 (ER)vm

UATvm

PERFvm

PROD Avms

PROD Bvms

mergeHosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix BuildseasyTravel CODE Branch v2.3 (Nov)

promotions

PROMO Only Dev related Code


FEATURECHANGE

PROMOMeet Kelly RochellSenior Business Analyst easyTravel

- Works for Tina


- Works for Tim


- Works for Tim

“Haha, we have about as much

chance of getting new machines provisioned in

that timeframe as I have in getting a decent build from Scott.”


FEATURECHANGE


- Works for Tina


- Works for Tim


- Works for Tim

“Hey Frank, it might be easier if the machines

you get me weren’t as slow as you. I’m

tempted to have our guys bring in

their home machines and

deploy to those.”


FEATURECHANGE


- Works for Tina


- Works for Tim


- Works for Tim

“Honestly, if I put in a ticket to provision new VMs with the infrastructure department it

will have mold on it by November.”


Senior Business Analyst easyTravel

- Works for Tina


- Works for Tim


- Works for Tim




Frank’s Traditional Ops Reasoning on procuring new VMsPROMOFEATURECHANGE

Managed VM Infrastructure

DEV1vm

DEV2vm

DEV3vm

DEV4 (ER)vm

QA1vm

QA2vm

QA3vm

QA4 (ER)vm

UATvm

PERFvm

PROD Avms

PROD Bvms

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

promotions

PROMO

Enter a Dev 5 Provisioning Ticket

Enter a QA 5 Provisioning Ticket

Outsourced Company Managing Environments locates frame & Builds Hostaccording to security, OS, CPU, Memory.

Outsourced Company Managing Environments locates frame & Builds Hostaccording to security, OS, CPU, Memory.

Franks staff will manually setupthe DEV environment for deploying Nov

Franks staff will manually setupthe QA environment for deploying Nov

DEV5vm

Hosts Fix Builds

QA5vm

Hosts 2.3 PromoLead Time to identify, provision and get cost contract – 30 to 60 days


FEATURECHANGE


- Works for Tina


- Works for Tim


- Works for Tim

“Guys, maybe we should just

concentrate on finding a

solution that will get us

there. This needs to get done.

Kelly let us know when you have our requirements.”


FEATURECHANGE


- Works for Tina


- Works for Tim


- Works for Tim

“Wow, it sounds like you guys have your work cut out and on

that note, thanks Tim, we will have your requirements in a few days”

Now let’s evaluate

this feature as it moves

through their

pipeline.

The good news – the pipeline is somewhat agile

Plan

Design

DevelopTest

Operate

The bad news – all the cycles are long!3 month cycles now require an interim

releasePlan

Design

DevelopTest

Operate

ScoreC-


Dev

Communication spreadsheets/meetings/phone/emailsBi

zIT

Mar

ket

Plan DesignFEATURECHANGE

PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent

IdentifiesBiz Details

High LevelEstimate

InitiatesTracking

InitiatesTracking


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

CMO Recognizes Major Opportunity Initiates Need

FEATURECHANGE

• Industry is fast past and always fluid.


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

High Level Business Functionality

• All Channel must change• Actually 3 Pipelines affected

• Mechanics of how it will work


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

Separately Feature tracking ++Risk, +Time, +Cost • Dev uses tracking software, but no

other groups do

• Frank adds the task to his trackingspreadsheet


Dev


zIT

QA

Plan Design FEATURECHANGE

PROMO

PMO

Desig

nIdentify FeatureImpacts

AppArchitect

CapacityDesign

Plan

StoryBoard

Identify FeatureImpacts

DesignTesting

Strategy

GatherTime &

Cost

Track on Overall

Plan


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


DesignTesting

Strategy

GatherTime &

Cost

Track on Overall

Plan

Meet Raheeb Patel Senior Quality Assurance LeadeasyTravel Contractor

- His company contracted by QA department- Follows his QA dept and corporate guidelines- Needs very specific detailed technical specs

to put his team in motion- Runs a large team of offshore manual testers


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


DesignTesting

Strategy

GatherTime &

Cost

Track on Overall

Plan

Design Feature Impact Assessment - +++RISK, +++TIME, +++COST• Done conversationally among

a few technical resources• Frequent inaccuracy and guessing.

Has major cascading effects on capacity, architecture, scheduling


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


DesignTesting

Strategy

GatherTime &

Cost

Track on Overall

Plan

Architecture of Feature - +++RISK, +++TIME, +++COST• Done conversationally among

architects • Many know the general architecture

but aren’t up to speed on all of the detailed changes over each release.


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

StoryBoard


DesignTesting

Strategy

GatherTime &

Cost

Track on Overall

Plan

Capacity Planning - +++RISK, +++TIME, +++COST• Lacks comprehensive understanding

of site traffic and visits• Lacks comprehensive understanding

of scalability concerns

CapacityDesign

Plan


Dev

Communication spreadsheets/meetings/phone/emails

Dev

Dev

Ops

FEATURECHANGE

Code Check-In RunBuild

Manual Dev Env Config

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data

DocumentTest Scripts

Manual Stage Env

Config

Manual Prod Env

Config

PROMO

Test


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Using Offshore Development - +RISK, ++TIME, +COST • Timezone diff often = long response

times for fix & feedback cycles• Lack of Backend System Support


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Developers on varying Platforms - +RISK, +TIME • Not developing on the same

technical stack as intended deploytargets - introduces unnecessaryrisks


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Overuse of Merging - +RISK, +TIME, +COST • Increased manual effort on changes


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Too many premature Check-Ins - ++RISK, ++TIME, ++COST

• No static policy checking on dev code • No runtime policy checking on dev code• No code review on critical functions • No architectural validation


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Manual Run Builds - +RISK, +TIME

• Often requires contacting one person• Limited Visibility and Feedback of

build results


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

Slow Build Time - +TIME • Build is taking over 50 minutes


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Diagnose

Dev4

9% Low and Limited Unit Coverage - ++RISK, -VALUE • Limited Visibility and Feedback

• Places high/all reliance on functional testing – Which is manual!!!


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

PROMO

Dev4

Limited Result Visibility - +RISK, +TIME • Only one or two people

• Manual skimming of build logs • Often too much promoted “thrown

over the wall”

Diagnose


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

Manual Changes to Environments - +RISK, -VALUE• Prone to cross environment errors• Manual adjustments forgotten and

often not repeated as necessary• Static environments get dirty• Kills innovation speed

PROMO

CheckResults

Dev4

Diagnose


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

Test

Manual Deployments in Development - +TIME, -VALUE• More manual delays in promotion

PROMO

Dev4

CheckResults

Diagnose


Dev


Dev

Dev

Ops

FEATURECHANGE



ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

Test

ers

ManuallyPrepare

Test Data


Manual Stage Env

Config

Manual Prod Env

Config

PROMO

Test

Dev4Dev4

Diagnose

Scott’s Dev Visibility & Status Sharing

• Few actually know the build health in Dev• Few actually know common Dev problems • Too little screened • Lacks macro level metrics


Dev

spreadsheets/meetings/phone/emails

Dev

Dev

Ops

Test Prod

Test

ers

DetermineTest Suite

Manual DeployQA 5

Prod

FEATURECHANGE

PROMO

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

Validate Build

DiagnoseRoot

Cause

Check Infrastruct

ure

Manual Deploy

UAT


Dev


Dev

Dev

Ops

Test Prod

Test

ers

DetermineTest Suite

Manual DeployQA 4

Prod

FEATURECHANGE

PROMO

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

Validate Build

DiagnoseRoot

Cause

Check Infrastruct

ure

Manual Deploy

UAT

Manual Test Selection - ++TIME, --VALUE, ++COST

• Selecting a lot more manual testing than what the change requires stifles new changes and costs time

• Manual Testing Greatly slows feedback time.


Dev


Dev

Dev

Ops

Test Prod

Test

ers

DetermineTest Suite

Manual DeployQA 4

Prod

FEATURECHANGE

PROMO

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

DiagnoseRoot

Cause

Check Infrastruct

ure

Manual Deploy

UAT

MTTR SLOW - ++TIME, --VALUE, ++COST • Test usually reported as

just failed with confusing description

• Finding exact issue is hard• Wasted Time - Finger

Pointing• Unnecessary people

involved in failure

Validate Build

Almost 2 months later

. . .

Now it’s prime time

November 7 at 8:10 p.m.

Tale of Two Pipelinesby Brett Hofer Before DevOps

Hi everyone, I’m Lisa Goodheart, your host of

Book Me Now!


Today is our Big Deal day where once a year we

showcase a select travel service live allowing them to put their best foot forward

demonstrating great deals to our viewers.


Here with me is Eric Wallace the VP of Sales

from easyTravel. He has a special treat for you

today.


Thanks Lisa, I’m honored to be here and to show your audience what we

believe is the most exciting travel site on the

market.


Our business negotiates some of the most

incredible packages for the cheapest prices and

then bundles them.


Today we will demonstrate an absolute exclusive for this

show. If your audience jumps on our site either web or on their phone and uses this one time PROMO code.

Your Book Me Bo


Your Book Me Now Exclusive Live Promo

It will randomly discount our packages between 5% to 95% upon searching and

if you begin to book it upon that search you can

book that deal.


Your Book Me Now Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Bo

Thank you Eric. This is really exciting. Our

audience will now notice the Promo Code scrolling

on the bottom of their screen.


Let’s demonstrate

Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi


We put in the Promo


ROLLTHEDICE


Where we want to take our Journey


ROLLTHEDICE

Belize


When we want to go…


ROLLTHEDICE

Belize

12/18/2016

1/5/2017


Now click search…


ROLLTHEDICE

Belize

12/18/2016

1/5/2017

20 seconds later . . .


Um this is unusually long…


ROLLTHEDICE

Belize

12/18/2016

1/5/2017



ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Wow, your site must be working pretty

hard on those deals.



ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Yes, it’s chugging away to find your

viewers the deals… is it

possible it’s your internet here?



ROLLTHEDICE



ROLLTHEDICE

And… that doesn’t look good. Maybe a

commercial?



ROLLTHEDICE

Well folks we are sorry but it looks like easyTravel is running

into a technical difficulty. We will be

back after these messages…


ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Off Air


ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Well that was a complete waste of

time and money. You people need to get your act together.

Off Air


ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Off AirYou’re not kidding, heads will roll for this one. That

was embarrassing and expensive.


ROLLTHEDICE

Belize

12/18/2016

1/5/2017

And that had nothing to do with our

internet.

Off Air

Tina’s house . . .

Are you kidding me!

And the War Room begins . . .

• Not only was all the money spent on the feature a loss but it turned out the 60% booking growth was a long term growth strategy for Mark the CMO.

• The actual event traffic was 1600% of peak.

• An estimated $4M in new net bookings was lost.

• Losses due to brand damage unquantifiable.

Now let’s walk through this same scenario in a DevOps culture . . .

Let’s see the story when the promo

feature is sent through their new DevOps

pipeline.

The good news – the pipeline IS Agile

2 Week Sprints Plan

Design

DevelopTest

Operate

The new Digital Services EnvironmentPipeline Transparency, Proximity & Availability Matters

Development Test Production


- Works for Tina


- Works for Tim


- Works for Tim




The DevOps Reasoning on incorporating this change (Use a feature toggle)

Source Control

easyTravel Release

Ops Infrastructure As Code

Performance and Dashboards

Infrastructure Cloud ServicesAvailable in Dev/Test &

SECURED Prod

PROMO

Dev Source Code

Test Scripts

CI_CD Configs

CLO

UD

API

LOAD BALANCE POLICY

easyTravelInstance

Tech StackImage for 2.0



PROMOFEATURECHANGE

Cloud Image Library


- Works for Tina


- Works for Tim


- Works for Tim




The DevOps Reasoning on incorporating this change (Use a feature toggle)

Source Control

easyTravel Release

Ops Infrastructure As Code

Performance and Dashboards

Infrastructure Cloud ServicesAvailable in Dev/Test &

SECURED Prod

PROMO

Dev Source Code

Test Scripts

CI_CD Configs

CLO

UD

API

LOAD BALANCE POLICY

easyTravelInstance




PROMOFEATURECHANGE

Cloud Image Library

Aligning All Release Source Code, Automation Scripts, Database Scripting,

Performance and Operational Assets in Source Control Is critical.

It is the PIPE!

Build Once, Deploy and Continuously Monitor in any environment


- Works for Tina“Hey Tim, I’m going to need to put this on a completely



The DevOps General FlowPROMOPROMOFEATURECHANGE

Test EnvironmentDev Environment

DeveloperLocal

Environments

Source Control

Production Environment

Dynatrace in Dev Dynatrace in Stage Dynatrace Prod

Load, Int, Func Tests

Report Results

Env Automation

Deploy Automation

Build ServerPerform Build

Run Unit Tests

Report Results

Environment Setup

Deploy Automation


Dev

FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi

zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

CMO Recognizes Major Opportunity Initiates Need

FEATURECHANGE

• Industry is fast past and always fluid.


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

High Level Business Functionality

• All Channel must change• Actually 3 Pipelines affected

• Mechanics of how it will work


Dev


zIT

Mar

ket


PROMO

PMO

Desig

n

Creates Need

ValidatesSets

Urgent


High LevelEstimate

InitiatesTracking

InitiatesTracking

Aligned all Release Tracks and Sprint Tasks • BA

• Dev• Ops• Test• Data• Security

• Automation• XA


Dev

Biz

ITQ

A


PROMO

PMO


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomArchitect

Design

FEEDBACK LOOP dashboards/emails/tracking technology/group chat

Desig

n


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomArchitect

Design

Design Feature Impact Assessment

• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on Impact decisions.


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomArchitect

Design

Design - Feature Impact Assessment

• Leveraging Dynatrace Actions provides high accuracy on page Impact decisions.


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomArchitect

Design

Architecture of Feature

• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on architectural decisions.


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomArchitect

Design

Architecture of Feature

• Leveraging Dynatrace end-to-end transactional visibility creates high accuracy on architectural decisions.



Dev

Biz

ITQ

A


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomationArchitect

Design

eXperience Assurance Decisions

• Leveraging all of the Dynatrace User Experience Management Data


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomationArchitect

Design

Automation Architecture - ++Time, ++Cost, ++Risk • Role in designing and managing all

automation to move the feature through the pipeline.

• Leveraging Technologies to Automate Processes & Orchestration in CI & CD


Dev


zIT

QA


PROMO

PMO

Desig


AppArchitect

CapacityDesign

Plan

StoryBoard


GatherTime &

Cost

Track on Overall

Plan

TestDesign

Plan

XADesign

PerfArchitectReview


AutomationArchitect

Design

Performance Architecture • Performance Architect Reviews

Performance & Scalability Concerns of the Feature

• Designs Dashboarding Strategy forMonitoring Feature & Health



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In

Setup Release Blueprint

CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose

Dev4Setup Developer Blueprint

ManualMerge

Code ConfigMgmnt

Perf

Eng Create

Perf Dashboard

InstrumentApp

Unit TestCI Server

CheckResults



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Perf

Eng Create

Perf Dashboard

InstrumentApp

Unit TestCI Server

CheckResultsBIG GOALS

• How fast can we determine a check-in will fail and get feedback to the right resource• How far can we progress a healthy build/app without manually touching it• How fast can we identify the problem and turnaround the fix



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Using Onsite & Localized Development - -RISK, --TIME, -COST • Collaborate Often

• No Black Box Development • Culture

CheckResults

Perf

Eng



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI ServerDevelopers use Cloud Instances - -RISK, -TIME

• Developer Instances match targetdeployment stack.

• Easy to onboard other developers• Allows switching technology stacks

CheckResults

Perf

Eng



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

CheckResults

Architectural Validation - --RISK, --TIME, +Value

• Developers, Leads and Solution Architects use Dynatrace to check transaction validity prior to check-in. Great on critical transactions and faster than code reviews.

Perf

Eng



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

CreatePerf

Dashboard

CreatePerf

Dashboard

CheckResults

Unit TestCI Server

Using a feature toggle - -TIME, -COST • Increased manual effort on changes

Perf

Eng



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

CreatePerf

Dashboard

CreatePerf

Dashboard

Unit Test

CI Server

CheckResults

Perf

Eng

Code Coverage Static Code Checking - -RISK, -TIME, -COST • Manage Testing Code Coverage

• Govern Coding Practices • Code Coverage is on Main

Dashboards

JaCoCo

Test


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit Test

CI Server

Source Control Respository - --RISK, -TIME, -COST • Aligning ALL Release assets

is critical

CheckResults

Test


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Build Automation - -RISK, -TIME

• Comprehensive Build Scripts• Include Dependency Resources • Leverage Profiles (Build script

is host aware)• Includes Monitoring of Tests

CheckResults

Ant

Test


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Automated Builds- -RISK, -TIME

• Scheduled off a check-in trigger• Orchestrate tests and deployments• Steps Exec Build Automation Goal

CheckResults

Test


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Fast Build Time - -TIME • Build takes only 15 minutes

CheckResults

Test


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CI-CDProv Env

CheckResults

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Unit & Unit Integration Testing - -RISK, -TIME

• 78% Coverage • All Unit Tests are monitored by

Dynatrace with Results maintainedby Build

CheckResults

Test



Dev

Dev

Dev

Ops

FEATURECHANGE

Code Check-In CI Server


CI-CDProv Env

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit Test

CheckResults

• dynaTrace Monitored Unit Results - --RISK, --TIME

•

• Regression Violations generate change actions in tracking software


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In

ReleaseCloud

Blueprint

CI-CDProv Env

Unit Test

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose

Dev4DeveloperCloud

Blueprint

ManualMerge

Code ConfigMgmnt

Perf

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Configuration Management --RISK, --TIME

• All ops/infrastructure configs are represented in code and

aligned with the release• EnvForRelease = Cloud/Container Blueprint + ConfigCode


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CI-CDProv Env

Unit Test

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Test

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Automated Cloud/Container Provisioning - --RISK, --TIME

• Establish Instance to Deploy

CI Server

CheckResults


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CI-CDProv Env

Unit Test

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Perf

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Automated Test Development- --RISK, --TIME

• All testers are test automation engineers.• All tests are integrated with Dynatrace

APM

Silk Performer

Selenium


Dev


Dev

Dev

Ops

FEATURECHANGE

Code Check-In


CI-CDProv Env

Unit Test

Test Prod

Test

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

PROMO

Test

Diagnose


ManualMerge

Code ConfigMgmnt

Perf

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Performance Dashboards are Deliverables --RISK, ++VALUE

• Major Features need Visibility• The dashboard indicates if the PROMO feature

is being used, response times, inter-dependency health or any errors related. Watched throughout lifecycle.



Dev

Dev

Dev

Ops

Test Prod

Test

Eng

Auto RunFull

Test Suite

EnvReady

Prod

FEATURECHANGE

PROMO

Manual Tests

ReportProblems

Validate Build

Validate Results

DiagnoseRoot

Cause

Check Infrastruct

ure

SelectiveLoadTest



Dev

Dev

Dev

Ops

Test Prod

Test

Eng

Auto RunFull

Test Suite

EnvReady

Prod

FEATURECHANGE

PROMO

Manual Tests

ReportProblems

Validate Build

Validate Results

DiagnoseRoot

Cause

Check Infrastruct

ure

SelectiveLoadTest

Finding Root Cause –TIME, ++VALUE

The Team under DevOps was able

to do four iterations and a full load test

three weeks ahead of schedule.

The feature has been in.


Now click search…


ROLLTHEDICE

Belize

12/18/2016

1/5/2017



ROLLTHEDICE

11/29/2016

12/5/2016

And there you go an 80% discount

on a trip to Belize…



ROLLTHEDICE

11/29/2016

12/5/2017

Wow, that is fast and incredible. I’m sure our viewers want to thank easyTravel. I’m being

told this is our highest viewed show to date!

The DevOps Xcelerator

Confidential, Dynatrace LLC

Master Performance

Architect

Confidential, Dynatrace LLC

Assess

• Identify Target Application• Interview Key Teams with Set Topics

BluePrint

• Record Gaps & Recommendations• Written Roadmap to Drive Implementation Plan

Implement

• Integrate into Automation Process• Build Dashboards• Educate on Process

12

3

30 D

ays

60 D

ays

What We Do

Confidential, Dynatrace, LLC

Questions or to Inquire about the DevOps Engagement email: [email protected]

Please put TALES WEBINAR in the Subject Line

mailto:[email protected]

Technology

A Tale of Two Pipelines: To DevOps or Not To DevOps