Measurement magic in world of DevOps

Kai Jokiniemi

https://fi.linkedin.com/in/kaijokiniemi

Magic of Measurement

Why?

Things are complicated.and faster.

One common misinterpretation / misuse: Bucks say “IT is not / achieving… “ It needs all to do it.

Need for information Ø As a basis for decisions Ø Gain situational view Ø in order to improve

WHAT IT IS ALL ABOUT?

Be aware of that, because it causes pitfalls• Measure test case amount, you will get cases but how about

usefulness?• Measure found faults, you will get faults reported but how about

usefulness?

Different view points and needs exists.

WHAT YOU MEASURE IS WHAT YOU GET

Learning

DELIVER

define

developmeasure

develop

measure

analyzeimprove

control

TRACK

Knowledge

develop

measure

analyze

improve control

DRIVE

Understanding

Is there something alarming?

Different aspects

Development Operations Finance

$

$

$

$

$$

$

$

$$

Different aspects

Development

Different aspects

Development

Different aspects

Development

Different aspects

Development

DoR - Definition of ReadyDoD - Definition of DoneDtP - Deployed to Production

Definition of

Ready

Definitionof

Done

Deployedto

ProductionBugs and

other feedback

ThingstoDo

Inputsfrom…

Coverage

DELIVER

Fillrate/TTIDoR100%

80%

Fillrate/TTIDoD100%

80%

Fillrate/TTIDtP100%

80%

#ToDoIssues

Time

#Bugs

Map with coverage

DoR - Definition of ReadyDoD - Definition of DoneDtP - Deployed to Production

Time Time

Coverage

DELIVER

Different aspects

OperationsThroughput

Services 92,1 %

CPU

Memory

Storage

TRACK

DELIVER

Orchestration

TRACK

DELIVER

Different aspects

Finance

Published feature amount vs actual sold.

Feature expected value vs realized value.• Story point amount per time equals money.• Invested money amount needs to be smaller than gained.• There needs to be known feature money forecast AND measure outcome against that. • In case not fulfilling, change.

TRACK

DELIVER

DRIVE

Different aspects

TRACK

DELIVER

DRIVE

DevelopmentØ Issues story point amount vs velocity (per

time)ü Bugs eats out development velocity =>

prediction

Ø probability indication: build success rate VS story point amount correlation and tests found bug fix average time

PredictabilityØ Gaining predictability: actual story

point per sprint vs velocity vs planned sp

Ø Fault amount - fix time Ø Fault hit rate vs coverage

Ø Failure rate - Fix timeØ CoverageØ Security vulnerabilitiesØ Code analysisØ Cycle time

Ø Service level Deployment succes rate Deployment recovery time

Ø Published features amount per month Deployment frequency

Ø Published feature amount vs actual sold Feature expected value vs realized value

deliver

drive

track

View point

deliverdrive

track

View point

deliver

drive

track

View point

deliver

drive

track

View point

SERVICE JAMTM

deliver

drive

trackView point

Meanwhile

deliver

drive

trackView point

deliver

drive

trackView point

deliver

drive

trackView point

How to deal data volumes?

Automate

Machine learning

Machine learning

effective with tons of logs

Especially

DELIVER

TRACK

DRIVE

TOOLS & ENVIRONMENT

OPERATIONAL EFFICIENCY

KPIs

DESIGNCREATE

ANALYZEAUTOMATE

ROOT ONE

From ideation to building.From release to the future.

CODE FOR GROWTH

Sources and additional reading

• Eficode materials “Tekoälyn vaikutukset ohjelmistotuotantoon -aamiaisseminaari 7.12. Helsingissä” Eficode materials “Devops ja mittaamisen magiaa -aamiaisseminaari Helsingissä 1.11.”

• DevOps and the Cost of Downtime: Fortune 1000, IDC• The Phoenix Project, Gene Kim, George Spafford, and Kevin Behr• Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 2014 Gartner, Inc. Cameron Haight

and Tapati Bandopadhyay• Metrics that Matter - IntelliTect

Come jam with us.

THANK YOU

Kai Jokiniemi

https://fi.linkedin.com/in/kaijokiniemi