Upload
josiah-renaudin
View
98
Download
3
Embed Size (px)
Citation preview
AW4Session6/8/1611:30AM
SlaytheDragonsofAgileMeasurement
Presentedby:
LarryMaccherone
AgileCraft
Broughttoyouby:
350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··[email protected]://www.techwell.com/
LarryMaccheroneAgileCraftAnindustry-recognizedleaderinagile,metrics,andvisualization,LarryMaccheronecurrentlyhelpsanumberofcompanieswiththedesignoftheiranalyticsproductsincludingAgileCraftandPendo.io.Previously,LarryledtheinsightsproductlineatRallySoftwarewhichenabledbetterdecisionswithdata,leveragedbigdatatechniquestoconductgroundbreakingresearch,andofferedthefirst-everagileperformancebenchmarkingcapability.BeforeRally,LarryworkedattheSoftwareEngineeringInstituteforsevenyearsconductingresearchonsoftwareengineeringmetricswithaparticularfocusonreintroducingquantitativeinsightbackintotheagilecommunities.
https://LinkedIn.com/in/LarryMaccherone
The Eight
Dragons of
Agile
Measurement
Topic presented by:
Larry Maccherone LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Jean
Tabaka She taught us how
to collaborate. She
made us all better.
She made us want
to be better. The
agile community
has lost a most
insightful, caring,
inspiring person.
https://LinkedIn.com/in/LarryMaccherone
Larry Maccherone LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Using measurement in
an agile environment
can be dangerous
https://LinkedIn.com/in/LarryMaccherone
The Dragons of Agile Measurement If you do metrics wrong, you will harm your agile transformation
1. Dragon: Measurement as a lever
Slayer: Measurement as feedback
2. Dragon: Unbalanced metrics
Slayer: 1 each for Do it
fast/right/on-time, and Keep doing it
3. Dragon: Metrics can replace
thinking
Slayer: Metrics compliment thinking
4. Dragon: Expensive metrics
Slayer: 1st work with the data you
are already passively gathering
5. Dragon: Using a convenient
metric
Slayer: Outcomes Decisions
Insight Metric (ODIM)
6. Dragon: Bad analysis
Slayer: Simple stats and
simulation
7. Dragon: Single outcome forecasts
Slayer: Forecasts w/ probability
8. Dragon: Human emotion and bias
Slayer: Tricks to avoid your own
biases and overcome those of
others
https://LinkedIn.com/in/LarryMaccherone
Dragon #1
Using measurement as a
lever to directly drive
behavior
https://LinkedIn.com/in/LarryMaccherone
Manipulating
Others
Dragon #1
Using metrics as a
lever to drive
someone else’s
behavior
https://LinkedIn.com/in/LarryMaccherone
Self
Improvement
Dragon slayer #1
Using metrics to
reflect on your own
performance
https://LinkedIn.com/in/LarryMaccherone
Dragon #2
Unbalanced Metrics
https://LinkedIn.com/in/LarryMaccherone
Do it fast
Do it on
time
Do it right
Keep doing
it
Balanced scorecard of metrics
at least one from each quadrant
https://LinkedIn.com/in/LarryMaccherone
Dragon #3
Metrics can replace thinking
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Dragon #4
Expensive metrics
https://LinkedIn.com/in/LarryMaccherone
Favor metrics derived from
existing records
https://LinkedIn.com/in/LarryMaccherone
Dragon #5
Using a convenient metric
aka “Lamp post metrics”
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Good players?
Monta Ellis
9th highest scorer
(8th last season)
Carmelo Anthony (Melo)
8th highest scorer
(3rd last season)
https://LinkedIn.com/in/LarryMaccherone
Not everything
that counts
can be counted,
and
not everything
that can be
counted,
counts.
https://LinkedIn.com/in/LarryMaccherone
Dragon slayer #5
ODIM
O U T C O M E
D E C I S I O N
I N S I G H T
M E A S U R E
THINK
EFFECT
like Vic Basili’s
Goal-Question-Metric (GQM)
but without
ISO/IEC 15939 baggage
https://LinkedIn.com/in/LarryMaccherone
Dragon #6
Bad analysis and
visualization
https://LinkedIn.com/in/LarryMaccherone
Top 10 criteria for great visualization
1. Answers the question,
"Compared with what?”
(SO What?)
2. Shows causality, or is at least
informed by it.
(NOW WHAT?)
3. Tells a story with whatever it
takes.
4. Is credible.
5. Has business value or impact in
its social context.
6. Shows
differences
easily.
7. Allows you to see the forest
AND the trees.
8. Informs along multiple
dimensions.
9. Leaves in the numbers where
possible.
10. Leaves out glitter.
Credits:
• Edward Tufte
• Stephen Few
• Gestalt (School of Psychology)
11. Uses good visual grammar
https://LinkedIn.com/in/LarryMaccherone
Time check?
Skip the next 8 slides if it’s
now past half the allotted time.
They go over the elements of
good visual grammar.
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (1)
When to use a straight line series?
1. Cumulative data
(later data points include
items from earlier data points)
example: Burn or Scope
series
2. Benchmark for column series
3. Connected by “ordinal inertia”
(ordinal usually = temporal)
However, spline is usually
what you want in this
instance.
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (2)
When to use a dashed line?
1. “Ideal” or reference
2. Forecast
3. Regression or fit
line
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (3)
When to use a column (vertical bar)
series?
1. Counts/sums independent of
neighbors
example: Throughput or
velocity chart
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (4)
When to use a bar (horizontal)
series?
(Rarely)
1. The x-axis variable is causal?
2. In a table
3. Stylistic reasons
• Length of labels
• Two-variable display
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (5)
When to use stacked area?
1. Multiple cumulative series
(the ordinal sum means
something)
example: a cumulative
flow diagram (CFD)
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (6)
When to use a single area series or
multiple (non-cumulative) area
series?
NEVER?
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (7)
When to use a single spline series?
1. Filling in for missing granularity
2. Rolling average
https://LinkedIn.com/in/LarryMaccherone
11. Good visual grammar (8)
When to use a single scatter series?
1. Two variables that are
imperfectly
correlated
https://LinkedIn.com/in/LarryMaccherone
Dragon #7
Single outcome forecasts
https://LinkedIn.com/in/LarryMaccherone
Monte Carlo Forecasting What it looks like
Live demo: http://lumenize.com (use Chrome)
https://LinkedIn.com/in/LarryMaccherone
Seek to
change the nature of
the conversation
https://LinkedIn.com/in/LarryMaccherone
Every decision is a
forecast!
https://LinkedIn.com/in/LarryMaccherone
You are forecasting that
your choice will have better
outcomes than the other
alternatives
https://LinkedIn.com/in/LarryMaccherone
So…
quality of decision depends upon:
1. alternatives considered, and
2. models used to forecast the
outcome of those alternatives.
Probabilistic models are superior
https://LinkedIn.com/in/LarryMaccherone
For a given alternative, let:
Pg = Probability of good thing happening
Vg = “Value” of good thing happening
Then:
Value of the alternative = Pg × Vg
https://LinkedIn.com/in/LarryMaccherone
An
lean/agile product
management example
https://LinkedIn.com/in/LarryMaccherone
$8M
Best case
(25%)
$1M
Likely case
(50%)
$1M
Worst case
(25%)
1
$2M $2M $1M
2
Which strategy is best…
…for your company?
PW × VW = .25 × -$1.00M = -$0.25M
PL × VL = .50 × $1.00M = $0.50M
PB × VB = .25 × $8.00M = $2.00M
-----------
$2.25M
…for your career?
PW × VW = .25 × $1.00M = $0.25M
PL × VL = .50 × $2.00M = $1.00M
PB × VB = .25 × $2.00M = $0.50M
-----------
$1.75M
https://LinkedIn.com/in/LarryMaccherone
If you get only 1 project then
strategy 2 is better
75% of the time
If you get ∞ projects then
strategy 1 is better
100% of the time
How many projects do you need for
strategy 1 to be better
more often than not?
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Play with it yourself at:
http://jsfiddle.net/lmaccherone/j3wh61r7/
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Dragon #8
Human emotion and bias
https://LinkedIn.com/in/LarryMaccherone @LMaccherone @TheAgileCraft
We don't see things
the way they are.
We see things
the way we are.
~The Talmud
Men's Airo Outdoor Shoes
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
Getting
probability
input you
can trust
https://LinkedIn.com/in/LarryMaccherone
Trained/Calibrated
Untrained/Uncalibrated
Statistical Error
“Ideal” Confidence
30%
40%
50%
60%
70%
80%
90%
100%
50% 60% 80% 90% 100%
25
75 71 65 58
21
17
68 152
65
45 21
70%
Assessed Chance Of Being Correct
Perc
ent C
orr
ect
99 # of Responses
We are inaccurate when assessing probabilities
Copyright HDR 2007
But, training can “calibrate” people so that of all the times they
say they are X% confident, they will be right X% of the time
https://LinkedIn.com/in/LarryMaccherone
Equivalent Bet calibration
What year did Newton published the Universal Laws of
Gravitation?
Pick year range that you are 90% certain it would fall within.
Win $1,000:
1. It is within your range; or
2. You spin this wheel and it lands green
Adjust your range until 1 and 2 seem equal.
Even pretending to bet money works.
90%
10%
https://LinkedIn.com/in/LarryMaccherone
Now what?
• Questions?
• Day-long seminar on agile metrics
• Workshop to design your own
metrics regimen
• AgileCraft Demo
• Make contact with me on LinkedIn
https://linkedin.com/in/larrymaccherone
Now what?
• Meet me at the AgileCraft booth to
answer questions
• Make contact with me on LinkedIn
https://LinkedIn.com/in/LarryMaccherone
https://LinkedIn.com/in/LarryMaccherone
“They” say…
Nobody knows what’s gonna happen
next: not on a freeway, not in an
airplane, not inside our own bodies
and certainly not on a racetrack with
40 other infantile egomaniacs.
– Days of Thunder
Trying to predict the future is like
trying to drive down a country road
at night with no lights while looking
out the back window.
– Peter Drucker
Never make predictions, especially
about the future.
– Casey Stengel
https://LinkedIn.com/in/LarryMaccherone
When you come to a
fork in the road…
take it!
~Yogi Berra
https://LinkedIn.com/in/LarryMaccherone
Now what?
• Questions?
• Day-long seminar on agile metrics
• Workshop to design your own
metrics regimen
• AgileCraft Demo
• Make contact with me on LinkedIn
https://linkedin.com/in/larrymaccherone
Now what?
• Meet me at the AgileCraft booth to
answer questions
• Share webinar recording of this
content to co-workers who couldn’t
be here https://attendee.gotowebinar.com/register
/5432450926944227330
• Make contact with me on LinkedIn
https://LinkedIn.com/in/LarryMaccherone