KT2 Keynote 11/18/2010 4:30:00 PM 

       

"Meaningful Metrics for Agile Teams and Organizations"

   

Presented by:

Niel Nickolaisen EnergySolutions

          

Brought to you by:  

  

330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Niel Nickolaisen EnergySolutions Niel Nickolaisen is one of the founders of Accelinnova, a think tank focused on improving organizational and IT agility. Niel has held technology executive and operations executive positions—typically in turnaround roles. His strategic and tactical alignment model significantly improves returns on technology and business initiatives. Niel writes an enterprise CIO column for TechTarget and is the co-author of Stand Back and Deliver: Accelerating Business Agility. CRN Magazine named Niel one of the top 25 IT Executives of 2008.

Meaningful Metrics for Agile Teams and Organizations

The BIG IdeaWe should carefully select metrics because . . .

Metrics Influence Behavior

Why Be Careful?

Meaningfulmetrics lead to meaningful behavior – this is a good thing.

Meaningless metrics lead to meaningless behavior – we do better if we avoid this.

How About An Example?

Manufactured Stone

The Situation

• Two types of stone. • Flats • Corners

• Primary production metric is:– Square feet produced per person hour.

• It takes 3 times more person hours to produce a square foot of corners than it does flats. 

• Total cycle time (production and curing) is about one day. 

How Does This Metric Influence Behavior?

Actual Results

• The company has combined inventory turns of 4 per year.

• But, there are not enough corners to meet customer demand.

• As a result, On‐Time Shipments are lower than 50%.

• But, each plant exceeds its square foot per person hour goal – Hooray!

In Other Words . . .

How Can We Solve This Problem?

First Option

Change that troublesome On‐Time Shipment metric!

Old On‐Time

• Order is late if a single line item does not ship.

New On‐Time

• Order is on‐time if a single line item ships.

The Results?

On‐Time Score Soars to over 95%!

“What is it good for? Absolutely Nothing” - War

The Big Quiz

• Did the square feet per person‐hour lead to meaningful behavior?

• Did it improve the company’s production processes? Its competitive advantage?

• Is this a meaningful metric?

• What would have been the characteristics of a more meaningful metric?

ANOTHER EXAMPLE

• The company wants to encourage employee involvement. 

• Company defines a metric and goal for:– Five employee suggestions per month.

• What happened? A labeling frenzy.

How About One More?

How meaningful is this measure of development productivity?How many of the lines were usable?

How many of the lines were high priority?How many of the lines generated value?

OK, So What Do We Do?

My over‐arching goal is to use metrics that:–Measure accomplishment rather than activity.

–Lead to continuous process improvement.

Acceptance Criteria

Meaningful Metrics:

1. Represent reality2. Measure processes not people3. Are few in number4. Are mostly non‐financial5. Align with strategy6. Show trends

1. Represent Reality

According to the great philosopher Polonius . . .

“To thine own self be true”

And, the most important characteristic of successful leaders?

Self Awareness

2.  Measuring Processes Not People

Some philosophical mumbo jumbo:

– “85% of a person's effectiveness is determined by the system he works within, only 15% by his own skill. “ – Edward Deming

– “Our real aim is to bring out the capabilities of each individual. The ultimate aim is to draw out people’s motivations.” – Inside the Mind of Toyota

If we measure people, we miss the majority of the improvement opportunities and crush motivation.

Let’s Return to An Example

• Does lines of code per day (or per developer day / hour) measure processes or people? 

• If processes, what processes does it tell us to improve?

3.  Are Few In Number

Lines of Developer Code Per Day % Requirements Tested

Code Coverage Code Merge Conflicts

On‐Time % Security Mapped To Code

On‐Budget Ratio of Internal / External Bugs Found

Defects Per Line % Technical Debt Remaining

Customer Request Completion Estimate Accuracy

Bugs Fixed % Documentation Complete

% Data Redundancy Maintenance Cost Per Line of Code

If we want to measure (and improve) processes and not people, how many do we need to drive meaningful development behavior?

Are Few In Number . . .

• To avoid metric conflicts (lines of code per developer day / % documentation complete).

• So that we can focus our continuous process improvements on those that generate the most value.

• “The Only Metric You Will Ever Need” – The  Net Promoter Score– How many of our customers would recommend us to someone they know?

– How many of our customers – if given the choice – would use us again?

4.  Are Mostly Non‐Financial

What choice do we have?• What about revenue

generated per line of code? What is the direct, measurable link?

• How about product profitability? How meaningful is this for our compliance module? Our security module?

5.  Align With Strategy

We favor Accomplishment over Activity

But, what defines Accomplishment?

6.  Show Trends

Is this contribution margin performance good or bad?

June

Total Variable Costs as a % of Revenue 59.07%

Materials as a % of Revenue 32.70%

Direct Labor as a % of Revenue 6.00%

Variable Overhead as a % of Revenue 8.10%

Commissions as a % of Revenue 3.40%

Does This Help?

So, What Do We Do?

Start with 

strategy

MarketDifferentiating

High

Low

Mission CriticalLow High

Differentiate

Parity

Partner?

Who cares?

Purpose Based Alignment Model

So What?

• Two big benefits:–First, in our market, what creates competitive advantage? How can we measure how well we are doing this?

–Second, in our market, how do we compare to “parity”?

• This also defines Accomplishment over Activity

Mapping Differentiating to Measures

How do we measure the processes that directly generate competitive advantage?

Specialty Retailer

Differentiating = Product SelectionHow to measure software

development?

Product Line Growth

Decision Quality

Quality and Timeliness of Business Analytics

Healthcare Software Company

Differentiating = Product Innovation

Measure? Percent of revenue from products released in the past two years.

Mapping Parity to Measures

What is barely sufficient?How does this map to metrics?

Specialty Retailer

Starting Point: Overly complex business rules (better than parity)

In order to improve this process, measure % of required customizations in ERP, POS, CRM systems.

Healthcare Software Company

Differentiating = Product InnovationGoal: Free up development to work on

new stuffMeasure? % of staff supporting 3+ year

old releases drove improvements in lifecycle management

Some Metric Areas To Consider

Cycle Time

Concept to First IterationFeature Request to DeploymentDefect to Patch

Business Case

Product PerformanceMarket Share Growth% of Revenue From New Products

Customer Service

Net Promoter Score

Do We Keep Our Commitments?

Helpful versus Right

Customer Retention

Recap

Meaningful Metrics:–Represent reality–Measure processes not people.–Are few in number.–Are mostly non‐financial.–Align with strategy.–Show trends.

Getting Started

Brainstorm

A process (not people) to improve

Map

The process to purpose –differentiating / parity – to establish the design goal

Select

Meaningful metrics that will monitor improvements to the process

Pilot

The metrics. If they work, the process will improve.

If not, they won’t.

If they stop working, stop using them.

When Good Enough

Move to the next process improvement candidate

A Final Thought

Think it through.Goal, reduce calls

to the service desk by 50%.

Results? Unplug half the phone lines.

Questions?

• nnick@headwaters.com