© John Durbin Slide 1May 2004 At the Chicago SPIN Metrics - Creating a Balanced Scorecard aligned to Corporate Objectives and Goals John Durbin [email protected]

At the Chicago SPIN © John Durbin Slide 1May 2004

Metrics - Creating a Balanced Scorecard aligned to

Corporate Objectives and Goals

John Durbin

[email protected]

Chicago - SPIN


Balanced Scorecards at the Operational Level

How to: Align Metrics to organizational scorecards Align Bottom-up Balances Scorecard to the

next higher level Do you have points of special interest? Balanced Scorecard used is just an example for

bottom up alignment to “organizational goals”


Topics NOT covered…

Why Balanced Scorecards were devised? Good and Bad about Balanced Scorecards Metrics vs. Performance indicators Well defined metrics Successful metrics deployment Negative Behavioral impact of metrics Life cycle for metrics for changing behavior Etc.


What is a metric?

A Quantitative result from one or more measurementsAnd is Verifiable from a defined process

Review: for Common terminology


How are metrics used?

Support Decisions or Drive change or Diagnose problems



Performance Indicators - Flavors

Planning Execution Lessons Learned

Performance Indicators

Measurements / Metrics

Assessments

Surveys

Performance Indicators maybe used for:


© John Durbin Slide 7May 2004

Where to start?

Mission (Values – Interests) To help our clients create the metrics-driven

processes and systems needed to change to be more successful.

Strategic Imperatives Critical Success Factors (capabilities) Key Performance Indicators (KPIs) > Goals Align Management and Decision-Making

Processes


Performance measures provide actionable data and use targets for achieving operational excellence in critical processes to focus on proper allocation of resources and appropriate incentive for improvement.

Process

StrategyPerformance measures provide an ongoing mechanism for measuring an organization’s success in channeling its resources toward achieving its near-and long-term strategic goals.

PeoplePerformance measures align the objectives of the individual with the overall organizational strategy ensuring an overallcommitment toward common goals and objectives.

Balanced Scorecard PurposeBalanced Scorecard Purpose


Balanced Scorecard (Norton)

Financial

InternalBusiness

ProcessesLearning &

Growth

Customer

First use at Analog Devices in 1987

ObjectivesMeasuresTargetsInitiatives


Strategic Imperatives Critical Success FactorsKey Performance IndicatorsOr Goals

Be a Full Service Sales ExpertiseProvider Integration w/other

Business Units

Build Brand Aggressive MarketingImage Project Management

Expertise

Increase Value Increase ServicePerception Turnaround

Fix Infrastructure Train People Develop Systems

• Win-Loss Ratio• % Cross-BU Sales• Milestone Completion

• # New Customers• On-time Project Completion• # New Distributors• Market Share

• Service Response Time• Customer Satisfaction

• Inventory Turns• Training Effectiveness• Milestone Completion

Define Key Define Key IndicatorsIndicators


Need for Balanced Metricsfor Performance

Avoid missing key issue Long term focus Do the Right Things Avoid disasters (1-Digital Equip.>

Selling incentives on $ of computers; not profits, 2- Tech. Co. > $ ship product on-time: no $ for/of repair)


A - Balanced Scorecard

Customers give us top two scores of five- 70% Customers recommend our product - 80% Met or exceed profit targets - $$. Demonstrate SEI CMM level 4 by… ;-) … Employees more than satisfied > 75% “Grow” 300 new Six Sigma Black Belts


A Fed. Gov. Balanced Scorecard

Customer Perspective

Customer Satisfaction - % of customers satisfied with timeliness - % of customers satisfied with quality Effective Service Partnership - % of customers satisfied with the responsiveness, cooperation, and communication skills of the acquisition office

Internal Business Processes PerspectiveAcquisition Excellence: Effective Quality Control System - Ratio of protests sustained by General Accounting Office and Court of Federal ClaimsAcquisition Excellence: Effective use of Alternative Procurement Practices - Number of actions using Electronic CommerceFulfilling Public Policy Objectives - % achievement of socio-economic goals - % competitive procurement of total procurements

Learning and Growth Perspective Information Availability for Strategic Decision-making - The extent of reliable management information Quality Workforce - % of employees meeting mandatory qualification standards Employee Satisfaction: Quality Work Environment - % of employees satisfied with the work environment Employee Satisfaction: Executive Leadership - % of employees satisfied with the professionalism, culture, values and empowerment

Financial Perspective

Minimizing Administrative Costs - Cost to spend ratioMaximizing Contract Cost Avoidance - Cost avoidance through use of purchase cards - % of prompt payment interest paid of total $ disbursed


Balanced Scorecard For Operational Group

Support Organizational Goals & Results Aligns to Top Scorecard (s) Consistent focus and perspective How do we get a Balanced Scorecard

defined to align upward? GE’s (conceptual) Formula view or Pyramid conceptual model


Cook Book Recipe:Two Dimensions to Build

First, VERTICALLY

Build alignment to Goals and Metrics above you.

Second, HORIZONTALLY

Derive a balanced set of goals and metrics across top level perspective.

Last, Select and Prune Metrics


GE’s Big Y, little y & x

Results that matter Which Major activities will enable you to achieve those results?

Y = f (y1 , y2 , y3 ... )

On-going Sponsorship and Review

Management Stewardship


Y = f(yi) and yi = f(xij)

yiEnabling Results

that matterWhich Operating activities will enable

you to achieve those results?

= f (xi1 , xi2 , xi3 ... )

Process owner’s Review

Results/ Process Focus


ProjectManag’t

Team orWork Cells

Functional Management

StakeholdersOrganization

Operating Units

Drivers:Top-down

Enablers:Bottom-up Metrics Drive Improvements

Few Metrics

Operating unit Scorecards

Different Processes

Enterprise Balanced Scorecard:Financial, Customer, Internal Business, and Innovation and learning.

Metrics helps you do what you aredoing while you are doing it!

Conceptual Model for Your Balanced Scorecards

Across different initiatives

Deliverable and milestone focused

Process, diagnostic and Supportive

Enterprise

Milestone status - Product results

Key PerformanceIndicators

Process support


ProjectManag’t

Team orWork Cells

Functional Management

StakeholdersCustomersOrganization

other layers

Drivers:Top-down

Enablers:Bottom-up Metrics Drive

Vital Few MetricsCustomer Survey Cost/Benefits

PlanAnalyze

DesignBuild

OperateRUP, Agile, WaterfallHardware Dev.

Marketing - Logistics

Customer Satisfaction Employee Satisfaction, ROI Profits: $, DM, FF, etc.

Improvements Helps you do what you aredoing while you are doing it!

Align Metrics forConsistency of Purpose

Across projects and Methods

Deliverable and Phase focused

Process, Diagnostic and task focused

MarketBusiness Unit

Phase status - Exit criteria

Key ProcessIndicators

Process support


Performance Indicators

Metrics elseIndicators

Your Process-Development-Enhancement-Support

Inputs Outputs

Time & CostOf resources

Product SizeDefects

What?Why?

Attributes•People•Tools•Techniques•Physical Environment

What?


A Project Process’ Metrics Model

Input Process Output

Project Metrics

Productivity RateProduct Test Defect Density

Duration Variance PercentageCost per unit code

Attributes•People•Environment•Tools•Techniques

TimeCost

SizeDefects

Resources & People

DeliverablesTime

- Sch

edule

Plan Analyze Design Integrate Deploy OperateBuild

Q U A L I T Y Q U A L I T Y Risk?

Risk?

Risk?


Balanced Scorecard (Norton)

Financial

InternalBusiness

ProcessesLearning &

Growth

Customer

First use at Analog Devices in 1987

ObjectivesMeasuresTargetsInitiatives


Balanced Scorecard Horizontally

Financial Perspective: (ROI) How do we look to our shareholders?

Internal Business Perspective: (Process) What Business Processes must we EXCEL at?

Customer Perspective: (Vision) How should we appear to our customers?

Innovation & Learning Perspective: (Change) How do we continue to improve and add value?


Balanced Scorecard Vertically – A Drill Down

Improve Customer Satisfaction: Reduce delivered software defects

– Increase software Verification Better Requirements Traceability to …. Improve Inspections effectiveness to….

– Improve software Validation

Faster feature enhancements– Use Agile paradigm to continually verify …


Scorecard trials

Improve Customer Satisfaction Reduce delivered software defects

– Better customer acceptance coverage <???– Greater storage for test cases <???– New Configuration Management tools <???– Use Agile Methods <???

What is common to the above? (Internal vs goal focus)


XYZ Retail Goods Co.

L1- Products in every Wal-Mart L2 Timely and Accurate market research

L3 Improve link to data sourcesl4 Improve extract timing and accuracy of market

data. L2 Lower Costs

– L3 Reduce delays of misplaced goodsl4 Correct tracking system for Goods availability

L1- 20% annual sales growth


Critical Success Factors for Metric Definitions

Anticipate “What’s In It For Me?” For Sponsors - Owners For Participants

Seek common and clearly defined terminology (currency for Knowledge)

Must have owner committed not just involved


Select and Prune Metrics

“Not everything that can be counted counts. Not

everything that counts can be counted.” Albert Einstein


For a small group or persons -Minimize number

of Metrics

Too many metrics to “focus on” causes “thrashing” or dysfunction behavior.

“Foreground metrics” - people may be measured by << less the 5 (maybe 7)

“Background metrics” – people must not be measured on >> are for heads up and MUST be trusted as non performance measurement – these may be prior metrics that were always under “control”


Examples of Balanced Set for Software Development

Vital Few Metrics

(Like for internal delivery) Testing Metrics

(Like a stand alone group) SQERT <an FLA>

(Like for Contracts)


The Vital Few Metrics are a Balanced Set

Quality Process Quality Product Quality

Performance Productivity Rate Delivery Rate

Predictability Duration Variance Percentage Effort Variance Percentage


Example Metrics for Measuring Systems

Building

Productivity Rate =

(speed of the process)

Defect Density =

Delivery Rate =

(elapsed time)

Cost Per output unit =

Product SizeEffort

No. of Defects Found in Product TestProduct Size

Product Size Elapsed Time

Project CostProduct Size

Copyright 1996


1. Fault Density1. Fault Density

6. Effort Variance

Percentage

6. Effort Variance

Percentage

3. Productivity Rate

3. Productivity Rate

4. Delivery Rate

4. Delivery Rate

2. Defect Density

2. Defect Density

Metrics# of Faults

(for 3 month period)

# of Faults (for 3 month period)

Actual EffortActual Effort

Planned EffortPlanned Effort

# of Defects found During Product Test

# of Defects found During Product Test

# of Project Function Points

# of Project Function Points

Actual Start Date(of analysis)

Actual Start Date(of analysis)

Actual End Date(of first roll-out)

Actual End Date(of first roll-out)

Planned Start Date(of analysis)

Planned Start Date(of analysis)

Planned End Date(of first roll-out)

Planned End Date(of first roll-out)

Measurement Data

Minimize Source Data

5. Duration Variance

Percentage

5. Duration Variance

Percentage


A Testing Metrics are a Balanced Set

Productivity Test Execution Rate Problem Rate Problem Fix Rate

Process Quality Repair Effectiveness Percentage Repair Effort Percentage

Product Quality Number of problems by module

What could be added?


S.Q.E.R.T. Balanced Set

Size Time Cost Defects

Scope Quality Effort Risk Timeline


References… Cox, B.G. and Chinnappa, B. N. (1995). "Unique Features of Business

Surveys." Ch. 1 in Business Survey Methods. Wiley. NY. Deming, W. Edwards. (1986). Out of the Crisis. MIT Center for Advanced

Engineering Study. Cambridge, MA. Hibbard, J. (1997). "Knowing What We Know". Information Week. [Oct. 20].

pp. 46-64. Juran, J.M. (1989).Juran on Leadership for Quality. Free Press. New York. Gabor, A. (1990). The Man Who Discovered Quality. Penguin Books. New

York. Kaplan, R.S., and Norton, D.P. (1996) The Balanced Scorecard: Translating

Strategy into Action. Harvard Business School Press. Boston. Shewhart, W. (1939). Statistical Method from the Viewpoint of Quality

Control. in Deming, W.E., ed., Graduate School of the Department of Agriculture. Washington, DC.


Other sources

Me>> [email protected] http://www.balancedscorecard.org/ http://oamweb.osec.doc.gov/bsc/guide.htm For more than you wanted to know:

http://harvardbusinessonline.hbsp.harvard.edu/b02/en/home/index.jhtml?_requestid=20215

http://www.sixsigmabenchmarking.com/ http://www.som.cranfield.ac.uk/som/cbp/pma/


End of presentation

Remaining slides are FYI extras Goal Question Metric Deployment Critical Success factors


Goal-Question-Metric Paradigm

Top-down selection process Linked to organizational goals Ensures traceable and purposeful metrics G-Q-M process

Identify your KPI or goals What questions can you ask to determine

whether or not your goal was met? What metrics answer your question?


Applying G-Q-M

Must decide responsibilities for these actions Defining the Metrics Communicating Educating Gathering Data Reporting Data Applying Metrics

Decision, Change, Diagnosis

Victor R. Basili – Univ. Maryland


G-Q-M Relationship

Goal 1

Goal 2

Question 1 Metric 1

Metric 2

Metric 3

Metric 4

Metric 5

Metric 7

Question 2

Question 3

Question 4

Question 5


G-Q-M Example

Goal: Maximize customer satisfaction

Question: How long does it take to fix a problem?

Metric: Mean time to acknowledge problem.

Metric: Mean time to deliver solution.

Metric: Schedule versus actual delivery.

Question: How many problems are affecting customers?

Metric: Incoming defect rate.

Metric: Open critical and serious defects.


G-Q-M Example

Goal:

Question:

Metric:

Improve InspectionResults

Is time well focused? Are participants prepared?

- Review time per inspection (< 2 hours target)

- # Interruptions (zero target)

- Preparation per inspection time (1 to 2 hours)

- Materials Available (y,n)


Process Maturity for Metrics

Effe

ctiv

enes

s

Use over TIME

Technology transfer curveor process maturity curve

“Doing it” Metrics to supporttransition & learning

two or less metrics toincrease quality

Several metrics toincrease both qualityand productivity

Reduce to none or One Process Metrics to sustain quality or productivity

Check other processesfor metrics conflicts

Idealistic curve


Metrics Definition Template

M e t r i c sP e r s p e c t i v e

D e s c r i b e w h a t y o u a r e t r y i n g t o m e a s u r e : T i m e , c o s t , p r o d u c t s i z e ,d e f e c t s , v a l u e . A l s o , i n c l u d e t h e v i e w o f t h e m e t r i c : m e a s u r i n gp r o c e s s , p r o j e c t , p r o d u c t

M e t r i c P u r p o s e G O A L : D e f i n e w h a t g o a l i s t h i s m e t r i c s u p p o r t i n gQ U E S T I O N : D e f i n e w h a t q u e s t i o n t h i s m e t r i c i s a n s w e r i n gE X P L A N A T I O N : D e f i n e w h y t h i s m e t r i c i s b e i n g c a p t u r e d a n dh o w i t w i l l b e u s e d .

T o t h ei n o r d e r

t o

a t t r i b u t eo f t h ee n t i t y

g o a l ( s )u n d e r s t a n de v a l u a t ec o n t r o lp r e d i c t

B E N E F I T : W h a t i s t h e b e n e f i t o f u s i n g t h i s m e t r i c ?M e t r i c s F o r m u l a S h o w t h e f o r m u l a u s e d t o c a l c u l a t e t h e m e t r i c s

F r e q u e n c y H o w o f t e n i s t h e m e a s u r e m e n t d a t a g a t h e r e d a n d h o w o f t e n i s t h em e t r i c r e s u l t r e p o r t e d ? Q u a r t e r l y , a t p h a s e c o m p l e t i o n , a t p r o j e c tc o m p l e t i o n , a n n u a l l y

S u p p l i e r s o fM e a s u r e m e n t s

W h o a r e t h e p o t e n t i a l p e o p l e ( t i t l e ) t h a t c a n p r o v i d e t h i s d a t a

S o u r c e s o fM e a s u r e m e n t s

I d e n t i f y t h e d o c u m e n t s a n d t o o l s t h a t c o n t a i n t h e d a t a

M e a s u r e m e n t s C l e a r l y d e f i n e t h e m e a s u r e m e n t s ( d a t a p o i n t s ) t h a t n e e d t o b ec o l l e c t e d i n o r d e r t o c a l c u l a t e t h e m e t r i cD e f i n e w h a t s h o u l d a n d s h o u l d n o t b e i n c l u d e d i n t h e d a t a c o l l e c t e d

B a s e l i n e W h e n w i l l t h e b a s e l i n e b e e s t a b l i s h e d a n d w h e r e w i l l y o u g e t t h ed a t a ?

T a r g e t T a r g e t v a l u e s f o r t h e m e t r i c a n d t h e i n t e r p r e t a t i o nW h a t c o n s t i t u t e s “ g o o d n e s s ”T a r g e t v a l u e f o r a n e n g a g e m e n t b a s e d o n t h e c o n t r a c t

R e l a t e d M e t r i c s R e l a t i o n s h i p s b e t w e e n m e t r i c s s h o u l d b e d e s c r i b e d

E x a m p l e o fM e t r i c

G i v e a n e x a m p l e s i t u a t i o n , t h e m e t r i c c a l c u l a t i o n a n d r e s u l t

C o m m e n t s :

Œ

Œ

Ž‘’’

Why?

Who Owns?

Defined?

When?

Suppliers?

Baseline & Target?

RelatedMetrics?


Deployment - Critical Success Factors?

Definition Deployment

Management

Support

Reinforcing

Communications


Checklist for Better Behavior

Metrics should: Support goals linked to business objectives Support goals or higher level metrics Assess products and process (not people) Be manageable alongside the other metrics Be meaningful to people reporting Be understood by all affected Encourage correct positive behavior Be a positive (+) indicator (when possible) Be visibly acted upon (decisions and diagnostics) Be collected from within the target process Be about something that the measured group can

control

Documents

© John Durbin Slide 1May 2004 At the Chicago SPIN Metrics - Creating a Balanced Scorecard aligned to Corporate Objectives and Goals John Durbin [email protected]