View
224
Download
1
Tags:
Embed Size (px)
Citation preview
Measuring ROI in Measuring ROI in the Public Sectorthe Public Sector
Jack J. Phillips, Ph.D. March 9, 2004Jack J. Phillips, Ph.D. March 9, 2004
AgendaAgenda
• Review current status of ROI implementation
• Explore the myths and mysteries of ROI
• Identify the key steps involved in the ROI methodology
• Examine the key issues involved in ROI implementation in the public sector
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No business need for the program
Program linked to specific business needs
Results-Based ApproachResults-Based Approach
No assessment of performance issues
Assessment of performance effectiveness
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No specific measurable objectives
Specific objectives for behavior and business impact
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No effort to prepare program participants to achieve results
Results expectation communicated to participants
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No effort to prepare the work environment to support transfer
Environment prepared to support transfer
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No efforts to build partnerships with key managers
Partnerships established with key managers and clients
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
No measurement of results or benefit-cost analysis
Measurement of results and benefit-cost analysis
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Results-Based ApproachResults-Based Approach
Planning and reporting is input focused
Planning and reporting is output focused
Paradigm Shift for a Paradigm Shift for a Results-Based ApproachResults-Based Approach
Activity BasedActivity Based
Key Measurement IssuesKey Measurement Issues
•Time?
•Costs?
•Data Collection?
•Management Support?
•Skills
Why ROI?Why ROI?
• ROI is the ultimate level of evaluationROI is the ultimate level of evaluation• ROI is a concept familiar to most ROI is a concept familiar to most
managersmanagers• ROI has a rich history of applicationROI has a rich history of application• Many top executives are requiring ROI Many top executives are requiring ROI
informationinformation
The ROI issue cannot be The ROI issue cannot be ignored!ignored!
The ROI ProcessThe ROI Process
•Reaction, Satisfaction, and Planned Action
•Learning
•Application and Implementation
•Business Impact
•Return on Investment
•Intangible Measures
….and includes a technique to isolate the effect of the program or solution.
Generates six types of data
ROI by the NumbersROI by the Numbers
• Process refined over a 20-year period
• Thousands of impact studies conducted each year
• 100 case studies published
• 2,000 individuals certified
• 15 books developed to support the process
• 500 member professional network
• ROI Process adopted by hundreds of organizations in 38 countries
PayoffPayoff
• Aligns programs to business needsAligns programs to business needs
• Show contributions of selected Show contributions of selected programsprograms
• Earn respect of senior management/ Earn respect of senior management/ administratorsadministrators
• Build staff moraleBuild staff morale
• Justify/defend budgetsJustify/defend budgets
PayoffPayoff
• Improve support for human resources/ Improve support for human resources/ learning/developmentlearning/development
• Enhance design and implementation Enhance design and implementation processesprocesses
• Identify inefficient programs that need to Identify inefficient programs that need to be redesigned or eliminatedbe redesigned or eliminated
• Identify successful programs that can Identify successful programs that can be expandedbe expanded
ROI ApplicationsROI Applications• Associate Relations Associate Relations
ProgramsPrograms• Career Development Career Development
ProgramsPrograms• Competency SystemsCompetency Systems• Diversity ProgramsDiversity Programs• E-LearningE-Learning• Executive CoachingExecutive Coaching• Executive EducationExecutive Education• Gain Sharing ProgramsGain Sharing Programs• Global LeadershipGlobal Leadership
• Organization DevelopmentOrganization Development• Orientation SystemsOrientation Systems• Recruiting StrategiesRecruiting Strategies• Safety & Health ProgramsSafety & Health Programs• Self-Directed TeamsSelf-Directed Teams• Skill-Based/Knowledge-Skill-Based/Knowledge-
Based CompensationBased Compensation• Technology ImplementationTechnology Implementation• Total Quality ManagementTotal Quality Management• Wellness/Fitness InitiativesWellness/Fitness Initiatives
Who is Using the ROI Process?Who is Using the ROI Process?Private Sector OrganizationsPrivate Sector Organizations
• Allstate Insurance• Amazon.com• Apple Computer• AT&T• Bristol-Myers Squibb• Coca Cola• Dell Computers• Deloitte & Touche• Delta Airlines• DHL Worldwide Express• Federal Express• General Motors• Georgia Pacific• Hewlett Packard
• Hilton Hotels• Home Depot• Intel• Lockheed Martin• Motorola• NCR• Nextel• Olive Garden Restaurant• PricewaterhouseCoopers• Shell Oil• Singapore Airlines• TD Bank• Verizon Communications• Wachovia Bank
Who is Using the ROI Process?Who is Using the ROI Process?Government OrganizationsGovernment Organizations
• US Department of Defense
• US Internal Revenue Service
• US National Security Agency
• US Department of Veteran Affairs
• NASA
• State of New York
• State of Mississippi
• State of Texas
• City of Richmond, Virginia
• Government of New Zealand
• Government of Singapore
• Government of Italy
• Government of Australia
• Provincial Government of Ontario, Canada
ROI QuizROI Quiz
Appropriate InterventionsAppropriate Interventions
• Performance Improvement
• Training and Learning Solutions
• Organization Development
• HR Programs
• Change Initiatives
• Technology Implementation
Public Sector IssuesPublic Sector Issues
• Absence of revenues and profitsAbsence of revenues and profits• Absence of hard dataAbsence of hard data• Whose ROI?Whose ROI?• Government services are essential and, Government services are essential and,
therefore, may not need this level of therefore, may not need this level of evaluationevaluation
• Restricted range of options to correct Restricted range of options to correct problemsproblems
An EvaluationFramework
Case Applicationsand Practice
A ProcessModel
OperatingStandards and
Philosophy
ImplementationImplementation
ROI: The Pieces of the PuzzleROI: The Pieces of the Puzzle
Evaluation LevelsEvaluation LevelsLevelLevel Measurement FocusMeasurement Focus
1.1. Reaction & Reaction & Planned ActionPlanned Action
Measures participant satisfaction with Measures participant satisfaction with the program and captures planned the program and captures planned actionsactions
2.2. LearningLearning Measures changes in knowledge, Measures changes in knowledge, skills, and attitudesskills, and attitudes
3.3. ApplicationApplication Measures changes in on-the-job Measures changes in on-the-job behaviorbehavior
4.4. Business ImpactBusiness Impact Measures changes in business impact Measures changes in business impact variablesvariables
5.5. Return on Return on Investment Investment
Compares program benefits to the Compares program benefits to the costscosts
Characteristics of Evaluation LevelsCharacteristics of Evaluation Levels
Chain of Chain of Value ofValue of Customer Customer FrequencyFrequency Difficulty ofDifficulty ofImpactImpact InformationInformation FocusFocus of Useof Use
AssessmentAssessment
SatisfactionSatisfaction LowestLowest ConsumerConsumer FrequentFrequent Easy Easy
LearningLearning
ApplicationApplication
ImpactImpact
ROIROI HighestHighest ClientClient InfrequentInfrequent DifficultDifficultCustomers
Consumers: The customers who are actively involved in the process.
Client: The customers who fund, support, and approve the project
Linking Assessment with EvaluationNeeds Program
Assessment Objectives Evaluation
Business Impact BusinessNeeds Objectives Impact
Job Performance Application ApplicationNeeds Objectives
Skills/Knowledge Learning LearningNeeds Objectives
Preferences Satisfaction ReactionObjectives
4 4
3 3
2 2
1 1
Matching Evaluation Levels Matching Evaluation Levels with Objectiveswith Objectives
Develop Develop Objectives ofObjectives ofSolution (s)Solution (s)
Develop Develop EvaluationEvaluationPlans andPlans and
Baseline DataBaseline Data
CollectCollectData DuringData During
SolutionSolutionImplementationImplementation
CollectCollectData AfterData AfterSolutionSolution
ImplementationImplementation
Level 2:Learning
Level 4:Business Impact
Level 1:Reaction, Satisfaction, and Planned Actions
Level 3:Application/ Implementation
EvaluationPlanning
Data Collection
THE ROI PROCESSCalculating the Return on
Investmentof a Business Performance Solution
THE ROI PROCESSCalculating the Return on
Investmentof a Business Performance Solution
Isolate the Isolate the EffectsEffects
Convert DataConvert Datato Monetaryto Monetary
ValueValue
Calculate theCalculate theReturn on Return on InvestmentInvestment
Tabulate CostsTabulate Costsof Solutionof Solution
IdentifyIdentifyIntangibleIntangibleMeasuresMeasures
GenerateGenerateImpactImpactStudyStudy
Level 5:ROI
Intangible Measures
Data Analysis Reporting
Collecting Collecting
•Follow-Up Surveys•Follow-Up Questionnaires•Observation On the Job•Interviews with Participants•Follow-Up Focus Groups•Program Assignments•Action Planning•Performance Contracting•Program Follow-Up Session•Performance Monitoring
Level 3
Level 4Level 4
IsolatingIsolating•Use of a control group arrangement
•Trend line analysis of performance data
•Use of forecasting methods of performance data
•Participant’s estimate of program impact (percent)
•Supervisor’s estimate of program impact (percent)
•Management’s estimate of program impact (percent)
•Use of previous studies
•Subordinate’s report of other factors
•Calculating/Estimating the impact of other factors
•Use of customer input
• Converting output to contribution - standard value
• Converting the cost of quality - standard value
• Converting employee’s time
• Using historical costs
• Using internal and external experts
• Using data from external databases
• Linking with other measures
• Using participants’ estimates
• Using supervisors’ and managers’ estimates
• Using staff estimates
ConvertingConverting
•Needs Assessment (Prorated)
•Development Costs (Prorated)
•Program Materials
•Instructor/Facilitator Costs
•Facilities Costs
•Travel/Lodging/Meals
•Participant Salaries and Benefits
•Administrative/Overhead Costs
•Evaluation Costs
TabulatingTabulating
Intangible BenefitsIntangible Benefits
•Increased Job Satisfaction
•Increased Organizational Commitment
•Improved Teamwork
•Improved Customer Service
•Reduced Complaints
•Reduced Conflicts
•Reduced Stress
Return on InvestmentReturn on Investment
Benefits / Program Benefits Costs = Ratio Program Costs
Net Program Benefits ROI =
Program Costs
Calculating ROICalculating ROI
BCRBCR = = = 3.0 = 3.0
ROI ROI == = ???% = ???%
$240,000$80,000
Total Benefits - Program CostsProgram Costs
BCRBCR = = = 3.0 = 3.0
ROI ROI == = 200% = 200%
$240,000$80,000
$240,000 - $80,000$80,000
Calculating ROICalculating ROI
ROI Target OptionsROI Target Options
•Set the value as with other investments, e.g. 15%
•Set slightly above other investments, e.g. 25%
•Set at break even - 0%
•Set at client expectations
When properly implemented, high ROI When properly implemented, high ROI values can be achieved with programs on:values can be achieved with programs on:
• LeadershipLeadership
• Team BuildingTeam Building
• Management Management DevelopmentDevelopment
• Supervisor TrainingSupervisor Training
• Sales TrainingSales Training
. . . 100% to 700% ROI is not uncommon
ROI Best PracticesROI Best Practices
1.1. Implemented as a process improvement tool Implemented as a process improvement tool not a performance evaluation toolnot a performance evaluation tool
2.2. Impact studies are conducted involving 5-Impact studies are conducted involving 5-10% of the programs10% of the programs
3.3. A variety of data collection methods are usedA variety of data collection methods are used
4.4. The effects of learning/development are The effects of learning/development are isolated isolated
5.5. Business impact data are convertedBusiness impact data are converted
6.6. ROI evaluation targets are developedROI evaluation targets are developed
7.7. The ROI methodology generates a micro The ROI methodology generates a micro level scorecardlevel scorecard
8.8. Data are being integrated to create a macro Data are being integrated to create a macro scorecardscorecard
9.9. Implemented for about 3-5% of the budgetImplemented for about 3-5% of the budget
10.10. Routine ROI forecastingRoutine ROI forecasting
11.11. Used as a tool to strengthen/improveUsed as a tool to strengthen/improve
ROI Best PracticesROI Best Practices
Building a ScorecardBuilding a Scorecard
• Provides macro-level perspective of successProvides macro-level perspective of success
• Serves as a brief report versus detailed studyServes as a brief report versus detailed study
• Shows connection of training’s contribution to Shows connection of training’s contribution to business objectivesbusiness objectives
• Integrates various types of dataIntegrates various types of data
• Demonstrates alignment between programs, Demonstrates alignment between programs, strategic objectives, and operating goalsstrategic objectives, and operating goals
Micro Level Scorecard
0
1
2
3
4
0
1
2
3
0
1
2
3
4
5
0
1
2
3
4
Macro Level Scorecard
Seven Categories of DataSeven Categories of Data
•Indicators
•Reaction and Planned Action
•Learning
•Application
•Business Impact
•ROI
•Intangibles
Scorecard ReportingScorecard ReportingFirst UniversityFirst University
Key Implementation ActionsKey Implementation Actions
•Determine/establish responsibilities
•Develop skills/knowledge with ROI
•Develop transition/implementation plan
•Conduct ROI studies
•Prepare/revise evaluation policy/procedures/ guidelines
•Train/brief managers on the ROI Process
•Communicate progress/results
• Plan for evaluation early in the process
• Build evaluation into the process• Share the responsibilities
for evaluation• Require participants to
conduct major steps
• Use short-cut methods for major steps
• Use sampling to select the most appropriate programs for ROI analysis
Cost-Saving ApproachesCost-Saving Approaches
• Use estimates in the collection and analysis of data
• Develop internal capability to implement the ROI process
• Streamline the reporting process
• Utilize web-based software to reduce time
Cost-Saving ApproachesCost-Saving Approaches
Recommended TargetsRecommended Targets
Level 1 - Reaction
Level 2 - Learning
Level 3 - Application
Level 4 – Business Impact
Level 5 - ROI
100%
60%
30%
10%
5%
CriteriaCriteria
• Expected life cycle of the program
• The importance of the program in meeting the organization’s goals
• Cost of the program
• Visibility of the program
• The size of the target audience
• Extent of management interest
What Happens If We Do What Happens If We Do Nothing?Nothing?
•Budget?
•Influence?
•Support?
•Other Issues?
ROI RealityROI Reality
• ROI information is desired by clients/executives
• The ROI process provides a balanced, credible approach with six types of data
• All types of organizations are routinely using the ROI process
• The ROI process can be implemented without draining resources
• The ROI process is a long-term goal for many organizations
Status of Major Status of Major Learning and Learning and Development Development
MetricsMetrics
In In TrainingTraining Magazine’s Magazine’s 2003 Top 100 survey, 2003 Top 100 survey, 92% measure through 92% measure through Business Impact and Business Impact and 67% measure through 67% measure through
ROIROI
Program Analysis DominatesProgram Analysis Dominates
Satisfaction of Training
Participants
Impact of Training on
Performance
ROI of Training
80%88%
27%
78%
11%
63%
Organizations Utilizing Specific L&D Metrics
Organizations Wanting Specific L&D Metrics
ROI in training is a fast ROI in training is a fast growing metric . . . . 78% growing metric . . . . 78% of companies have ROI of companies have ROI
on their wish liston their wish list
Corporate Leadership Council 2002
Sample of Sample of Published ROI Published ROI
StudiesStudies