1

Training 2014

Session 501February 4, 2014

10:45 a.m. – 11:45 a.m.

ROI Basics

Learning Objectives

••••

Explain why ROI is needed todayIdentify the five levels of evaluationIdentify ten steps in the ROI MethodologyDetermine where and when to use ROI

Speaker Name: Patti Phillips, Ph.D.Company: ROI Institute, Inc.Phone: 205 678 8101Email: patti@roiinstitute.netWebsite: www.roiinstitute.net

2

The ROI calculation is simple.

ROI = X 100

Program BenefitsBCR = Program Costs

Net Program Benefits*Program Costs

****

BCR = =$750,000$425,000

ROI = X 100 =$750,000-$425,000

$425,000

*Net program benefits = Program Benefits – Program Costs

What makes a good ROI?

••••

Set the value at the same level as other investments – 15%Set slightly above other investments – 25%Set at break even – 0%Set at client expectations

What is positive about a negative ROI?

3

To make ROI work, five pieces come together to complete the ROI puzzle.

EVALUATION FRAMEWORK

The five-level evaluation framework* categorizes data to ensure questions important to allstakeholders are answered.

0

1

2

3

4

5

Level

Inputs andIndicators

Reaction andPlanned Action

Learning

Application

Business Impact

ROI

Measurement Focus

Measures of investment in thetraining program

Measures participant satisfactionwith the program or process andcaptures planned actions

Measures changes in knowledge,skills, and attitudes

Measures changes in performanceor action

Measures changes in key businessmeasures

Compares the program benefits tothe costs

Key Question

How many people attended the program? How manyhours of training? How many offerings? What are thecosts per person?

Is the program or process relevant, important, useful,and helpful to the participant and the jobenvironment?

Did participants increase or enhance knowledge,skills, or perceptions? Do they understand theinformation shared? Do they have the confidence todo what they need to do?

Are participants applying theknowledge/skills/information? If yes, what issupporting them? If no, why not?

How does the application impact output, quality, cost,time, and satisfaction? How do we know it was theprogram that caused this impact?

Do the monetary benefits of the program exceed theinvestment in the program?

EvaluationFramework

A ProcessModel

OperatingStandards or

GuidingPrinciples

CaseApplication and

Practice

ImplementationStrategy

4

The evaluation framework serves roles beyond categorizing evaluation data. The framework alsoguides the development of program objectives and ensures stakeholder needs are clearly identified.

•

•

•

By beginning the needs assessment at the highest level of need, an appropriate solution canbe identified. This is the first step toward aligning your programs with business results.

By developing program objectives at each level,the program can be positioned for success.Program objectives represent the measures to betaken during the evaluation process, thereby,eliminating the guesswork when a seniorexecutive says “Show Me the Money!” This isyour second step toward aligning your programswith business results.

By evaluating at each of the five levels, resultsimportant to all stakeholders are reported andbusiness impact and ROI can be connected to what people do with what they learned.Isolating program effect on results during the evaluation process is the third step towardlinking your programs with business results.

Matching Objectives with LevelsAn Exercise

Objective: Participants will demonstrate their knowledge of the levels of evaluation by correctlyidentifying the level to which each objective should be measured within 3 seconds from the time theobjective appears on the screen.

Objective

100 customers will request additional information from the website within twoweeks of the promotional briefing.

Given five steps to communicate effectively, participants will rank them inorder of application.

Participants will implement the change process within 3 months of completingthe course.

Within two weeks of the conferences, exhibitors will follow up with ten qualifiedprospects that visited their booth.

Six months after the program, grievances will go down on average 7 permonth due to the program.

Given benefits of $250,000 and costs of $150,000, the ROI will be 67%.

Level

________

________

________

________

________

________

Business Alignment

Start Here End Here

Payoff Needs 5 ROI Objective 5

Business Needs 4 Impact Objectives 4 Impact

Performance Needs 3 Application Objectives 3 Application

Learning Needs 2 Learning Objectives 2 Learning

Preference Needs 1 Reaction Objectives 1 Reaction

0 0Input Objectives

Program

InitialAnalysis

Input Needs

Business Alignment and Forecasting

Inputs

The ROI Process Model

Sample Linkage Between Needs, Objectives, and Evaluation

Level

5

4

3

2

1

Needs

$1.6 million costs due to falselyrejected syringes

False rejects is a problem; falserejects defined as syringes rejectedwhen they are actually usable.

Inspectors are incorrectly identifyingsyringes as unacceptable

Deficiency in skills to recognizeunacceptable syringes

One-day workshop and introduction tonew job aid

•

•

••

•

•

•

•

Objectives

25% ROI

Reduce the number of falserejects by 10 percent within sixmonths

Follow the five step processduring 100% of inspectionsUtilize job aid as neededIdentify barriers to following thefive step process

Understand five-step process andjob aidKnow the difference betweenacceptable and unacceptablesyringesUnderstand consequences ofincorrectly categorizing syringes

Program receives favorable ratingof 4 out of 5 on:

o Relevance of workshopcontent and job aid

o Importance of followingthe five-step inspectionprocess

o Intent to use job aidduring inspections

o Other measuresimportant to design anddelivery of content

•

•

•

•

•

•

•

•

•

Evaluation

Program costs compared tomonetary benefits of programMonetary benefit determined bycosts savings of reduced numberof false rejects

Monitor false rejects for sixmonthsParticipant estimates used forisolation

Follow-up questionnaire toparticipants to check frequency ofskill application and barriers threemonths after the training.Unscheduled audits over sixmonths

Demonstrate ability to identifyacceptable and unacceptablesyringesIndicate knowledge andunderstanding by completinglearning assessment

Reaction questionnaireadministered at the end of theworkshop

Sample Linkage Between Needs, Objectives, and Evaluation

Level5

4

3

2

NeedsThousands each month due toexcessive, unproductivemeetings.

Team spends too much time inmeetings. Too many people, toomuch time, too many meetings.

Managers and supervisors arenot managing their meetings.Too many people attendmeetings; no agenda; no follow-up; no structure; no discussionmanagement

Managers and supervisors needto develop meeting managementskills; learning is important, butdoing (Level 3) is moreimportant.

Objectives25% ROI

Three months after the program, there will be areduction in cost due to time in meetings asmeasured by:

• Reduced number of meetings

• Reduced number of people in meetings

• Reduced length of meetings

• Increase in productive use of time due tofewer meetings

Upon returning to the job, participants will:• Develop a detailed agenda outlining the

specific topics to be covered for 100% ofmeetings.

• Establish meeting ground rules at thebeginning of 100% of meetings.

• Follow up on meeting action items withinthree days following 100% of meetings.

During the workshop, participants will demonstratetheir ability to:• Develop a meeting profile• Develop a meeting agenda• Select the appropriate participants

•••

Manage discussionsHandle conflictManage follow-up activities

•

•

•

•

•

•••

EvaluationProgram costs compared to monetary benefits ofprogram = $123,008.Monetary benefit determined by time savingsfrom reduction in number of people in meetings,number of meetings, and length of meetings.Time value based on average salary andbenefits of time of those in meetings =$715,327.65.

Time in meetings, number of meetings, andnumber of people in meetings captured bymeeting leaders who attending the training. Ameeting profile was developed by participantsduring the workshop and then again threemonths post training.Isolating technique is estimation due to the factthat control groups were not feasible, and timeand number of meetings had not been trackedhistorically. An 81% confidence factor was usedto adjust for error of the estimated. This is basedon respondent average.

Follow-up questionnaire administered toparticipants three months after the program todetermine achievement with objectives.

Facilitator assessment of meeting profileWritten testSkill practice observation

1 One-day workshop andintroduction to new job aid

Program receives favorable rating of 4 out of 5from 80% of participants on:• Content relevance to the job

•

•

Intent to apply content immediately uponreturn to jobOther measures important to design anddelivery of content

• Reaction questionnaire administered at the endof the workshop

$430,000 - $230,000

ISOLATEEFFECTS

OF SOLUTION

CALCULATE

THE RETURN ONINVESTMENT

CAPTURECOSTS OFSOLUTION

IDENTIFY

INTANGIBLEMEASURES

DEVELOPEVALUATIONPLANS AND

BASELINE DATA

COLLECT DATADURING

SOLUTION

IMPLEMENTATION

DATA COLLLECTIONEVALUATION PLANNING DATA ANALYSIS

INTANGIBLEBENEFITS

DEVELOP/REVIEWOBJECTIVES

OF SOLUTION

COLLECT DATA

AFTER SOLUTIONIMPLEMENTATION

3. APPLICATION/IMPLEMENTATION

4. BUSINESS IMPACT

1. SATISFACTION/REACTION

2. LEARNING

EVALUATION FRAMEWORK

Level

1. Reaction, Satisfaction& Planned Action

Measurement Focus

Measures participant satisfaction with theprogram/project and captures planned action.

2. Learning

3. Application

4. Business Impact

5. Return on Investment

Measures changes in knowledge, skills,and attitudes.

Measures implementation and changes inbehavior in the performance setting.

Measures changes in business impactvariables.

Compares benefits to the costs.

!!!!!!

HR SolutionsTraining Solutions/E-LearningMajor ProjectsTechnology SolutionsOrganization Development SolutionsChange Initiatives

CONVERTDATA TO

MONETARYVALUE

THE ROI PROCESS!Calculating the Return on Investment

of Performance Solutions

ROI Calculation - Example

ROI =Net Program/Project Benefits

Program/Project Costs

Costs of program/project $230,000

Benefits of program/project (1st year) $430,000

ROI = = 0.87 x 100 = 87%$230,000

9

OPERATING STANDARDS / GUIDING PRINCIPLES

GUIDING PRINCIPLE

1. When a higher level of evaluation is conducted, datamust be collected at lower levels.

2. When an evaluation is planned for a higher level, theprevious level of evaluation does not have to becomprehensive.

3. When collecting and analyzing data, use only themost credible sources.

4. When analyzing data, choose the most conservativealternative for calculations.

5. At least one method must be used to isolate theeffects of the solution.

6. If no improvement data are available for a populationor from a specific source, it is assumed that little or noimprovement has occurred.

7. Estimates of improvements should be adjusted for thepotential error of the estimate.

8. Extreme data items and unsupported claims shouldnot be used in ROI calculations.

9. Only the first year of benefits (annual) should be usedin the ROI analysis for short-term solutions.

10. Costs of the solution should be fully loaded for ROIanalysis.

11. Intangible measures are defined as measures thatare purposely not converted to monetary values.

12. The results from the ROI Methodology must becommunicated to all key stakeholders

MEANING

Tell the complete story of programsuccess.

Conserve resources for the higher level ofevaluations.

Use the most credible sources.

Choose most conservative alternative.

Give credit where credit is due.

Make no assumptions for non-respondents.

Adjust estimates for error.

Omit the extremes.

Report first year benefits only for short-term programs

Account for all costs.

Report intangible benefits.

Communicate and use your evaluationdata.

10

In the end, the ROI Methodology generates six types of data. These data represent the chain ofimpact that occurs as people are involved in programs, view them as relevant and important to theirjobs, acquire knowledge skill and information needed to make the program work, apply what theylearn, and as a consequence impact the business in terms of improved output, quality, cost, time, jobsatisfaction, customer satisfaction and work habits. We know that improvement is due to the programbecause we isolate the effects of the program. To ensure the benefits exceed the cost of theprogram, the ROI is calculated.

Remember: While ROI is the ultimate comparison of investment in a program and the benefits of theprogram, without the lower levels of evaluation, we cannot explain how we developed the ROI wereported. As important, without all of the data, we have no basis for program and processimprovement.

Reaction & Planned Action

Learning & Confidence

Application & Implementation

Isolate the Effects of the Program

Impact

ROIIntangible Benefits

How can you use these data?

CASE APPLICATION AND PRACTICE

The framework, process model and guiding principles set the foundation for ROI. Butwithout application, they are merely theoretical. Hundreds of organizations are puttingtheory to practice by routinely applying ROI. To make ROI work for you, however, youneed to develop your own case studies.

Program

Performance Management(Restaurant Chain)

Process Improvement Team(Apple Computer)

Skill-Based Pay(Construction Materials Firm)

Sexual HarassmentPrevention (Health CareChain)

Safety Incentive Plan (SteelCompany)

Diversity (NextelCommunications)

Retention Improvement(Financial Services)

AbsenteeismControl/Reduction Program

Stress Management Program(Electric Utility)

Executive LeadershipDevelopment (Financial)

E-Learning (Petroleum)

Executive Coaching (NortelNetworks)

Competency Development(Veteran’s HealthAdministration)

First Level LeadershipDevelopment (Auto Rental)

Key Impact Measures

A variety of measures, such as productivity, quality,time, costs, turnover, and absenteeism

Productivity and labor efficiency

Labor costs, turnover, absenteeism

Complaints, turnover, absenteeism, job satisfaction

Accident frequency rate, accident severity rates

Retention, employee satisfaction

Turnover, staffing levels, employee satisfaction

Absenteeism, customer satisfaction

Medical costs, turnover, absenteeism

Team projects, individual projects, retention

Sales

Several measures, including productivity, quality, costcontrol, and product development time

Time savings, work quality, faster response

Various measures – at least two per manager

ROI

298%

182%

805%

1052%

379%

163%

258%

882%

320%

62%

206%

788%

159%

105%7

11

Iwilldothis: Bythisdate:

IMPLEMENTATION STRATEGY

DThe last piece of the evaluation puzzle is the implementation strategy. Conducting oneor two ROI studies does not create a successful accountability practice. You need astrategy to ensure a sustainable practice.

Why does the ROI Methodology work? Because it:

••••••

Provides a balanced set of measuresFollows a step-by-step processBridges evaluation disciplinesBalances research and realityIs flexiblePresents a credible approach to developing data stakeholders need.

Initial steps you can take as you develop your ROI strategy include:

1.2.3.4.5.

Assess your readiness for ROIIdentify your stakeholders and their data needsDetermine the purpose of your evaluation practiceIdentify programs suitable for Impact and ROI evaluationDevelop capability

My Action Plan

Purpose drives the initiative to achieve outcomes.

Implementing our evaluation strategy gives us the information we need to help developour people, improve our processes, and contribute to the bottom-line!

We ensure that taxpayer dollars are allocated to the programs that directly support theagency mission.

12

Pro

gram

2

Pro

gram

3

Pro

gram

4

Pro

gram

5

Pro

gram

1

Learn to:

! Identify expectations! Select or design appropriate

strategies! Communicate and gain support

for the evaluation plan! Plan ROI studies! Manage data collection! Analyze and interpret data! Report conclusions! Make recommendations! Use the data

Resources:

! Books on measurement, evaluation, andROI• ROI Resource Center

(www.roiresourcecenter.net)• Sage Publications (www.sagepub.com)• Berrett-Koehler Publications

(www.bkconnection.com)• Pfeiffer/John Wiley

(www.pfeiffer.com)! Workshops and Webinars

• ROI Institute, Inc. (www.roiinstitute.net)• Training Magazine

(www.trainingmag.com)• ASTD (www.astd.org)• Evaluator’s Institute

(http://www.evaluatorsinstitute.com)

13

Criteria for selecting programs for evaluating programs at Level 4 and Level 5.

Think about programs you offer. Rank them 1-5 against the following criteria. Those thatresult in the highest score are candidates for ROI.

Criteria

Expensive

Long-Life Cycle

Identified through comprehensive needs assessment

Aligned to organization/mission strategy

Highly visible within the organization/community

Large target audience

Large time commitment

Senior management /client interest

Develop Capabilities

Stakeholder Data Needs

Stakeholders Data Needed Data Provided Data Gap

Dis

agre

e

Agr

ee

15

ROI Readiness:

Read each question and check off the most appropriate level of agreement.

1. My organization is considered a large organization witha wide variety of programs.

2. My department has a large budget that reflects the interest of seniormanagement.

3. My organization has a culture of measurement and is focused on establishing avariety of measures including ROI.

4. My organization is undergoing significant changes.

5. There is pressure from senior management to measure results of our function.

6. My department currently has a very low investment in measurement andevaluation.

7. My department has experienced more than one program disaster in the past.

8. My department has a new leader.

9. My team would like to be the leaders in processes that support our field.

10. The image of my department is less than satisfactory.

11. My clients are demanding that our department show bottom-line results.

12. My department competes with other functions within our organization forresources.

13. There is increased focus on linking programs from all departmentsto the strategic direction of the organization.

14. My department is a key player in change initiatives currently taking place in myorganization.

15. Our overall departmental budget is growing, and we are required to prove thevalue of our programs.

1

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

2

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

3

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

4

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

5

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

16

If you scored:

15 – 30

31 – 45

46 – 60

61 - 75

Scoring

You are not yet a candidate for ROI.

You are not a strong candidate for ROI; however, it is time to start pursing some type ofprocess.

You are a candidate for building skills to implement ROI. At this point there is no pressure toshow ROI; however, now is the perfect opportunity to improve the evaluation process.

You should already be implementing a comprehensive measurement and evaluation process,including ROI.