Return on Investment:Training and DevelopmentSession 1ROI and Evaluation

ObjectivesAt the completion of this unit, students will be able to:Discuss the importance of determining ROI in training environments.Identify stakeholders and develop questions to determine their views and needs.Write training goals and observable objectives based on stakeholders needs.

ObjectivesDescribe the importance of the assessment of learning for ROI.Recognize the differences between short-term and long-term assessment.Identify items that belong on a training budget sheet to determine ROI.Calculate expenses and ROI.Write a report on training that includes all relevant data.

Return on InvestmentRooted in manufacturing.Advanced to banking, health care, non-profit, public and education sectors.Part of quality and efficiency methodologies.

ROI Is Used To:Quantify the effectiveness of training.Manage the training budget.Provide evidence to management and other stakeholders.Build trust and respect for ourselves and our unit.Earn the ears of senior management.Identify areas for improvement.Provide data requested by senior management.Keep our jobs.

ModelsBenefit/Cost Ratio

ROI (%)

Why is ROI important to you?

Evaluation Levels1. Reaction and Planned Action2. Learning3. Application and Implementation4. Business Impact5. Return on Investment

StakeholdersWho are they?Anyone who pays for, participates in, benefits from, or has decision-making responsibility for HRD and training. What should they know?The need for training.The results from training.The costs of training.

Needs Assessment and Task AnalysisNeeds assessment: identifies gaps between what is and what should be in the organization.Task analysis: investigates the specific skill(s), knowledge or attitudes and at which point there is a breakdown in use or performance. If the analysis uncovers a lack of knowledge, skills or attitude, then training is required. If it reveals faulty equipment, poor work conditions or lack of incentive, then another solution is needed.

End of Session 1

Return on Investment: Training and DevelopmentSession 2Goals, Objectives, Assessment

Goals Organizational GoalsThese goals are typically targeted toward the organizational level and include such things as productivity, net income, inventory and cash cycles, and customer satisfaction.Learning GoalsThese goals typically focus on the individuals in the organization and their performance, knowledge and skills.

Writing ObjectivesEasy as A, B, C, D

Audience: Who?Behavior: What do they do?Condition: What is the setting and method of evaluation?Degree: Measurement to be met.

Example ObjectiveAudienceBehaviorConditionDegreeAt the completion of the course the learner will be able to score 85% or better on a written multiple choice 25 question test

Example VerbsVerbs to Use:DiscussExplainDemonstrateIdentify ListPerformCompare/ContrastScore (on an assessment)Verbs to AvoidUnderstandComprehendKnowDo

Objectives in ROICourse objectives: What is it that the learners will be able to do upon course completion?Application objectives: What goals do the HRD/training practitioner have for the application of the skills gained in training?Impact objectives: What effect will accomplishing the application objectives have on the organization?

Example ObjectivesCourse objective: Learners will be able to make 15 entries in a customer database in 15 minutes with no more than 1 error.

Application objective: Learners will be able to reduce the data entry error rate by 50 percent over the next 6 months.

Impact objective: Employee time spent correcting database errors is reduced by 25 percent from last years rate.Increasing Scope

Learning AssessmentAlignment between assessment and objective is paramount.Easy to do with well-written objectives.Cognitive, psychomotor and affective domains are assessed in different ways.Cognitive: written and oral tests; discussion; compare/contrast exercises.Psychomotor: demonstration; performance.Affective: longitudinal observation and/or discussion.Be sure to use the right type of assessment for each objective of the learning event.

End of Session 2

* *Assessment: In the context of training and development, assessment is of the persons knowledge, skills and abilities related to the training.

Evaluation: In the context of training and development, evaluation is the evaluation of the training program for its ability to meet the goals and objectives.

These terms, while similar, have distinct meanings in the context of this module.***Calculating the return on investment, or ROI, began in the manufacturing field where it was easy to measure time to complete a task and the number of widgets produced. ROI then moved to the banking field, where it was used regularly. Today, ROI is becoming a part of every area of business.

We calculate ROI for a number of reasons:To justify the training budget (and to make the case to maintain or increase training dollars).To determine the effectiveness of training:The method used, use of time for trainer and employee, usefulness of information, perceived quality of the training.Was there a change in time, cost or behavior as an outcome of the training.Provide evidence to management and other stakeholders.Because you were directed to.

A number of process and quality systems include ROI as an important facet. These include Total Quality Management (TQM); Continuous Process Improvement (CPI); Six Sigma; and Baldrige.

Ask students to brainstorm, Why do we use ROI? Then review points on this slide.

Quantify the effectiveness of training: showing what effect training can and does have on dollars, time, production, quality and efficiency. Manage the training budget: helps to identify, structure, and budget HRD and training activities.Provide evidence to management/stakeholders: demonstrates results to management and stakeholders that address their concerns.Build trust and respect for ourselves and our unit: provides solid, usually objective, evidence of the value that HRD and training adds to the organization.Earn the ears of senior management: helps HR and HRD professionals earn a place at the table with senior and executive management.Identify areas for improvement: with ROI information, you can identify where to focus your energies to have maximum effect on training and development improvements.Provide data to senior management: to help them make more informed decisions.Keep our jobs: by justifying and illustrating our value to the organization.

**Benefit/Cost Ratio: This is a basic definition of ROI. This is a quantification of the relation between the benefits of a program and its costs. When BCR is greater than one, the benefits outweigh the costs and the program is considered a success. When BCR is less than one, the cost exceeds the benefits and demonstrates that improvements or changes probably need to be made to justify continuation of the program.

Another useful and often used definition/formula expresses the ROI as the percentage return on the costs incurred. This has the advantage of speaking to many investors and stakeholders in their language.

A result greater than 100% means that the program has a net benefit after accounting for the costs involved in running it. For instance, an ROI% = 150% means that the program yields a 150% return on money invested; i.e., the program yields $1.50 for every dollar that the program costs.

A result less than 100% means the program has a net cost. This means that the program does not recoup its cost after accounting for the benefit. When this happens, there may be a hidden or social benefit that is not quantifiable, such as an increase in employee morale. In these cases, stakeholders and decision makers need to ascertain whether the scale of loss is justifiable given the money spent. A loss of 3 percent of several thousand dollars may be worth it to realize a happier workplace, but 3 percent of several million may not; there may be easier or less expensive ways to create a happier workplace. This is where ROI really becomes useful. Used properly it can be an objective method to compare the benefits, costs and returns for two or more programs.

Note See Figure 2-2 on p. 37 and throughout the Phillips text for the model. This is a widely adopted and complete model for the process of calculating ROI for training and development.

Ask students, Why is ROI important to you?

HRD practitioners and their supervisors must be able to determine if learning/change has occurred. Two main reasons are (1) to report to supervisors, and (2) for your own personal knowledge and improvement of your work or the training departments work.When people can show that what they do has value and how much value it has, organizations can make better choices. Accountability breeds success. Driving improvement with ROI may help strengthen your position.

**Kirkpatricks Four Levels of Evaluation; Phillips added ROI as the fifth.Reaction and Planned Action Frequency: each learning event.

Was the customer (trainee) satisfied with the experience? What did they like? What did they learn? Was there anything missing? Consider using a Likert rating scale for feedback. Was the facilitator knowledgeable? Was the subject interesting? Were facilities adequate for the training? Was the training scheduled at a good time? Additional comments?

Learning Frequency: pre- and post-training.

Was there a change in attitude, skills and/or knowledge? Assess learning before training, during training and after training to accurately assess learning.

Application and Implementation Frequency: pre- and post-training and particular periods after training is complete (e.g., three months, six months, one year).

This speaks to behavior. Are the trainees doing things differently at work by using the trained knowledge and skills?Pre-/post-test, observation, interview. Allow time for the change to occur. Ask employee, supervisor, subordinates for their perception of change in attitude or performance.

Business Impact Frequency: regular intervals over the calendar or fiscal year monthly or quarterly is typical.

This speaks to overall change for the business as a result of the training programthe results. Is there improved quality, improved production, decreased costs, increased job satisfaction, reduced problems or accidents, increased sales?

ROI Frequency: With each new training event or when significant changes are made to existing events.

Consider the costs of training versus the benefits of training. How did the bottom line change? Were the benefits greater than the cost? Goals and objectives are based on needs assessment, task analysis and desired outcomes. They guide the design and development of training content, assessment and evaluation.

Goals are the most general level of a desired state for the organization or the individual. They provide overarching guidelines to measure performance.***Objectives are very specific measures for an organization or an individual. In a training environment, we need to keep the organizational goals and the learning goals in mind as we write objectives for any single training session. Objectives guide the learning.

ABCD of writing objectives, using observable verbs:AudienceThe group of learners the objective is written for; the learner or the studentBehaviorThe action or observable verb which describes what the learner will be able to do after completing the instruction (e.g., describe, compare/contrast, demonstrate). For a more complete list of verbs, please see the verb list in the readings.ConditionThe tools, resources, setting the students will have and the assessment method to be used.DegreeThe standard or degree of accuracy to be considered proficient. This can be based on a normative scale, measured against a standard of performance.The ABCD method doesnt necessarily mean that the elements appear in that order in the objective. The important thing is to make sure that all the elements are included. The most difficult element--perhaps because it is the most obvious--is the condition. The condition typically takes the form of the assessment to be used. For example, suppose you dont or cant use multiple-choice tests because the organization thinks there is too much testing stress among their employees or you dont have the tests available, so you decide to use observation to assess skills. You could drop the condition portion and restate the objective as:

At the end of the course, the learner will be able to score 85 percent or better under observation from the instructor.

The scoring (or degree) might not make sense in this case, so you may change that also. Maybe something like:

At the completion of the course, the learner will be able to complete at least 8 out of 10 tasks at a satisfactory level under observation from the instructor.

Of course, this is now a slightly different objective.

Remember:A: Audience: Who is it that is going to do this? Typically this is simply the learner or student.B: Behavior : What is A going to do? Achieve a score on a test? Complete a project successfully? Build a miniature bridge that supports 45 lbs? Build a device that prevents an egg from breaking in a 30-foot fall?C: Condition: What is the device or type of assessment? A paper? A test? A creation?D: Degree: What measure or standard needs to be achieved (and if not obvious, who or what will score it)? What is defined as success? A certain score on a test? A complete paper?

These verbs can be used to help write observable/measurable objectives. Lists of verbs can be found on Blooms Taxonomy Web sites.**In addition to the objectives trainers must create and set for the learners, there are two other types of objectives that HRD and training professionals must set for themselves: application and impact objectives.

Course Objectives = Kirkpatrick Level 2 : knowledge, skills, attitudes.

Application Objectives = Kirkpatrick Level 3: actual use in the work setting.

Impact Objectives = Kirkpatrick Level 4: organization level: customer satisfaction, work atmosphere, increased productivity, etc. (Impact objectives must be achieved to reach high levels of ROI.)

See examples on the next slide.** **Content: Learning AssessmentHow to tie assessment to objectives: If you begin with well-written objectives, creating learning assessments is easy. Your objective will already specify what it is the learner should be able to do and the parameters which will indicate proficiency. Your responsibility then is to set up the opportunities for the learners to show that they have mastered the objective. This might be accomplished during the training session or after the training session via demonstrated/observed changes in the work setting.

Cognitive changes cannot be directly observed, but the results of cognitive changes can be. Therefore, they can be evaluated, like with a written or oral test. Example: How do we know if someone knows how to add? Addition is purely cognitive. We know because we ask them what 2+2 equals. If they were unable to do this before training but can do so after training, then they have learned how to add. Another example: Training is held to teach the strategic planning process. How do you know whether trainees learned anything about the strategic planning process? Testing for this might include asking them to describe the strategic planning process in detail.

Psychomotor changes are relatively straightforward. Can the learner perform the action to an acceptable level? For example, can they open the customer database and make a complete, correct entry in a timely manner?

Affective changes are VERY difficult to measure. The challenge is to somehow assess attitude. The primary way to objectively and effectively assess these types of changes are from observation, interviews, and communication with the person over a period of time. Example: Assessing an employees morale or commitment to the organization. Although learners may exhibit changes in the time directly after training, changes like these take time to really measure. Regular interaction, communication and observation of their behavior over a 3-6 month period will tell you if their attitudes changed or not. Looking at their attitude change in the first two weeks after training may yield inaccurate results, as this is known as the honeymoon period after training when learners are most likely to keep using the things they learned. This is another reason why the application objectives at the organizational level are so important. If learners learn but dont apply the learning consistently over time, application objectives may not be met.