Class 13

Chapter 9

Evaluation of Training

Evaluation Phase

Evaluation

Objectives

Organizational

Constraints

Design

Issues

Evaluation

Strategy and

Design

Process

Measures

Outcome

Measures

• Reaction

• Learning

• Behavior

• Results

Input Process Output

Resistance to Training Evaluation

Argument Counterargument

There is nothing to evaluate. Evaluation is simply measuring

the degree to which training

objectives are achieved.

No really cares. Organization decision-makers

utilize training outcome data in

budgeting and staffing.

Evaluation is a threat to my

job.

Managers use evaluation as a tool

for improving the likelihood of a

successful intervention, not as a

pass/fail grade.

Evaluation = 2 Types

Process Evaluation

Outcome Evaluation

Purpose of Process Evaluation

Identify areas where training process

can be improved

Improved process leads to improved

outcomes

Types of Questions to Answer in a Process

Analysis (Before Implementation)

Were needs diagnosed correctly?

Were needs translated into training objectives?

Was an evaluation system designed to measure

objectives?

Was the training program designed to meet all the training

objectives?

Are the training methods appropriate for the learning

objectives?

Were the guidelines developed in the design phase used

to develop the training content, materials and processes?

Types of Questions to answer in a Process

Analysis (During Implementation)

Was the trainer and techniques well matched to the

objectives?

Were lecture portions of the training effective?

Was involvement encouraged/solicited?

Were questions used effectively?

Did the trainer conduct the various training methodologies

(case study, role-play, etc.) appropriately?

Was enough time allotted and was it used as intended?

Did trainees follow instructions?

Was there effective debriefing following exercises?

Was time allowed for questions?

Who Is Interested in the Process Data

Training

Department

Trainer: Yes, to determine what works well and what

does not.

Other trainers: Yes, to the extent that process is generalizable

Training manager Only if training is not successful or a problem

is shown for a particular trainer.

Customers

Trainees: No

Trainees’ supervisor No

Upper management: No

Outcome Evaluation

What do you evaluate at the

end of training???

Your Training Objectives

Outcome Evaluation

The KEY outcomes achieved by your training

Trainee Reactions

Learning

Transfer to the job

Organizational Results

The same as your objectives

Outcome Evaluation Design

Alignment Between TNA and Outcomes

Training Evaluation

Objectives Outcomes

Measured the

same way

TNA

Design EvaluationDetermines

Develop

Training

Objectives

Evaluation

Plan and

Design

Design Phase Activities Evaluation

Phase

Reactions

Learning

Performance

• Person

• Unit

• Org.

Reaction Questionnaires

Training reaction questionnaires do

not assess learning,

but DO assess the trainee’s

attitudes about and perception of

the training.

Reaction Questionnaires, continued

At a minimum, the following categories

should be assessed in a reaction questionnaire:

Relevance/value of training (to trainees)

Training content and materials

Trainer’s behavior/effectiveness

Facilities, activities and procedures

Reaction Questionnaires, continued

It should also provide an

overall evaluation of the training experience.

Steps to Consider in Developing a

Reaction Questionnaire – Part 1 of 2

1. Determine what you want to find out (consider training

objectives).

2. Develop a written set of questions to obtain the information.

3. Develop a scale to quantify respondents’ data.

4. Make forms anonymous so participants will feel free to respond

honestly.

5. Ask for subgroup information (e.g., young vs. old; minority vs.

non-minority).

• This could be valuable in determining effectiveness of

training for different cultures, for example, which may be

lost in an overall assessment.

Steps to Consider in Developing a

Reaction Questionnaire – Part 2 of 2

6. Allow space for “Additional Comments” in order to allow

participants the opportunity to mention things you might not have

considered.

7. Decide the best time to give the questionnaire to get the

information you want.

a. If right after training, ask someone other than the instructor

to administer and pick up the information.

b. If some time later, develop a mechanism to obtain a high

response rate (e.g., encourage the supervisor to allow

trainees to complete the questionnaire on company time).

Types of Outcomes and Examples of

Factors Affecting those Outcomes

Trainee characteristics

Trainer behaviorReactions

Trainee readiness for the course

Trainee motivation to learn

Design, materials, and content

Learning

Training’s transfer of training effectiveness

Motivational forces in the job setting

Opportunity to apply learned KSAs on the job

Job Behavior

External environment of the organization

Internal environment of the organization

Employee performance, KSAs, and needs

Organizational

Results

Examples of Assessing Objectives

Training objectives are set in the Design

Phase based on “Gaps” identified.

Training objectives are assessed in the

Evaluation Phase

Let’s look at some examples of how this is done for

reaction objectives.

Reaction Objectives - Example

At least 75% of participants will rate the following aspects of training as 4 or higher on a 5 point scale

(with 5 being the highest rating):

1. Quality of the training content

2. Relevance of the training content

3. Effectiveness of each trainer

4. Overall effectiveness

Training Content

Strongly Disagree - Strongly Agree

1 2 3 4 5Rating of

4 or 5Ave %

Quality 0% 0% 13% 40% 33% 73%

78%Relevance 0% 0% 13% 53% 33% 86%

Overall

Rating0% 0% 27% 47% 27% 74%

Table of Reaction Results

Ratings 1 = Low 5 = High

Trainer 1 2 3 4 5Rating of

4 or 5

Average

Rating

A 0% 0% 27% 41% 32% 73% 4.05

B 0% 13% 19% 36% 32% 68% 3.87

C 0% 0% 0% 40% 60% 100% 4.60

Trainer Overall Effectiveness

Financial Evaluation

Actual savings based on outcomes

Cost of performance prior to training

- Cost of performance after training

= Performance savings

- Cost of Training

= Cost Savings

Return on Investment Ratio (ROI)

= Cost Savings / Cost of Training

Cost Savings for

Grievance Reduction Training

Pretraining Post training

Management time

10 hours/grievance

10 hrs. X 63 grievances = 630

hrs.

10 hrs. X 8 grievances = 80

hrs.

Union Rep’s time

7.5 hrs/grievance

7.5 X 63 grievances = 472 ½

hrs 7.5 X 8 grievances = 60 hrs.

Total Cost

Management time

Union rep’s time

Total

630 hrs X $50 = $31,500.00

472 ½ hrs X $.= $11,812.50

$43,312.50

80 hrs. X $50 = $4,000.00

60 hrs. X $25 = $1,500.00

$5,500.00

Reduction in cost of grievances: 43,312.50 – 5,500.00 = $37,812.50

Cost of training: -32,430.00

Cost saving for the first year $ 5,382.50

Who Is Interested in the Outcome Data

Outcome Data

Reaction Learning Behavior Results

Training Department

Trainer Yes Yes Yes No

Other Trainers Perhaps Perhaps Perhaps No

Training Manager Yes Yes Yes Yes

Customers

Trainees Yes Yes Yes Perhaps

Trainees’ Supervisor Not really Only if no

transfer

Yes Yes

Management No No Perhaps Yes