7/27/2019 Human Resource Development Chapter 9
1/30
Effective Training: Systems,Strategies, and Practices, 4thEdition
Chapter Nine
Evaluation of Training
P. Nick Blanchard and James W. Thacker
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-1
7/27/2019 Human Resource Development Chapter 9
2/30
Evaluation PhaseInput Process Output
EvaluationObjectives
Design Issues
OrganizationalConstraints
Evaluation
Strategyand
Design
Process
Measures
OutcomeMeasures
- Reaction- Learning- Behavior- Results
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-2
7/27/2019 Human Resource Development Chapter 9
3/30
Potential Questions to Be Addressedin a Process Analysis
(Before Training)
Part 1 of 2Were needs diagnosed correctly?What data sources were used?Was a knowledge/skill deficiency identified?
Were trainees assessed to determine their prerequisite KSAs?Were needs translated into training objectives?Were all objectives identified?Were the objectives written in a clear, appropriate manner?
Was an evaluation system designed to measure objectives?
Was the training program designed to meet all the training objectives?Was previous learning that supports or inhibits training identified?Were individual differences assessed/factored into training design?Was trainee motivation to learn assessed?
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-3
7/27/2019 Human Resource Development Chapter 9
4/30
Potential Questions to Be Addressedin a Process Analysis
(Before Training)
Part 2 of 2
What steps were taken to address trainee motivation to learn? Were processes built into training to facilitate recall and transfer?
Were steps included to call attention to key learning events? What steps are included in the training to aid trainees in symboliccoding and cognitive organization? What opportunities are included in the training to provide symbolicand behavioral practice? What actions are included in the training to ensure transfer oflearning to the job?
Are the training methods appropriate for the learning objectives?
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-4
7/27/2019 Human Resource Development Chapter 9
5/30
Potential Questions to Be Addressedin a Process Analysis
(During Training) Were the trainer, training techniques, and training/learning objectives well
matched?
Were lecture portions of the training effective?
Was involvement encouraged/solicited?Were questions used effectively?
Did the trainer conduct the various training methodologies (case study, role-play, etc.) appropriately?
Was enough time allotted?
Did the trainer use the allotted time for activities?
Was enough time allotted?Did trainees follow instructions?
Was there effective debriefing following exercises?
Did the trainer follow the training design and lesson plans?
Was enough time given for each of the requirements?Was time allowed for questions?
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-5
7/27/2019 Human Resource Development Chapter 9
6/30
Who Is Interested in the Process Data
Training Department
Trainer: Yes, to determine what works well and what does not.
Other trainers: Yes, to the extent that process is generalizable.Training manager: Only if training is not successful or a
problem is present with a particular trainer.
Customers
Trainees: NoTrainees supervisor: NoUpper management: No
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-6
7/27/2019 Human Resource Development Chapter 9
7/30
Reaction Questionnaire for the
Trainer
Part 1 of 2Please circle the number that reflects the degree to which you agree or disagree with thefollowing statements.1= Strongly disagree, 2= Disagree, 3= Neither agree nor disagree 4= Agree 5 =Stronglyagree
1. The trainer did a good job of stating the objectivesat the beginning of training .....1 2 3 4 5
2. The trainer made good use of visual aids (easel, whiteboard) when making the presentations .....1 2 3 4 5
3. The trainer was good at keeping everyone interestedin the topics ... 1 2 3 4 5
4. The trainer encouraged questions and participationfrom trainees .....1 2 3 4 5
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-7
7/27/2019 Human Resource Development Chapter 9
8/30
Reaction Questionnaire for the
Trainer
Part 2 of 25. The trainer made sure everyone understood the concepts
before moving on to the next topic..1 2 3 4 5
6. The trainer summarized important concepts beforemoving to the next module1 2 3 4 5
7. Overall, how would you rate this trainer? (Check one.)____1. Poor; I would not recommend this trainer to others.____2. Adequate; I would recommend this trainer only if no others were
available.____3. Average____4. Good; I would recommend this trainer above most others.____5. Excellent; this trainer is among the very best Ive ever worked with.
8. Additional comments:
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-8
7/27/2019 Human Resource Development Chapter 9
9/30
Steps to Consider in Developing a
Reaction Questionnaire Part 1 of 2
1. Determine what you want to find out (consider training objectives).
2. Develop a written set of questions to obtain the information.
3. Develop a scale to quantify respondents data.
4. Make forms anonymous so participants will feel free to respond honestly.
5. Ask for information useful in determining differences in reactions by
subgroups (e.g., young vs. old; minority vs. non-minority). This couldbe valuable in determining effectiveness of training for differentcultures, for example, which may be lost in an overall assessment.
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-9
7/27/2019 Human Resource Development Chapter 9
10/30
Steps to Consider in Developing a
Reaction Questionnaire Part 2 of 2
Note: Care must be taken when asking for this information. If you ask toomany questions about race, gender, age, tenure, and so on, participants
will begin to feel that they can be identified without their name on thequestionnaire.6. Allow space for Additional Comments in order to allow participants the
opportunity to mention things you might not have considered.7. Decide the best time to give the questionnaire to get the information you
want.
a. If right after training, ask someone other than the instructor toadminister and pick up the information.b. If some time later, develop a mechanism to obtain a high response
rate (e.g., encourage the supervisor to allow trainees to complete thequestionnaire on company time).
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-10
7/27/2019 Human Resource Development Chapter 9
11/30
Procedures for Developing a
Multiple-Choice Test1. Examine objectives to gain a clear understanding of the content area you
wish to test.2. Write the questions in a clear manner. Shorter is better.3. Choose alternatives to the correct response from typical errors made during
training. Make alternatives realistic.4. Do not consistently make the correct response always longer than incorrect
responses.5. Provide four options. More than four takes longer to read, and it is difficult
enough to write three reasonable alternatives along with the correctanswer.
6. Pretest items by giving the test to those expected to know the material. Askfor feedback on clarity. Note any questions that many of them get wrong.
7. Give revised items to a group of fully trained (experienced) employees anda group of untrained (inexperienced) employees. The former should scorewell and the latter should do poorly.
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-11
7/27/2019 Human Resource Development Chapter 9
12/30
Test of Knowledge Organization for
Civil Engineers 1 of 2The following list of concepts is related to road construction. Use them tofill in the appropriate blank boxes in the map. Try to fill in the boxes so thatrelated terms or concepts are clustered together. Concepts can be relatedbecause they occur at the same time, one is necessary for the other, or
one leads to the other.Each of the listed concepts is used only once. Note that some of theconcepts are already mapped for you.
Concepts:Asphalt placement Prime/tack coat StriperCompaction/rolling Rollers StripingCut/fill Signage SurveyDump truck Site access TrafficHot materials
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-12
7/27/2019 Human Resource Development Chapter 9
13/30
Test of Knowledge Organization for
Civil Engineers 2 of 2Water Trucks Compaction
earthwork
Safety
Heavy equipment
curing
finishing
safety
paver
Spray truck
Safety
Road access
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-13
7/27/2019 Human Resource Development Chapter 9
14/30
Example of an Attitudinal MeasureAttitudes Toward EmpowermentPlease indicate the degree to which you agree or disagree with the following statements.1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree4 = Agree 5 = Strongly agree
1. Empowering employees is just another way to get morework done with fewer people. (reverse scored)..1 2 3 4 5
2. Empowering of employees allows everyone tocontribute their ideas to the betterment of the company.1 2 3 4 5
3. The empowerment program has improved my relationshipwith my supervisor1 2 3 4 5
4. Empowerment has brought more meaning to my life atthis company..1 2 3 4 5
5. Empowerment interventions should be introduced in otherplants in this company..1 2 3 4 5
6. The empowerment process has been a positive influence inlabor-management relations.1 2 3 4 5
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-14
7/27/2019 Human Resource Development Chapter 9
15/30
Guidelines for Writing EffectiveQuestionnaires Part 1 of 4
2. As k one quest ion at a t ime.
Bad: Both the organizations goals and my role within theorganization are clear.
1. Write simply and c lear ly, and make the meaning obv ious .
Bad: To what extent do supervisors provide information
regarding the quality of performance of people at yourlevel?
Good: How often does your boss give you feedback on your job?
Good: The organizations goals are clear.My role within the organization is clear.
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-15
7/27/2019 Human Resource Development Chapter 9
16/30
Guidelines for Writing EffectiveQuestionnaires Part 2 of 4
Good: During the past three months how often did you receivefeedback on your work?
1 2 3 4 5not once 13 times about once 2-4 times once a
a month a week a week day or more
3. Provide disc rete respon se option s.
Bad: During the past three months how often did you receive
feedback on your work?1 2 3 4 5
rarely occasionally frequently
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-16
7/27/2019 Human Resource Development Chapter 9
17/30
Guidelines for Writing EffectiveQuestionnaires Part 3 of 4
4. Limi t the number of respon se opt ions.
Bad: What percent of the time are you sure of what your
compensation will be?1 2 3 4 5 6 7 8 9 10
010% 1120% 2130% 3140% 4150% 5160% 6170% 7180% 8190% 91100%
Good: What percent of the time are you sure of what your
compensation will be?
1 2 3 4 5
020% 2140% 4160% 6180% 81100%
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-17
7/27/2019 Human Resource Development Chapter 9
18/30
Guidelines for Writing EffectiveQuestionnaires Part 4 of 4
5. Match the respon se mode to the quest ion.
Bad: To what extent are you satisfied with your job?
1 2 3 4 5strongly disagree agree stronglydisagree agree
Good: To what extent are you satisfied with your job?
1 2 3 4 5
not at all a little bit some quite a lot very much
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-18
7/27/2019 Human Resource Development Chapter 9
19/30
Scripted Situation Item for Evaluationof a School Superintendent
The following is a scenario about a school superintendent. Read the scenario
and place an X next to the behavior you believe your superintendent would
follow.
The administrator receives a letter from a parent objecting to the content ofthe science section on reproduction. The parent strongly objects to hisdaughter having exposure to such materials and demands something bedone. The effective administrator would most likely: (check one)
____ Ask the teacher to provide handouts, materials, and curriculumcontent for review.
____ Check the science curriculum for the board-approved approach toreproduction, and compare board guidelines to course content.
____ Ask the head of the science department for his or her opinion aboutthe teachers lesson plan.
____ Check to see if the parent has made similar complaints in the past.
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-19
7/27/2019 Human Resource Development Chapter 9
20/30
Types of Outcomes and Examples ofFactors Affecting those Outcomes
Perceived match between traineeExceptions and what training provided
Reactions
Trainee readiness for the courseTrainee motivation to learnDesign, materials, and contentTrainer(s) behaviors
Learning
Transfer of trainingMotivational forces in the job settingOpportunity to applying training on the jobKSAs
Job Behavior
External environment of the organization:Economy, regulations, suppliers, etc.
Internal environment of the organization:Policies, procedures, systems
Employee performance, KSAs, and needs
Organizational
Results
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-20
7/27/2019 Human Resource Development Chapter 9
21/30
Cost Savings for Grievance ReductionTraining
Pretraining Post training
Management time(for those going to
3rdstep) 10 hours
per grievance
10 hrs. X 63 grievances = 630 hrs. 10 hrs. X 8 grievances = 80 hrs.
Union Reps time
(paid bymanagement)7.5hrs per grievance
7.5 X 63 grievances = 472 hrs 7.5 X 8 grievances = 60 hrs.
Total Cost
Management timeUnion reps time
Total
630 hrs X $50 per hr. = $31,500.00
472 hrs X $25 per hr.= $11,812.50
$43,312.50
80 hrs. X $50 per hr. = $4,000.0060 hrs. X $25 per hr. = $1,500.00
$5,500.00
Reduction in cost of grievances going to the third step 43,312.505,500.00 = $37,812.50Cost of training -32,430.00
Cost saving for the first year $ 5,382.50
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-21
7/27/2019 Human Resource Development Chapter 9
22/30
Training Investment Analysis WorkSheetPart 1 of 5
Objective:
Audience:
Returns measured over: One year:
Other:
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-22
7/27/2019 Human Resource Development Chapter 9
23/30
Training Investment Analysis WorkSheetPart 2 of 5
Part 1: Calculating the Revenue Produced by Training
Option A-Itemized Analysis
Increased sales: Additional sales per employee
X Revenues (or margin) per sale
X Number of employees
= Revenue Produced by Training
Higher Productivity: Percent increase in productivity
X Cost per employee (salary plusbenefits plus overhead)
X Number of employees
= Revenue Produced by Training
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-23
7/27/2019 Human Resource Development Chapter 9
24/30
Reduced errors: Average cost per error
X Number of errors avoided peremployee
X Number of employees
= Revenue Produced by Training
Client retention: Average revenue per client
X Number of clients retained= Revenue Produced by Training
Training Investment Analysis WorkSheetPart 3 of 5
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-24
7/27/2019 Human Resource Development Chapter 9
25/30
Training Investment Analysis WorkSheetPart 4 of 5
Employee retention: Average cost of a new employee
( training plus lost productivity)
Number of employees retained
X Revenue Produced by Training
Other: =
Total Revenue Produced by Training
Option B-Summary Analysis$ - =
Revenue Revenue Revenue
After Training Without Training Produced by Training
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-25
7/27/2019 Human Resource Development Chapter 9
26/30
Training Investment Analysis WorkSheetPart 5 of 5
Part 2: Calculating the Return
- =
Revenue Cost of Total Return
Produced Training on TrainingBy Training Investment
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-26
7/27/2019 Human Resource Development Chapter 9
27/30
Calculation of the Utility of theGrievance Training Part 1 of 2
Formula: U = (N)(T)(DT)(SDY) - C
N = 30
T = 1 year (This is probably an overly conservative estimate)DT = .2 DT = XtXu
SD (r yy)
Xt = average job performance of the trained supervisors
Xu = average job performance of the untrained supervisorsSD =standard deviation of job performance for the untrainedsupervisors
r yy = reliability of job performance measure
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-27
7/27/2019 Human Resource Development Chapter 9
28/30
Calculation of the Utility of theGrievance Training Part 2 of 2
DT is a measure of the improvement (in standard deviation units) inperformance that trained supervisors will exhibit.
SDY = $14,000 This is based on: .40 X $35,000 = $14,000
The above assumes average salary of 35,000 dollars. The .40 comesfrom the 40% rule, which is a calculation based on 40% of the averagesalary of trainees.
So based on the above information, the utilityof the training based onthis formula is:
30 X 1 X .2 X 14,000 - 32,020 = $51,980
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-28
7/27/2019 Human Resource Development Chapter 9
29/30
Who Is Interested in the OutcomeData
Outcome Data
Reaction Learning Behavior Results
Training Department
Trainer Yes Yes Yes No
Other Trainers Perhaps Perhaps Perhaps No
Training Manager Yes Yes Yes Yes
Customers
Trainees Yes Yes Yes PerhapsTrainees Supervisor Not really Only if no
transferYes Yes
Upper Management No No Perhaps Yes
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-29
7/27/2019 Human Resource Development Chapter 9
30/30
All rights reserved. No part of this publication may be reproduced,stored in a retrieval system, or transmitted, in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise,without the prior written permission of the publisher. Printed in the
United States of America.
Copyright 2010 Pearson Education, Inc.publishing as Prentice Hall