Session 3 – Evaluation Approach
1. Reaction, Satisfaction
2. Learning3.
Transfer/application
4. Business Impact
5. Return on Investment
6. Intangibles
Element Description
1. Reaction Happiness with learning interventions
2. Learning Changes in knowledge, skills and attitudes
3. Application and implementation
Changes in on the job behaviors over time (competence)
4. Business Impact Changes in impact variables
5. ROI Intervention benefits in relation to cost
Intangibles Intervention benefits in relation to policy and/or strategic direction and related factors
• Skills to master:– Ability to define a planning
framework for each level:• Reaction• Learning• Transfer/application• Business
– Ability to work from the business level to the reaction level
Level Need Program Objective
Evaluation Strategy
Metric
1
2
3
4
5
Training Need Program Objectives
Evaluation Approach
Level
Business need Impact objectives
Business impact Level 4
Job performance needs
Application objectives
Transfer and application
Level 3
Competence needs
(Skills; knowledge; attitudes)
Learning objectives
Learning and competence
testing
Level 2
Preferences Satisfaction objectives
Reaction Level 1
Training Need Program Objectives
Evaluation Approach
Level
Business need Impact objectives
Business impact Level 4
Job performance needs
Application objectives
Transfer and application
Level 3
Competence needs
(Skills; knowledge; attitudes)
Learning objectives
Learning and competence
testing
Level 2
Preferences Satisfaction objectives
Reaction Level 1
Training Need Program Objectives
Evaluation Approach
Level
Business need Impact objectives
Business impact Level 4
Job performance needs
Application objectives
Transfer and application
Level 3
Competence needs
(Skills; knowledge; attitudes)
Learning objectives
Learning and competence
testing
Level 2
Preferences Satisfaction objectives
Reaction Level 1
Cognitive Affective Psychomotor
Recall Awareness Imitation - copy
Understanding Respond Manipulation – follow instructions
Application Value (understand and act)
Develop precision
Analysis Organize personal value system - talk
Articulation – combine and relate related skills
Synthesize Internalize personal value system - walk
Naturalization - automate
Evaluate
Training Need Program Objectives
Evaluation Approach
Level
Business need Impact objectives
Business impact Level 4
Job performance needs
Application objectives
Transfer and application
Level 3
Competence needs
(Skills; knowledge; attitudes)
Learning objectives
Learning and competence
testing
Level 2
Preferences Satisfaction objectives
Reaction Level 1
Disagree Agree
1 2 3 4 5 6
The content was relevant for what I required
The manual followed the lectures logically
The presentation material was of a high quality
Presentations were interesting and engaging
The lecturer know the material
The lecturer knows the subject well
The lecturer was well prepared
The venue was conducive to learning
The food was of an acceptable standard
• Wissan Motor Manufacturing Company has been rated as a very top quality, highly productive automobile manufacturer. Wissan relates much of its success on four major areas: reliable systems, problem resolution, follow-up, and continuous improvement through learning.
• One of the successful learning programs is on shop-floor supervision where teams and team leaders are taught how to:– Plan as a team– Conduct team review meetings– Have performance discussions– Implement shop floor disciplinary code
• The program is designed to improve quality, reduce costs, improve delivery, and improve safety. Delivery is defined as the percent of daily production schedule met.
• Questions for discussion• At what level could this program be evaluated? How?• What are the highest level measures?
• In response to increased claims of sexual harassment, the Department of Indiscreet Affairs, a large federal agency, developed a one-day sexual harassment workshop for supervisors and managers. After the workshop, supervisors were expected to meet with their employees and discuss the issues with them.
After attending this workshop, participants should be able to:
• Understand, interpret, and enforce the agency’s sexual harassment policy.
• Identify comments, actions, and behavior that might constitute sexual harassment.
• Reduce the number of complaints regarding sexual harassment.
• Communicate to employees the policy and expected behavior.
Questions for discussion:• At what levels could you evaluate this program?
Please explain.• How would you evaluate at Levels 2, 3, and 4?
• Accra Wholesale Brickworks, find that there production targets are not being met. The Company is midsize but established in the City. It employs a production team consisting of three foreman, that supervise approximately 8 workers each and these teams take responsibility for procurement of the raw materials, mixing, filling the moulds, drying and baking. The production function is managed by a section manager.
Decision Tree 1:
Their systems are manual and
procurement/production planning is not
possible
Can this be solved
through training?
Decide type of training and proceed
to Decision Tree 2
Decision Tree 1:
Foreman don’t know how to conduct
proper production planning and
monitoring meetings
Can this be solved
through training?
Decide type of training and proceed
to Decision Tree 2
Decision Tree 2:
The basic business need is to raise improve profits
False/True
Decide what metric will you use to show
the value of an increase in production
Decision Tree 2:
The basic business need is to raise
production
False/True
Decide what metric will you use to show
the value of an increase in production
Level Need Program Objective
Evaluation Strategy
Metric
1
2
3
4
5
EXERCISE