Are we fulfilling our promise?Assessing and reporting training quality
IATEFL BESIG Annual Conference
Stuttgart, Germany
November 16, 2012
Charles Rei
Learning objectives
By the end of this session we will be able to...
1. Identify the benefits of comprehensive training assessment.
2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.
3. Identify external factors which affect assessment.
4. Use surveys, performance-based assessment, and client contact to evaluate training.
5. Identify the main elements and organization of a comprehensive training evaluation report.
TrainingPerks
• A gift to the employees
Education
• Knowledge-based
• Standardized
Goals
• Performance-based
• Personal development
Current situation• Surveys / Feedback forms• Observation• Interviews• Testing• Self-assessment• Can do statements assessment• Portfolios and projects• Regular reports• Status meetings• Cancellations and renewals• None whatsoever
Formative vs. Summative
Benefits of comprehensive assessment
• Learner accountability• Their dedication matters
• Organizational / professional development• Strengths and areas for improvement• Quality processes• Sharing best practice
• Added / demonstrated value• Client communications and relationships• Value-based pricing vs. Cost-plus pricing• Targeted sales leads (added services, target market)
THE FOUR LEVELSAssessment Criteria
Kirkpatrick’s 4 Levels - Reaction
• Enjoyment?• Priority?• Useful?• Satisfied?• Attendance• Hours• Post-lesson contact
Kirkpatrick’s 4 Levels - Learning
• Remember?• What did you learn?• Project performance• Testing• Performance-based rubrics
• Trainer / Self assessment
Kirkpatrick’s 4 Levels - Behavior
• Did you apply the lessons?
• Pre-task reflection• Models and templates• Authentic materials
Kirkpatrick’s 4 Levels - Results
• Productivity measurements
• Better relationships• Reduced stress• Misunderstanding and conflicts
• Performance appraisals
Holton’s External Elements
Holton’s External Elements
Holton’s External Elements
ASSESSMENT METHODSHow exactly should we get this information?
The 5 Ws
Surveys / Feedback• All 4 levels
• Tips for sucessful surveys• Face-to-face• Concise• Organized / Consistent• Standardized vs. Customized• Most important questions first• Use open questions sparingly• Create realistic baseline expectations
Performance-based assessment
• Learning• Transfer designPerformance Step Yes No
Welcome and thank the audience
State purpose of the presentation
Give an outline of the presentation
Use discourse markers in outline
Mention admin rules (questions, breaks)
Signal the end of the introduction
Introduce the first topic using signposting
Maintain eye contact with audience
Speak clearly
Use appropriate body language
Total
Presentation Introduction
Client contact• Externalities• Behavior and results
• Targeted trainer-learner conversations• What’s going on in your department right now?• Are there any big projects coming up?• Did that lesson on trade fairs help you in Frankfurt? What helped
the most? How was it different than what we practiced?• Have you written any emails in English this week? Can you print it
for us?
• Client check-ins
BRINGING IT TOGETHERReporting
Ten inputs
1. Needs
2. Objectives
3. Content
4. Participants
5. Schedule
6. Facilities
7. Instructor(s)
8. Training materials
9. Program coordination
10. Program evaluation
4 levels
• Reaction – customer satisfaction
• Learning – improved knowledge and skills
• Behavior – application outside the classroom
• Results – impacts on the business
Externalities
• Motivation elements• Ability elements• Environment elements
Recommendations
• What is impacting our assessment?• 10 Inputs?• External factors?
• How do we improve?• Change an input?• Influence an external
factor?
• What should we share internally?
• What should we share externally?
INTO PRACTICEWorkshopping the four levels
Audience survey - Reaction
http://vot.rs/5ab67c
Focus groups - Learning
Discuss:• Three things you learned from today’s sessions
• 4 Levels• Don‘t assume transfer (reality checks)• Learner motivation questions
• One thing you would like to hear more about• Operationaizing it. (examples)
Focus groups - Behavior• Discuss how you will apply what you have learned from
today’s sessions.• Focus group results:
• Review the feedback form• Be more proactive about asking for achievements and successes• Look for testing service (before and after)• 3 months later, what has stuck? (before and after rating)• Use the worksheet as basis
Focus groups - Results• What impact on your organization do you expect to see as
a result of today’s training sessions?
TAKING IT FROM HEREConclusion
References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human
Resource Development Quarterly, 7(1), 5-21.• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San
Francisco: Berrett-Koehler.• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatrick's Four Levels. T+D
Magazine, pp. 34-37.• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The Canadian
Learning Journal, pp. 19-21.• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon &
Schuster.• O'Toole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These Years.
Training and Development in Australia, pp. 23-25.• Praslova, L. (2010). Adaptation of Kirkpatrick's Four Level Model of Training Criteria to
Assessment of Learning Outcomes and Program Evaluation in Higher Education. Educational Assessment, Evaluation & Accountability(22), 215-225.
• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four Levels Model". Journal of Workplace Learning, 22(5), 319-331.
• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp. 46-48.
For more information / questions
Contact: Charles Rei
http://www.reicommunicationtraining.com
@Charlesrei1
Presentation and handouts at Business English Ideas
http://businessenglishideas.blogspot.de/
Ongoing conversation on the BESIG Ning
http://iatefl-besig.ning.com/
Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unreasonable
client expectation?• How confident do you feel that your organization would
outperform the competition on these questions?• How valuable would it be if your organization earned top
scores on this consistently?• Can you think of another way to ensure consistent
performance in these areas?
Thank you!
Recommended
