- 1. Are we fulfilling our promise?Assessing and reporting training qualityIATEFL BESIG Annual ConferenceStuttgart, GermanyNovember 16, 2012Charles Rei
2. Learning objectivesBy the end of this session we will be able to...1. Identify the benefits of comprehensive training assessment.2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.3. Identify external factors which affect assessment.4. Use surveys, performance-based assessment, and client contact to evaluate training.5. Identify the main elements and organization of a comprehensive training evaluation report. 3. Training Perks A gift to the employees Knowledge- Education based Standardized Performance- based Goals Personal development 4. Current situation Surveys / Feedback forms Observation Interviews Formative Testing vs. Self-assessment Summative Can do statements assessment Portfolios and projects Regular reports Status meetings Cancellations and renewals None whatsoever 5. Benefits of comprehensive assessment Learner accountability Their dedication matters Organizational / professional development Strengths and areas for improvement Quality processes Sharing best practice Added / demonstrated value Client communications and relationships Value-based pricing vs. Cost-plus pricing Targeted sales leads (added services, target market) 6. THE FOUR LEVELSAssessment Criteria 7. Kirkpatricks 4 Levels - Reaction Enjoyment? Priority? Useful? Satisfied? Attendance Hours Post-lesson contact 8. Kirkpatricks 4 Levels - Learning Remember? What did you learn? Project performance Testing Performance-basedrubrics Trainer / Selfassessment 9. Kirkpatricks 4 Levels - Behavior Did you apply thelessons? Pre-task reflection Models and templates Authentic materials 10. Kirkpatricks 4 Levels - Results Productivitymeasurements Better relationships Reduced stress Misunderstanding andconflicts Performanceappraisals 11. Holtons External Elements 12. Holtons External Elements 13. Holtons External Elements 14. ASSESSMENT METHODSHow exactly should we get this information? 15. The 5 Ws 16. Surveys / Feedback All 4 levels Tips for sucessful surveys Face-to-face Concise Organized / Consistent Standardized vs. Customized Most important questions first Use open questions sparingly Create realistic baseline expectations 17. Performance-based assessmentPresentation Introduction LearningPerformance StepYes No Transfer designWelcome and thank the audienceState purpose of the presentationGive an outline of the presentationUse discourse markers in outlineMention admin rules (questions, breaks)Signal the end of the introductionIntroduce the first topic using signpostingMaintain eye contact with audienceSpeak clearlyUse appropriate body language Total 18. Client contact Externalities Behavior and results Targeted trainer-learner conversations Whats going on in your department right now? Are there any big projects coming up? Did that lesson on trade fairs help you in Frankfurt? What helped the most? How was it different than what we practiced? Have you written any emails in English this week? Can you print it for us? Client check-ins 19. BRINGING IT TOGETHERReporting 20. Ten inputs1.Needs2.Objectives3.Content4.Participants5.Schedule6.Facilities7.Instructor(s)8.Training materials9.Program coordination10. Program evaluation 21. 4 levels Reaction customersatisfaction Learning improvedknowledge and skills Behavior applicationoutside the classroom Results impacts onthe business 22. Externalities Motivation elements Ability elements Environment elements 23. Recommendations What is impacting our assessment? 10 Inputs? External factors? How do we improve? Change an input? Influence an externalfactor? What should we shareinternally? What should we shareexternally? 24. INTO PRACTICEWorkshopping the four levels 25. Audience survey - Reactionhttp://vot.rs/5ab67c 26. Focus groups - LearningDiscuss: Three things you learned from todays sessions 4 Levels Dont assume transfer (reality checks) Learner motivation questions One thing you would like to hear more about Operationaizing it. (examples) 27. Focus groups - Behavior Discuss how you will apply what you have learned fromtodays sessions. Focus group results: Review the feedback form Be more proactive about asking for achievements and successes Look for testing service (before and after) 3 months later, what has stuck? (before and after rating) Use the worksheet as basis 28. Focus groups - Results What impact on your organization do you expect to see as a result of todays training sessions? 29. TAKING IT FROM HEREConclusion 30. References Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. HumanResource Development Quarterly, 7(1), 5-21. Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. SanFrancisco: Berrett-Koehler. Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatricks Four Levels. T+DMagazine, pp. 34-37. Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. TheCanadian Learning Journal, pp. 19-21. Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon& Schuster. OToole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All TheseYears. Training and Development in Australia, pp. 23-25. Praslova, L. (2010). Adaptation of Kirkpatricks Four Level Model of TrainingCriteria to Assessment of Learning Outcomes and Program Evaluation in HigherEducation. Educational Assessment, Evaluation & Accountability(22), 215-225. Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "FourLevels Model". Journal of Workplace Learning, 22(5), 319-331. Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp.46-48. 31. For more information / questionsContact:Charles Reicharles.email@example.com://www.reicommunicationtraining.com@Charlesrei1Presentation and handouts at Business English Ideashttp://businessenglishideas.blogspot.de/Ongoing conversation on the BESIG Ning http://iatefl-besig.ning.com/ 32. Quality processes If you were the client, would you care about these issues? Do any of these questions represent an unreasonableclient expectation? How confident do you feel that your organization wouldoutperform the competition on these questions? How valuable would it be if your organization earned topscores on this consistently? Can you think of another way to ensure consistentperformance in these areas?Thank you!