Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.comSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
Measuring eLearning to Create Virtuous Cycles of Continuous Improvement
DevLearn – September/October 2015
Bridging Gap between Research and Practice
Simplicity is the Ultimate Sophistication
Leonardo da Vinci
Engagement &Understanding
Remembering
Application
The Decisive Dozenfor Learning Design and Learning Measurement
http://is.gd/ddResearch
Baseline1. Content
2. Exposure
3. Guiding Attention
4. Creating Correct Conceptions
5. Repetition
6. Feedback
7. Variation
8. Retrieval Practice
9. Context Alignment
10. Spacing
11. Persuasion
12. Perseverance
http://is.gd/DecisiveDozen
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
How effective are your organization’s smile sheets—those that you use in your elearning?
We DO NOT USE smile sheets in our
elearning.
1
We DO NOT USE THE DATA we
collect.
2
Our smile sheets are
VERY EFFECTIVE
4
Our smile sheets are NOT VERY EFFECTIVE
3
How do you typically measure Level 2 Learning Results—in your elearning?
With items that focus on KEY TERMINOLOGY
1
With items that focus on
KEY CONCEPTS
2
With SIMULATIONSor HANDS‐ON EXERCISES
4
With REALISTICSCENARIO‐BASED
DECISIONS
3
Which of the following practices is done routinely in your organization to measure learning results?
We measure learning results a week or more after learning events.
1
We measure post‐training on‐the‐job application of what
was learned.
2
We track on‐the‐job learning.
4
We watch learners as they use our
elearning programs.
3
The Kirkpatrick Model Rules!Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
Later on the job, learners remember
what they’ve learned.
The learners get a return on their efforts.
Performance can be prompted through job aids, signage,
intuitive cues, performance
support, management, etc.
Learners can learn on-the-job through
trial & practice, insight learning, help from others,
social media, studying on their
own, etc.
© Copyright 2009-2015 Work-Learning Research, Inc.
YouTube: http://is.gd/LearningLandscape
Most Smile Sheets Not Correlatedwith Learning!
Level 1to
Level 3
r=.16
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Very Weak Relationship between Levels
Correlation between levels?
Level 1to
Level 2
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied
Psychology, 93, 280-295.
Correlation between levels?
No
Practical Significance
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
Level 1to
Level 2
r=.09
The Critical Importance Of Remembering
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
The Learning Landscape
On-the-Job Performance
Remembering
Learning
© Copyright 2009-2015 Work-Learning Research, Inc., www.work-learning.com
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
Learners
Learners
If our learners start here.
But end up here.
Have we maximized the
learning benefits?
Less Remembering
More Remembering
© Copyright 2012-2015 by Work-Learning Research, Inc. (www.work-learning.com)
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
There are many possible after-training results,
depending on:
(1) Design of the Learning(2) After-Learning Follow-up
© Copyright 2012-2015 by Work-Learning Research, Inc. (www.work-learning.com)
YouTube: http://is.gd/LearningForgettingCurves
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
© Copyright 2012-2015 by Work-Learning Research, Inc. (www.work-learning.com)
What does an end-of-course
assessment tell us?
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
© Copyright 2012-2015 by Work-Learning Research, Inc. (www.work-learning.com)
If we measure
here?BiasedMetric!!
Understanding
Remembering
Application
DelayedTests
WHERE did you Measure Learning?
91 %
Aggleton, J. P., & Waskett, L. (1999). The ability of odours to serve as state-dependent cues for real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal of Psychology, 90, 1-7.
Balsam, P. D. (1985). The functions of context in learning and performance. In P. D. Balsam & A. Tomie (Eds.) Context and Learning. Hillsdale, NJ: Erlbaum.Bjork, R. A., & Richardson-Klavehn, A. (1989). On the puzzling relationship between environmental context and human memory. In C. Izawa (Ed.) Current Issues in Cognitive Processes: The Tulane
Floweree Symposium on Cognition (pp. 313-344). Hillsdale, NJ: Erlbaum.Bouton, M. E. (1993). Context, time, and memory retrieval in the interference paradigms of Pavlovian learning. Psychological Bulletin, 114, 80-99.
Bower, G. H., Monteiro, K. P., and Gilligan, S. G. (1978). Emotional mood as context for learning and recall. Journal of Verbal Learning and Verbal Behavior, 17, 573-585.Bransford, J. D., Franks, J. J., Morris, C. D., & Stein, B. S. (1979). Some general constraints on learning and memory research. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of processing in human
memory (pp. 331-354). Hillsdale, NJ: Erlbaum.Cassaday, H. J., Bloomfield, R. E., Hayward, N. (2002). Relaxed conditions can provide memory cues in both undergraduates and primary school children. British Journal of Educational Psychology,
72(4), 531-547.Cousins, R., & Hanley, J. R. (1996). The effect of environmental context on recall and category clustering scores following relational and individual item processing: A test of the outshining
hypothesis. Memory, 4, 79-90.Dalton, P. (1993). The role of stimulus familiarity in context-dependent recognition. Memory & Cognition, 21, 223-234.
Davies, G. (1986). Context effects in episodic memory: A review. Cahiers de Psychologie Cognitive, 6, 157-174.Dijkstra, K., Kaschak, M. P., & Zwaan, R. A. (2007). Body posture facilitates retrieval of autobiographical memories. Cognition, 102, 139-149.
Dulsky, S. G. (1935). The effect of a change of background on recall and relearning. Journal of Experimental Psychology, 18, 725-740.Eich, E. (1985). Context, memory, and integrated item/context imagery. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 764-770.
Eich, E. (1995). Mood as a mediator of place dependent memory. Journal of Experimental Psychology: General, 124(3), 293-308.Eich, J. E. (1980). The cue dependent nature of state dependent retrieval. Memory and Cognition, 8, 157-173.
Fernandez, A., & Glenberg, A. M. (1985). Changing environmental context does not reliably affect memory. Memory & Cognition, 13, 333-345.Gartman, L. M., & Johnson, N. F. (1972). Massed versus distributed repetition of homographs: A test of the differential encoding hypothesis. Journal of Verbal Learning and Verbal Behavior, 11, 801-
808.Godden, D. R., and Baddeley, A. D. (1975). Context dependency in two natural environments: on land and underwater. British Journal of Psychology, 91, 99-104.
Godden, D., & Baddeley, A. (1980). When does context influence recognition memory? British Journal of Psychology, 71, 99-104.Grant, H. M., Bredahl, L. C., Clay, J., Ferrie, J., Groves, J. E., McDorman, T. A., & Dark, V. J. (1998). Context-dependent memory for meaningful material: Information for students. Applied Cognitive
Psychology, 12, 617-623. Herz, R. S. (1997). The effects of cue distinctiveness on odor-based context-dependent memory. Memory & Cognition, 25(3), 375-380.
Jacoby, L. L. (1983). Remembering the data: Analyzing interactive processes in reading. Journal of Verbal Learning and Verbal Behavior, 22, 485-508.Johnson, A. J., Miles, C. (2008). Chewing gum and context-dependent memory: The independent roles of chewing gum and mint flavour. British Journal of Psychology, 99(2), 293-306.
Marian, V., & Kaushanskaya, M. (2007). Language context guides memory content. Psychonomic Bulletin & Review, 14(5), 925-933.Marian, V., & Neisser, E. (2000). Language-dependent recall of autobiographical memories. Journal of Experimental Psychology: General, 129, 361-368.
Mead, K. M. L., & Ball, L. J. (2007). Music tonality and context-dependent recall: The influence of key change and mood mediation. European Journal of Cognitive Psychology, 19(1), 59-79.Pan, S. (1926). The influence of context upon learning and recall. Journal of Experimental Psychology, 9, 468-491.
Parker, A., & Gellatly, A. (1997). Moveable cues: A practical method for reducing context-dependent forgetting. Applied Cognitive Psychology, 11, 163-173.Prestera, G. E., Clariana, R., & Peck, A. (2005) Memory-Context Effects of Screen Color in Multiple-Choice and Fill-in Tests. Journal of Educational Multimedia and Hypermedia, 14(4), 2005, 415-436.
Riccio, D. C., Richardson, R., & Ebner, D. L. (1984). Memory retrieval deficits based upon altered contextual cues: A paradox. Psychological Bulletin, 96, 152-165.Roediger, H. L., III, & Guynn, M. J. (1996). Retrieval processes. In E. L. Bjork & R. A. Bjork (eds.), Memory (pp. 197-236). San Diego, CA: Academic Press.
Russo, R., Ward, G., Geurts, H., & Scheres, A. (1999). When unfamiliarity matters: Changing environmental context between study and test affects recognition memory for unfamiliar stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 488-499.
Schab, F. R. (1990). Odors and remembrances of things past. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 648-655.Schroers, M., Prigot, J., & Fagen, J. (2007) The effect of a salient odor context on memory retrieval in young infants. Infant Behavior & Development. 30(4), 685-689.
Smith, S. M. (1979). Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory, 5, 460-471.Smith, S. M. (1982). Enhancement of recall using multiple environmental contexts during learning. Memory & Cognition, 10, 405-412.Smith, S. M. (1984). A comparison of two techniques for reducing context-dependent forgetting. Memory & Cognition, 12, 477-482.
Smith, S. M. (1985). Background music and context-dependent memory. American Journal of Psychology, 98, 591-603.Smith, S. M. (1988). Environmental context-dependent memory. In G. M. Davies & D. M. Thomson (eds.) Memory in Context: Context in Memory (pp. 13-34), Chichester, UK: Wiley.
Smith, S. M. (1995). Mood is a component of mental context: Comment on Eich (1995). Journal of Experimental Psychology: General, 124(3), 309-310.Smith, S. M., & Rothkopf, E. Z. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction, 1, 341-358.
Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: A review and meta-analysis. Psychonomic Bulletin & Review, 8, 203-220.Smith, S. M., Glenberg, A., and Bjork, R. A. (1978). Environmental context and human memory. Memory and Cognition, 6, 342-353.
Spear, N. E. (1978). The processing of memories: Forgetting and retention. Hillsdale, NJ: Erlbaum.Thompson, L. A., Williams, K. L., L'Esperance, P. R., Cornelius, J. (2001) Context-dependent memory under stressful conditions: The case of skydiving. Human Factors, 43(4), 611-619.
Tulving, E., & Thompson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373.Vela, E. (1984). Memory as a function of environmental context. Paper presented at the 30th annual meeting of the Southwest Psychological Association, New Orleans, LA.
Weiss, W., & Margolis, G. (1954). The effect of context stimuli on learning and retention. Journal of Experimental Psychology, 48, 318-322.Wright, D. L., & Shea, C. H. (1991). Context dependencies in motor skills. Memory & Cognition, 19, 361-370.
Tested in the learning room or in a different room. Does this matter?
Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.
0
5
10
15
20
25
Tested ina
DifferentRoom
Tested inthe
LearningRoom
MeasurementLevels
Level AReal World
Level C Scenario-Based
Decisions
Level DMemorization
Questions
Level B Simulation
Level EAttendance
Level FAffiliation
Certification
Quasi-Certification
“Level C represents the last level of certification that can be considered to assess an ability to perform on the job. Level D represents the first quantum jump away from fidelity in assessment and should be
used with caution.”
Scenario‐Based Question Example
Alena wants to start a firm that helps farmers grow food organically. She has a degree in sustainable agriculture and has worked for a non‐profit organization for seven years doing similar work. She’s developed a marketing plan, a financial plan, and has found several farmers who would pay her if she went out on her own. What should Alena do first—before she tells her boss that she’s quitting to start her own firm?
A. Create a cash flow statement to determine whether her predicted income will support the business through the first year.
B. Form a group of advisors with experience in both small‐business management and agriculture.
C. Determine whether she has enough seed money and start‐up capital to get started.
D. Analyze her values and goals to ensure that the proposed business will support them.
Smile Sheets Can be Improved!
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
4.1
Compared to:
• Previous• Standard• Others
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
We’d like to trust our learners…
But the research shows that they don’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Transmogrify
Strongly AgreeAgree
Neither Agree Nor DisagreeDisagree
Strongly Disagree
We Start with Fuzzy
Adjectives
54321
Magically We Turn Adjectives
Into Numbers
We Average
Responses, Losing
More Info
3.9
4.2
3.7
4.0
4.1
4.0
4.2
3.4
We Choose One
Question and Report Results
4.1
My Journey in tryingTo create a better
Smile Sheet
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or
removed? Are we avoiding numeric averages that discourage a standards-based decision on
success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at least try
to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
Three Key Goals:
1. Are the Questions Well-Designed?
2. Are Learners Making Good Smile Sheet Decisions?
3. Are The Data we’re Getting Clear and Actionable?
Ensuring that we are focused on the science-
of-learning factors that matter!
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or
removed? Are we avoiding numeric averages that discourage a standards-based decision on
success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at least try
to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
What Most Smile Sheets
Provide
QUATERNARYGOALS
Traditional Smile Sheets
Learners rate instructors as credible and engaging?
Learners say that classroom environment was conducive
to learning?
Learners satisfied with experience?
Learners think course was well-organized?
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?
A. I’m NOT AT ALL ABLE to put the concepts into practice.
B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.
C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.
D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.
E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.
A Better Smile Sheet Question
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?
A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.
B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.
C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job.
D. I will make this a HIGH PRIORITY when I get back to my day-to-day job.
E. I will make this one of my HIGHEST PRIORITIES when I get back to my day-to-day job.
A Question About Motivation…
Traditional Smile Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
Performance‐Focused Smile
Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
OPEN TO
BIAS
© Copyright 2009 Work-Learning Research, Inc.
Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?
A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.
B. I have a BASIC FAMILIARITY with the concepts.
C. I have a SOLID UNDERSTANDING of the concepts.
D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.
E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.
Standards
Unacceptable
Unlikely
?
Acceptable
?
AcceptableUnacceptable
AcceptableSuperior
This learning event will make a significant contribution to my work, in terms of the following: SELECT AS MANY AS YOU LIKE!
A. Improving my personal productivity.
B. Increasing my ability to innovate.
C. Enabling me to collaborate more effectively.
D. Improving my management performance.
E. Enabling me to generate more revenue.
F. Enabling me to lower costs.
The Future ofLearning Measurement
Nano-Coaching CyclePerform Task
Submit Work Product
Notify Coach
Coach Reviews Work Product
Specific Feedback
Approved?
• Photo• Checklist• Text• Video• Audio• Document• Direct
Observation
Email & Dashboard
Coaching Support
• Checklists, Guidelines
• Coach the Coach
• Photo• Checklist• Text
Comment• Audio• Document• Direct
Goal Achieved!
YES NO
©2014-2015 Cognitive Advisors LLC Used With Permission
MartyRosenheck
CEO
Any Learning
Experience
Learning Record Store(LRS)
Analytics and
Reporting
+ Context, Results, and Extensions
Activity Stream
<Actor> <Verb> <Object>
Learner Played Simulation
Learner Cleaned Crankshaft
Mike HruskaPresident/CEO
©2015 Problem SolutionsUsed With Permission
xAPI
xAPI Example
www.pipelinexapi.com
Mike HruskaPresident/CEO
©2015 Problem SolutionsUsed With Permission
• HPML is an XML Schema‐based language.
• “Takes raw data as input and specifies the computations required to turn those data into measurements and assessments.”
• Provides a list of constructs, which can represent experiences, tasks, conditions, etc.
• Based as an XML Schema
• Study underway at Simulation Interoperability Standards Organization (SISO) for standard
References:• Stacy, W., Ayers, J., Freeman, J., & Haimson, C. (2006). Representing Human Performance
with Human Performance Measurement Language. Washington, DC: Aptima, Inc. • Walker, A., Tolland, M., & Webb, S. (2015) Using a Human Performance Markup Language
for Simulator‐Based Training. Woburn, MA: Aptima, Inc.
Human Performance Measurement Language (HPML)
Mike HruskaPresident/CEO
©2015 Problem SolutionsUsed With Permission
Correlating learning performance with actual performance.
Andrew Downes
Comparing two versions of a program.
Subscription Learning
Learners Subscribe or are Subscribed
Many Learning Events
Spaced Over Time
Usually Short Nuggets
Usually Relies onPush Technology
Usually Utilizesthe Spacing Effect
To Learn More: SubscriptionLearning.com
Subscription Learning can be – THEORETICALLY – a series of informational nuggets
October
Nuggets
Nuggets Nuggets
Nuggets Nuggets
Nuggets Nuggets
Nuggets
Nuggets
Subscription Learning IS MORE THAN INFORMATIONAL NUGGETS
OctoberCEO Video
Challenge Q3FeedbackReflection
Challenge Q4Challenge Q5FeedbackAssignment
Assign. Review
Self‐Assessment
Challenge Q6Challenge Q7Feedback
Trainer Video
CEO VideoReflection
Challenge Q8Assignment
Assign. ReviewChallenge Q9Feedback
Reinforcer 1
Challenge Q10Challenge Q11
FeedbackReinforcer 2
Challenge Q12Challenge Q13
FeedbackReinforcer 3
Challenge Q14Feedback
Reinforcer 4Change Effort
Welcome!Challenge Q1Challenge Q2Feedback
Subscription Learning ALSO ENABLES STEALTH EVALUATION !!!
OctoberCEO Video
Challenge Q3FeedbackReflection
Challenge Q4Challenge Q5FeedbackAssignment
Assign Review
Self‐Assessment
Challenge Q6Challenge Q7Feedback
Trainer Video
CEO VideoReflection
Challenge Q8Assignment
Assign ReviewChallenge Q9Feedback
Reinforcer 1
Challenge Q10Challenge Q11
FeedbackReinforcer 2
Challenge Q12Challenge Q13
FeedbackReinforcer 3
Challenge Q14Feedback
Reinforcer 4Change Effort
Welcome!Challenge Q1Challenge Q2Feedback
GuerrillaEvaluation
• Our goal: Creating better feedback loops.
• Gather satisfactory data at a good cost point.
• Consider taking a user-testing approach.
· With Online Meeting Tools, it’s easy, it’s almost FREE, it takes hardly any time at all.
• Sample across cohort of users.
• Moral licensing.
How many of you have EVER seen a learner
using the elearning you’ve
developed?
Julie Dirksen, AuthorDesign for How People Learn
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
© Copyright 2009-2015 Work-Learning Research, Inc.Regular
smile sheets
Immediate tests of
remembering & decision-
making
Does Job Performance
Improve?
Does Business Performance
Improve?
IR
L1-Delayed
L2-Delayed
L3-Supports
PD
RS JS PS
L1
L2Are Prompt Supports
Used? Effective?
Are Just-in-Time Learning
Supports Used?
Effective?
Are Retrieval Supports
Used? Effective?
Smile sheets
Retrieval &
Decision Making
Evaluation of Coaching & Learner Learning
Delayed smile
sheets
Delayed tests of remembering & decision-making
Do the learners actually benefit
from the learning in their work or
career?
Does the work environment
support performance?
Are Prompting Devices Used?
Effective?
On smile sheets, ask about
motivation to apply
L1
L2
L3
L4
Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation
• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.
• Training centric. Ignores prompting mechanisms & on-the-job learning.
• Ignores the role that management and the business side must play.
• Implies that higher levels are more important than lower levels.
• Ignores the causal chain from learning to remembering to performance to results.
• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.
• Pushes us to value learner ratings as predictive of learning and on-the-job performance.
“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…
[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.
[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)
http://is.gd/TrainingResearch2012
“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to
improve the value of training.” (p. 94‐95)
Tim Mooney and Rob BrinkerhoffCourageous Training:
Bold Actions for Business Results.
Final Thoughts:
We have a responsibility to build learning programs that are effective.
Valid feedback enables improvement.
We must work to get good feedback.
We should aim to create virtuous cycles of continuous improvement.
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.comSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
Measuring eLearning to Create Virtuous Cycles of Continuous Improvement
DevLearn – September/October 2015
Thank You!!