Embed Size (px)
MEASUREMENT STRATEGY CORE PRINCIPLES
RELEVANTCREDIBLE &
COMPELLING
SIMPLE &
CONSISTENTEFFICIENT
MEASUREMENT KEY IMPACT QUESTION
What will the
participants be
doing differently if
this programme
works?
What impact will
it have on those
around them?
What specific
business /department
KPI’s would you want
to see improve as a
result of the
programme?
What will success
look like for the
learners and the
business?
INDIVIDUAL GROUP UNIT CORPORATE
MEASUREMENT CATEGORY
PREDICTIVE MEASUREMENT
FORMATIVE MEASUREMENT
BASELINE MEASUREMENT
IN-PROCESS MEASUREMENT
RESTROSPECTIVE MEASUREMENT
KIRKPATRICK – PHILLIPS MODEL
Knowledge,
skills, & attitude
LEVEL 2LEARNING
KIRKPATRICK – PHILLIPS MODEL
LEVEL 2LEARNING
LEVEL 1REACTION
LEVEL 3BEHAVIOR
LEVEL 4RESULTS
VALUE OF INFORMATION FREQUENCY OF USEDIFFICULTY OF ASSESSMENT
Least valuable
most valuable
frequent easy
infrequent difficult
LEVEL 1 : REACTION
Evaluate trainee’s reactions tothe program:Did they like the program?Did they like the facilitators?Did they like accommodation & facilities?
Guidelines for evaluating reaction1. Determine what you want to find out2. Design a form that will quantify
reactions3. Encourage written comments &
suggestion4. Get 100% immediate response5. Get honest response6. Develop acceptable standards7. Measure reactions againts standards, &
take apropiate action8. Communicate reaction as appropiate
LEVEL 2 : LEARNING
Measuring learning meansdetermining one or more of thefollowing:What knowledge was learned?What skills were developed orimproved?What attitude were changed?
Guidelines for evaluating learning1. Use control group if practical2. Evaluate knowledge, skills and/or
attitude both before and after theprogram
3. Use a paper-and-pencil test to measureknowledge
4. Use performance test to measure skills5. Get 100% immediate responses6. Use the results of evaluation to take
appropriate action
LEVEL 3 : BEHAVIOR
• The frequency of applicationof new skills/ knowledge/ attitudes (on the job)
• The effectiveness of theskills/knowledge/attitudes(as applied on the job)
Guidelines for evaluating BEHAVIOR1. Use control group if practical2. Allow time for behavioral change and
application to take place3. Evaluate both before and after
program if practical4. Survey and/or interview one or more
of the following: trainees, theirimmediate supervisor, theirsubordinates, and others who oftenobserve their behavior
5. Get 100% response or a sampling6. Repeat the evaluation at appropiate
times7. Consider soct versus benefits
LEVEL 4 : RESULT
• Indicate the extent towhich you think thisprogram has influencedeach of these measures inyour work unit, department, or businesunit:
• Productivity / quality / customer response time / cost control / employeesatisfaction / customersatisfaction
Guidelines for evaluating RESULT1. Use control group if practical2. Allow time results to be achieved3. Evaluate both before and after program
if practical4. Repeat the evaluation at appropiate
times5. Consider soct versus benefits6. Be satisfied with the evidence if prrof is
not possible
MEASUREMENT OF LEARNING EFFECTIVENESSITEM INSTRUMENT ; REMARKS
Background knowledge, prior
skills, abilities
Conduct a pre-test
Self-regulation in learning
context (Toering, Elferink-
Gemser, Jonker, Van Heuvelen, &
Visscher, 2012)
Use of tools that support self-regulation; Collect Self-regulation of Learning and Self-Report
Scale prior and post learning intervention. The online form is available at
https://docs.google.com/spreadsheet/viewform?formkey=dHFwNnBiRjY1QWM0eE5ielA5X0pVd
Xc6MQ.
Learning content Analysis of learning content provided for the learner;
Learning support tools Use of auxiliary support tools online, on the device; usage by the learner; measurement of
learner engagement; learner analytics
Learning transfer Conduct post-tests, apply knowledge to specific tasks that map to learning objectives; these
tests include, quizzes, case studies, written assignments and peer review
Learner feedback Qualitative feedback and textual analysis
Learning style/preferences Learning modality preference inventory and Kiersey Temperament Inventory. It is anticipated
that the learning style has second order effects on learning. (see also (Neuhauser, 2002)).
Learning style may have stronger influence on learner engagement and attention span.
Time on task Record of tasks completed and the use of tools. This data may be collected on the device (aka
MoLE).
Instructional method Types used and the nature of the tasks
Teacher effects Controlled experiment
CAUSAL CHAIN DIAGRAM
