View
726
Download
0
Category
Preview:
DESCRIPTION
Citation preview
Phillips Associates
CONSTRUCT THE ULTIMATE MULTIPLE CHOICE TEST!
STOP MEASURING RECALL AND START MEASURING
APPLICATION Presented by: Ken Phillips Phillips Associates August 16, 2011
Phillips Associates 2
AGENDA ➤ Review Kirkpatrick/Phillips five level
evaluation model
➤ Examine Level 2 evaluation facts ➤ Analyze multiple choice test creation
errors in a sample Level 2 knowledge test
➤ Learn 10 tips for creating valid, scientifically sound multiple choice test questions
Phillips Associates 3
Levels of Evaluation Measurement Focus Time Frame
Level 1: Reaction Participant reaction to a learning program
Conclusion of program
Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes
Conclusion of program or within 6 to 8 weeks after
Level 3: Behavior Degree to which participants applied back-on-the-job what was learned
2 to 12 months
Level 4: Results Degree to which targeted business outcomes were achieved
9 to 18 months
Level 5: ROI Degree to which monetary program benefits exceed program costs
9 to 18 months
KIRKPATRICK/PHILLIPS EVALUATION MODEL
Phillips Associates 4
% LIVE CLASSROOM VS. TECH. BASED PROGRAMS EVALUATED AT LEVELS 1 – 5*
Level 1-Reaction 2-Learning 3-Behavior 4-Results 5- ROI
Live Classroom
81%
50%
25%
14%
6%
Tech. Based
52%
43%
15%
10%
6%
*ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009
Phillips Associates 5
➤ 81% of organizations evaluate at least some learning programs at Level 2
➤ 55% of organizations view data collected as having high or very high value
*ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009
LEVEL 2 EVALUATION FACTS*
Phillips Associates 6
WHY THE DISCONNECT? ➤ Knowledge tests created are overly
simplistic
➤ Knowledge test administered too close to learning event
➤ Test items focus on recall not application
➤ The data collected is not used
Phillips Associates 7
WHATʼS THE SOLUTION ➤ Create valid, scientifically sound Level 2
knowledge tests that measure job application not just recall
➤ Include knowledge tests as part of learning program design and use for both evaluation and reinforcement
➤ Aggregate knowledge test score data and use to demonstrate program value as well as improve test item validity
Phillips Associates 8
SAMPLE LEVEL 2 MULTIPLE CHOICE
KNOWLEDGE
TEST
Phillips Associates 9
Whatʼs one thing that all 16 multiple
choice test questions have in
common?
Phillips Associates 10
They test for knowledge or recall – the
lowest level of cognitive ability
Phillips Associates 11
TEST CREATION TIP 1
➤ Test for application or understanding not
just knowledge
Phillips Associates 12
EXAMPLES ➤ What does TV stand for?
➤ What is the main function of a TV?
➤ What physical principle is used to display images on a TV?
➤ Your TV is not working properly. What is the most likely cause of the following set of symptoms:…?
Phillips Associates 13
You sent a detailed program design document to a group of SMEs to solicit their feedback, and have incorporated a number of their suggestions into the program design. This is an example of what element in the ADDIE model?
A. Analysis D. Implement
B. Design E. Evaluation
C. Development
EXAMPLE
Phillips Associates 14
10. Effective listening is defined as?
A. Attempting to understand the person speaking from his or her perspective and not your own.
B. Encouraging another person to talk.
C. Consolidating the important parts of a discussion.
D. Creating a win-win communication situation.
WHATʼS WRONG WITH THIS?
Phillips Associates 15
➤ Keep response options equivalent in
length & writing style
TEST CREATION TIP 2
Phillips Associates 16
4. A Collaborative Communication Style is also known as a?
A. Win/Win communication style. B. Low concern for self and high concern
for other communication style. C. Engaging and outgoing communication
style. D. Intermediate concern for self and high
concern for other communication style.
WHATʼS WRONG WITH THIS?
Phillips Associates 17
➤ Question should not reveal correct
answer
TEST CREATION TIP 3
Phillips Associates 18
4. A Collaborative Communication Style is also known as a(n)?
A. Win/Win communication style. B. Low concern for self and high concern
for other communication style. C. Engaging and outgoing communication
style. D. Intermediate concern for self and high
concern for other communication style.
EXAMPLE
Phillips Associates 19
14. What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another?
A. Arguing
B. Interrupting
C. Summarizing
D. Initiating
WHATʼS WRONG WITH THIS?
Phillips Associates 20
➤ All response choices must be plausible
TEST CREATION TIP 4
Phillips Associates 21
6. A benefit of being an effective listener is?
A. Facilitates getting to know another person better.
B. Increases the likelihood of having a mutual exchange.
C. Helps another person to get to know himself or herself better.
D. All the above.
WHATʼS WRONG WITH THIS?
Phillips Associates 22
➤ Avoid “All the above/None of the above”
TEST CREATION TIP 5
Phillips Associates 23
12. Which is not one of the four communication styles?
A. Intuitive
B. Avoiding
C. Aggressive
D. Collaborative
WHATʼS WRONG WITH THIS?
Phillips Associates 24
➤ Write questions same way material taught
TEST CREATION TIP 6
Phillips Associates 25
9. A perception check is a? A. Technique for moving a conversation
from the objective, where the focus is on facts, to the subjective, where the focus is on feelings.
B. Technique for pulling together the important parts of one long conversation.
C. Technique for helping someone to express his or her real self.
D. Technique for conveying interest in what another person is saying.
WHATʼS WRONG WITH THIS?
Phillips Associates 26
➤ Include central idea and most of phrasing in
stem
TEST CREATION TIP 7
Phillips Associates 27
9. A Perception Check communication technique is used to? A. Move a conversation from the
objective to the subjective. B. Pull together the important parts of
one long conversation. C. Help someone express his or her real
self. D. Convey interest in what another
person is saying.
EXAMPLE
Phillips Associates 28
WHATʼS WRONG WITH THIS? 2. ____________________ are used to report
specific, observable actions or details in an objective, non-evaluative manner.
A. Behavior descriptions
B. Open probes
C. Evaluative statements
D. Direct statements
Phillips Associates 29
➤ Write stem as a question – responses can
finish question, but donʼt use fill-in-the-
blank
TEST CREATION TIP 8
Phillips Associates 30
WHATʼS WRONG WITH THIS? 1. Interpersonal communication is often
rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers?
A. Environmental, Interpersonal and Intrapersonal
B. Environmental, Interpersonal and Cultural C. Structural, Interpersonal and Cultural
D. Organizational, Intrapersonal and Structural
Phillips Associates 31
➤ Eliminate excessive verbiage or irrelevant information from the stem
TEST CREATION TIP 9
Phillips Associates 32
EXAMPLE 1. What are the three categories of
communication barriers?
A. Environmental, Interpersonal and Intrapersonal
B. Environmental, Interpersonal and Cultural
C. Structural, Interpersonal and Cultural
D. Organizational, Intrapersonal and Structural
Phillips Associates 33
➤ Spread correct answers across A – D
TEST CREATION TIP 10
Phillips Associates 34
➤ Develop Level 2 Multiple Choice Tests that are:
• Valid
• Fair to the learner
• Fair to the organization &
• Test for understanding
OVERALL SUMMARY
Phillips Associates 35
FREE ARTICLES
Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.
Phillips, Ken, “Eight Tips on Developing Valid Level 1Evaluation Forms”, Training Today, Fall 2007,pps. 8 & 14.
Phillips Associates 36
Ken Phillips
Phillips Associates
34137 N. Wooded Glen Drive
Grayslake, Illinois 60030
(847) 231-6068
www.phillipsassociates.com
ken@phillipsassociates.com
Recommended