Embed Size (px)
Citation preview
The
Learning Research
Quiz Show
Bridging Gap between Research and Practice
Slides available at: www.is.gd/willstuff
Research
Simplicity is the Ultimate Sophistication
Leonardo da Vinci
The Decisive Dozenfor Learning Design and Learning Measurement
http://is.gd/ddResearch
1. Content
2. ExposureBaseline
3. Guiding Attention
4. Creating Correct Conceptions
5. Repetition
6. Feedback
7. Variation
Engagement & Understanding
8. Retrieval Practice
9. Context Alignment
10. SpacingRemembering
11. Persuasion
12. PerseveranceApplication
Available at: is.gd/manifesto22
The 22 Principles
http://Work-Learning.com/catalog
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Question 1How much are smile-sheet results correlated with learning outcomes?
A. High marks on smile sheets indicate that the training was likely VERY SUCCESSFUL in creating learning.
B. High marks indicate that the training was likely to be AT LEAST SOMEWHAT SUCCESSFUL in creating learning.
C. High marks on smile sheets tell us VIRTUALLY NOTHING about the success of our training programs in creating learning.
Level 1to
Level 2
r=.09
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).
A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
1990’s
Level 1to
Level 2
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.
(2008). A review and meta-analysis of the nomological network of
trainee reactions. Journal of Applied Psychology, 93, 280-295.
2000’s
www.SmileSheets.com
Question 2When designing learning, should we base our learning methods on the
preferences of our learners?
A. Learners are fairly good judges of their own learning, so we should trust their learning-design intuitions.
B. Learners are not always good judges of their own learning, so we should utilize other design imperatives.
C. Another answer would be obviously better.
We’d like to trust our learners…
But the research shows that they don’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Question 3You’ve got a 45 minute elearning program just about
developed. Then, your client says they want a 60 minute program instead. For the last 15 minutes,
what’s will be more potent, EXTRA REVIEW or EXTRA QUESTIONS?
A. Extra Review will better support remembering.
B. Extra Questions (without feedback) will better support remembering.
C. Extra Questions (with feedback) will better support remembering.
0
10
20
30
40
50
60
70
Jones, H. E. (1923-1924). Experimental studies of college teaching: The effect of examination on permanence of learning. Archives of Psychology, 10, 1-70.
60 minutesClassroom
55 minutesLecture
---5 minutesAnsweringQuestions
Retrieval Practice – Research
Retrieval Practice – Research
0
1
2
3
4
Nungester, R. J., & Duschastel, P. C. (1982). Testing versus review: Effects on retention. Journal of Educational Psychology, 74, 18-22.
Learn-Learn
Learn-Retrieve
Retrieval Practice – Research
0
10
20
30
40
50
60
Butler, A. C., & Roediger III, H. L. (2007). Testing improves long-termretention in a simulated classroom setting. European Journal of Cognitive Psychology, 19, 514-527.
NoTests
Test---No
Feedback
Test---
WithFeedback
Question 4When should we give learners feedback on quiz questions?
A. Immediately after each question.
B. Immediately after the end of the quiz.
C. After a delay of an hour or more.
D. After a delay of several days or more.
Question 5If an organization switches from
classroom training to elearning, can they expect cost savings in general?
A. Yes. eLearning is generally LESS COSTLY to develop and deploy.
B. No. eLearning is generally MORE COSTLY to develop and deploy.
C. No. eLearning generally COSTS ABOUT THE SAME as classroom training.
While the common wisdom argues that e‐learning is less costly:
“Survey data suggest that training costs across organizations remain relatively constant as training shifts from face‐to‐face to technology‐based methods.” (p. 87)
http://is.gd/TrainingResearch2012
Read it
Skimmed it
Heard of it
Never Heard of it
How many of you have read this article?
Question 6To brainstorm new ideas, what’s the
best way to support creativity?
Having groups of people brainstorm TOGETHER
Having groups of people brainstorm
AS INDIVIDUALS
Mullen, B., Johnson, C., & Salas, E. (1991). Productivity loss in brainstorming groups: A metaanalytic integration.
Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming groups: Toward the solution of a riddle.
Question 7Is it a best practice that we present learners with learning objectives?
A. YES, by presenting learners with well-designed learning objectives, we give them an ADVANCE ORGANIZER.
B. YES, by presenting learners with well-designed learning objectives, we GUIDE LEARNER ATTENTION to the most critical information.
C. NO, learning objectives are JUST ONE TOOL in the instructional-design toolbox that can accomplish the same exact thing.
Question 8Compared to NOT giving learning objectives to learners, what is the
likely result of presenting objectives targeting half of the content?
A. The content targeted by the learning objectives will be better remembered.
B. The content targeted will be better remembered, but the content not targeted will be less remembered.
C. Both the targeted and non-targeted content will be better remembered.
Rothkopf, E. Z., & Billington, M. J. (1979). Goal-guided learning from text: Inferring adescriptive processing model from inspection times and eye movements. Journal of Educational Psychology, 71, 310-327.
0
10
20
30
40
50
60
70
80
No Objectives
NotTargeted
Targeted
0
10
20
30
40
50
60
70
80
Objectives
NotTargeted
Targeted
Question 9Is it okay to present learners with learning objectives that utilize the
word “UNDERSTAND?”
A. No, “UNDERSTAND” is not measurable, so it shouldn’t be used.
B. No, “UNDERSTAND” is too general, so it shouldn’t be used.
C. Yes, “UNDERSTAND” is fine to use as long as other words in the objective are clear and specific.
Rothkopf, E. Z., & Billington, M. J. (1979). Goal-guided learning from text: Inferring adescriptive processing model from inspection times and eye movements. Journal of Educational Psychology, 71, 310-327.
0
10
20
30
40
50
60
70
80
No Objectives
NotTargeted
Targeted
0
10
20
30
40
50
60
70
80
Objectives
NotTargeted
Targeted
You will UNDERSTAND that WHEAT was once HARVESTED by HITTING it with a FLAIL.
“An objective that generally identifies the information to be learned … will produce robust effects.
Including other information (per Mager’s, 1962, definition) will not significantly help and it may hinder the effects of the objectives.”
Mager’s Criteriafor Instructional Objectives:
The performancethe learners
should be able to do.
1
The conditionsunder which they should be able to
achieve that performance.
2
The criteriafor acceptable performance.
3
Hamilton, 1985, p. 78
Hamilton, R. J. (1985). A framework for the evaluation of the effectiveness of adjunct questions and objectives. Review of Educational Research, 55, 47‐85.
Who are our objectives for?Learning
Professionals
Learners
Video on Learning Objectives:www.is.gd/LOBJ_VIDEO
Question 10In designing learning, we should take learner characteristics into account?
A. YES, We should provide different learning interactions based on learners’ different LEARNING STYLES.
B. YES, We should provide MULTIPLE TYPES of learning interactions to support all learners.
C. YES, We should provide different learning interactions based on learners differing prior knowledge.
D. We should do ALL OF THE ABOVE.
Science has proven, the learning-styles approach is not effective!
Science has proven, the learning-styles approach is not effective!
Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105‐119.
“We conclude therefore, that at present, there is no adequate evidence base to justify incorporating learning‐styles assessments into general educational practice. Thus, limited education resources would better be devoted to adopting other educational practices that have a strong evidence base, of which there are an increasing number.”
Science has proven, the learning-styles approach is not effective!
Are learning‐styles predictions validated by research?
"No. Several reviews that span decades have evaluated the literature on learning styles (e.g., Arter & Jenkins, 1979; Kampwirth& Bates, 1980; Kavale & Forness, 1987; Kavale, Hirshoren, & Forness, 1998; Pashler et al., 2009; Snider, 1992; Stahl, 1999; Tarver & Dawson, 1978), and each has drawn the conclusion that there is no viable evidence to support the theory.”
Willingham, D. T., Hughes, E. M., & Dobolyi, D. G. (2015). The scientific status of learning styles theories. Teaching of Psychology, 42(3), 266‐271.
Question 11Which of the following
statements is true?
A. People remember 10% of what they READ.
B. People remember 20% of what they HEAR.
C. People remember 30% of what they SEE.
D. People remember 50% of what they SEE and HEAR.
E. All of the above.
F. None of the above.
1020
30
50
7080
Reading Seeing Hearing Seeing &Hearing
Collaboration Doing
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to
solve problems. Cognitive Science, 13 , 145-182.
RetentionRates
“People remember 10% of what they read…”
www.Debunker.Club
www.is.gd/BadCone
Question 4 -- RevisitedWhen should we give learners feedback on quiz questions?
A. Immediately after each question.
B. Immediately after the end of the quiz.
C. After a delay of an hour or more.
D. After a delay of several days or more.
0
10
20
30
40
50
60
Butler, A. C., & Roediger III, H. L. (2008). Feedback Enhances the Positive Effects and Reduces the Negative Effects of Multiple-Choice Testing. Memory & Cognition. 36 (3), 604-616.
NoQuiz
Quiz,But No
Feedback
Immed.Feedback
After-Test
Feedback
Feedback Research
0
10
20
30
40
50
60
Brosvic, Epstein, Cook, Dihoff (2005). Efficacy of error for the correction of initiallyincorrect assumptions and of feedback for the affirmation of correct responding: Learning in the classroom. Psychological Record, 55, 401-418.
No Feedback
FeedbackImmed.
FeedbackAfter1 Day
Feedback Research
10
11
12
13
14
15
16
Results (retrieval after 3 days)
More, Arthur, J. (1969). Delay of feedback and the acquisition and retention of verbalmaterials in the classroom. Journal of Educational Psychology, 60, 339-342.
FeedbackImmed.
FeedbackAfter2.5
Hours
FeedbackAfter
1Day Feedback
After4
Days
Feedback Research
• Immediate Feedback Creates Understanding
•More Feedback Needed
•More Elaborate Feedback
•More Emotional Support
•More Important to Give Feedback on Correct Answers
Early in Learningor
Complex Material•Delayed Feedback Supports Remembering
• Less Feedback Needed
• Less Elaborate Feedback
•No/Less Emotional Support
• Less Important to Give Feedback on Correct Answers
Later in Learningor
Simple Material
www.work‐learning.com/catalog.html
Question 12Which of the following
statements is true?
A. People forget about 90% of what they learn one week after learning it.
B. People forget at a rate first discovered by Hermann Ebbinghaus.
C. People forget about half of what they learn within 24 hours.
D. All of the above.
E. None of the above.
The Truth:
Forgetting depends on many things!
www.work‐learning.com/catalog.html
I’ve learned that people will forget
what you said, people will forget what you did, but people will never forget how you made them feel.
Maya Angelou
http://quoteinvestigator.com/2014/04/06/they‐feel/
They may forget what you said — but they will
never forget how you made them feel.
Carl W. Buehner
On Forgetting
You know what's right, just do right. You don't really have to ask anybody. The truth is—right may not be expedient, it may not be profitable, but it will satisfy your soul...
Maya Angelou
https://www.youtube.com/watch?v=bxrV2J_OjGo
Rhetorical QuestionWhat did Robert Frost mean when
he wrote, “Good fences make good neighbors”?
A. Good fences make good neighbors.
B. Good fences make bad neighbors.
C. Good neighbors make good fences.
D. Some other answer?
E. He meant for us to think more deeply before jumping to conclusions.
Question 13Today, which type of research offers the most
critical information for learning design?
A. Research surveys of learner preferences.
B. Research examining learning methods and their outcomes.
C. Research on brain-based learning and neuroscience.
D. Training needs analysis, especially cognitive task analysis.
Glial Grandeur
From Researchers at the University of Edinburgh
http://is.gd/grandeur
Someday, neuroscience may have a ton to teach us about learning.
"I don't think brain science has anything to say for business practice."
"We still don't really know how the brain works."
"The state of our knowledge [of the brain] is childlike."
John MedinaNeuroscientist
June 2015
November 2012:
“Neuroscience Applied to Education: Mostly Unimpressive”
January 2014:
“I’ve often written that it’s hard to bring neuroscientific data to bear on issues in education… Hard, but not impossible.
Daniel WillinghamResearch Psychologist
Daniel Busso & Courtney Pollock:
“There is little doubt that our knowledge of the developing brain is poised to make important contributions to the lives of parents, educators and policymakers…
Some have voiced concerns about the viability of educational neuroscience, suggesting that neuroscience can inform education only indirectly…
“Others insist that neuroscience is only one small component of a multi-pronged research strategy to address educational challenges, rather than a panacea…”
Busso, D. S., & Pollack, C. (2015). No brain left behind: Consequences of neuroscience discourse for education. Learning, Media and Technology, 40(2), 168‐186.
Question 14Are we as learning professionals more susceptible
to recommendations if they mention “neuroscience” or “brain science?”
A. Yes, there is scientific evidence that people are more likely to believe arguments if they use scientific terminology.
B. Yes, there is scientific evidence that people are more likely to believe arguments if they use the term “neuroscience.”
C. No, this is an urban legend.
“Brain Science” “Neuroscience”“Several highly cited studies have shown that superfluous neuroscience information may bias the judgement of non-experts….”
“However, the idea that neuroscience is uniquely persuasive has been met with little empirical support….
Nevertheless, excitement about neuroscience is concerning because it can be exploited by unscrupulous consumer–product manufacturers.”
Busso, D. S., & Pollack, C. (2015). No brain left behind: Consequences of neuroscience discourse for education. Learning, Media and Technology, 40(2), 168‐186.
“Powerful, often self-interested, commercial forces serve as mediators between research and practice, and this raises some pressing questions for future work in the field: what does responsible [research-to practice] translation look like?”
From a well‐known elearning vendor.
Gives impression that their products are based on neuroscience.
But in fact, the spacing effect and retrieval practice findings are NOT based on neuroscience!
1992Bruce and Bahrick counted up all the articles related to the SPACING EFFECT and found 321.
1993First mention of fMRI in PsycINFO.
200110th article that mentions fMRI.
2015
“Although behavioral studies have consistently reported the spacing effect in learning, its cognitive and neural mechanisms are still not clearly elucidated.”
1974 First Mention of Retrieval Practice.
1988 Seminal research review of Retrieval Practice by Robert Bjork.
Bjork, R. A. (1988). Retrieval practice and the maintenance of knowledge. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory: Current research and issues, Vol. 1. Memory in everyday life (pp. 396‐401). Oxford, England: John Wiley.
Question 15Learners find repetition boring, therefore repetitions should be
avoided most of the time.
A. True! Repetitions kill engagement.
B. False! Repetition is not always bad.
C. False! Repetition is one of the most important learning factors.
0
10
20
30
40
Bromage, B. K., & Mayer, R. E. (1986). Quantitative and qualitative effects of repetition on learning from technical text. Journal of Educational Psychology, 78, 271‐278.
1
2
3
Repetitions Work!
0
2
4
6
8
10
% Deaths With Cancer Surgery
Begg CB, Cramer LD, Hoskins WJ, Brennan MF, 1998 "Impact of Hospital Volume on Operative Mortality for Major Cancer Surgery," JAMA 280(20): 1747–1751.
HighVolume
MediumVolume
LowVolume
Repetitions Work in On‐the‐Job Learning too!
Helps us absorb information we missed earlier
Helps us remember things we’d forgotten
Strengthens and enriches what we already know
Why is Repetition so Powerful?
Examples of Repetition
RetrievalPractice
Testing, Practice, Exercises
Simulations, Case Studies, Role Plays
Discussions, Collaboration, Dialogue
Video, Audio, Graphics, other MediaMediaDelivery
Verbatim Repetition (exact content)
Presentation Paraphrases, Rewrites, Redesigns
Stories, Examples, Demonstrations
Question 16Repetition has powerful benefits for learning, if it
isn’t rote, boring, or irrelevant. What’s the best way to support remembering with repetitions?
A. Repeat concepts fairly close in time so that learners can easily process the material.
B. Repeat concepts at fairly wide intervals to strengthen long-term memory.
C. Repeat concepts at a slight delay, getting the benefits of both short and long delays.
Krug, D., Davis, T. B., & Glover, J. A. (1990). Massed versus distributed repeated reading:A case of forgetting helping recall? Journal of Educational Psychology, 82, 366-371.
Research Example
0
10
20
30
40
ImmediateRepetition
SpacedRepetition
Research Example
Karpicke, J.D. & Roediger, H.L. (2007). Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology: Learning, Memory and Cognition, 33, 704-719.
0
10
20
30
40
50
No Spacing
Expanding Spacing
Equal Spacing
Research Example
Dellarosa, D., & Bourne, L. E. (1985). Surface form and the spacing effect. Memory & Cognition, 13, 529-537. From Experiment 1.
10
20
30
40
50
60
SinglePresentation
Non-SpacedRepetition
SpacedRepetition
Widely-Spaced
Repetition
Question 17At the end of your learning program, what’s the
best way to get learners to actually put what they’ve learned into practice in their work?
A. Have the learners make a written commitment to implement a few key goals.
B. Have the learners make a commitment to a few key goals, and have them commit to a time and place where they’ll work on the goals.
C. Have the learners make a commitment to a few key goals and inform their manager of those goals.
Setting a Goal = “I intend to Lose Weight”
Setting a Trigger =
“I intend to Walk for 30 Minutes on Monday, Wednesday, and Friday as soon as I wake up.”
IF Situation, THEN Action
Goal Setting vs. Trigger Setting
Review Article: Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta‐analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69‐119.
Reviewed 94 separate experiments and found a medium‐to‐high magnitude (d = .65) for the benefits of implementation intentions. 92 of 94 experiments showed positive results!!
Attending a workshop, self‐examination, buying organic, recycling, exercise, diet, solving law cases, taking vitamins.
www.is.gd/TriggeredActionPlanning
Slides available at: www.is.gd/willstuff
Last QuestionWhat’s your current practice in terms of the
scientific research on learning?
A. I am a ravenous reader of research-related sources.
B. I try to keep abreast of the major research findings.
C. I don’t always pay close attention.
D. I learned what I need to know in my graduate program, in other schooling, or in my work.
E. I don’t believe research is very valuable.
Maya Angelou
https://www.youtube.com/watch?v=bxrV2J_OjGo
Try to live your life in a way that you will not regret years of useless virtue and inertia and timidity. Take up the battle. Take it up. It's yours, this is your life. This is your world...
Make it a better world…
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learnwww.SmileSheets.com
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.comSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
The Learning Research Quiz Show
ISPI Massachusetts – November 2015
Slides available at: www.is.gd/willstuff
