Evidence-based thinking about learning and blogs.uw.edu/.../2012/05/Evidence-based-thinking-about- thinking…

  • Published on
    06-Jul-2018

  • View
    212

  • Download
    0

Transcript

Evidence-based thinking about learning and instruction University of Washington May 3, 2012 2 Integrating the design, building, monitoring, and improvement of learning environments; individualize learning experiences using our scale; and, ultimately, drive greater student career success. Former CLO for K12, Inc. structured use of technology, cognitive science, on-line and off-line materials for 1,700 teachers, 55k students Former Publisher and General Manager for DK Multimedia, Inc. Management consultant with McKinsey & Company Education: - Ph.D. in Electrical Engineering and Computer Science from MIT - M.D. from Harvard Medical School - M.A. in Electrical Engineering and Computer Science from MIT - M.A. in Mathematics from Oxford University - B.S. in Electrical Engineering and B.S. with Honors in Mathematics from the University of Washington Bror Saxberg Chief Learning Officer, Kaplan, Inc. 3 Kaplan University Kaplan Legal Education Kaplan Professional Education Nursing Kaplan Continuing Education KNEXT KTPA Kaplan Tutoring Kaplan Bar Review Kaplan Publishing Kaplan Higher Ed Europe Kaplan Professional Europe Kaplan Higher Ed Asia Kaplan Professional Asia Kaplan Higher Ed Australia Kaplan Professional Australia In Country Pathways China Franklyn Scholar Carrick Education Global Knowledge Solutions Kaplan education spans domains and geography Kaplan University Group Kaplan Higher Education Campuses Kaplan Test Prep Kaplan Asia Pacific Kaplan United Kingdom Kaplan Intl Colleges Global Pathways Kaplan International Colleges 4 What Our Students Told Us They Want from a Worlds Best Educator Promise Pillars Definitions We strive to make education as personalized to you as possibletailoring our courses around your individual needs. We are dedicated to getting you the results that matter in the time that matters. We move quickly with constant innovation to better meet your needs. We are here to help you achieve success at critical milestones along your educational journey. 5 What were trying to do 6 What were trying to do 7 What were trying to do 8 What were trying to do 9 What were trying to do 10 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 11 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 12 Much research to guide us Learning Events (hidden - inside students minds) Student Performance (observable -indicates knowledge) Instructional Events (in the learning environment) Knowledge Explicit: Information, Explanation, Examples, Demos Implicit: Practice tasks/activities (prompts and response) Diagnosis and feedback Explicit/Declarative/Conceptual/What Implicit/Procedural/How Knowledge Components (Procedures + Facts, Concepts, Principles, Processes) Response accuracy/errors Response fluency/speed Number of trials Amount of assistance (hints) Reasoning Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center) 13 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Supportive/Conceptual Fact Concept Process Principle 14 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Sequence of decision and action steps to perform tasks; when and how to do things Prosecuting a criminal Deciding if capital gains tax applies Supportive/Conceptual Fact Concept Process Principle 15 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Sequence of decision and action steps to perform tasks; when and how to do things Prosecuting a criminal Deciding if capital gains tax applies Supportive/Conceptual Fact Isolated, unique piece of information; one instance 52 Grosvenor Place 2+3=5 Concept Process Principle 16 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Sequence of decision and action steps to perform tasks; when and how to do things Prosecuting a criminal Deciding if capital gains tax applies Supportive/Conceptual Fact Isolated, unique piece of information; one instance 52 Grosvenor Place 2+3=5 Concept Sets of items that share common attributes, common name; multiple examples Dog Money Happiness Process Principle 17 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Sequence of decision and action steps to perform tasks; when and how to do things Prosecuting a criminal Deciding if capital gains tax applies Supportive/Conceptual Fact Isolated, unique piece of information; one instance 52 Grosvenor Place 2+3=5 Concept Sets of items that share common attributes, common name; multiple examples Dog Money Happiness Process Flow of events or procedures; how things work Workflow Chemical process Principle 18 5 types of outcomes determine TYPE of information and practice Knowledge Component Definition Example Procedure Sequence of decision and action steps to perform tasks; when and how to do things Prosecuting a criminal Deciding if capital gains tax applies Supportive/Conceptual Fact Isolated, unique piece of information; one instance 52 Grosvenor Place 2+3=5 Concept Sets of items that share common attributes, common name; multiple examples Dog Money Happiness Process Flow of events or procedures; how things work Workflow Chemical process Principle Guidelines, rules that govern, predict, explain events; relationships among concepts Supply and demand 80/20 principle Novices need structure 19 3 stages of learning determine instructional elements and sequence Stage Characteristics Implications for Instructional Design 1. 2. 3. Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition. Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007) 20 3 stages of learning determine instructional elements and sequence Stage Characteristics Implications for Instructional Design 1. Declarative Knowledge about, that, what why; Can be stated verbally; Conceptual network Conscious Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures 2. 3. Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition. Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007) 21 3 stages of learning determine instructional elements and sequence Stage Characteristics Implications for Instructional Design 1. Declarative Knowledge about, that, what why; Can be stated verbally; Conceptual network Conscious Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures 2. Procedural Knowledge how Sequence of if-thens Potential to become unconscious Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance 3. Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition. Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007) 22 3 stages of learning determine instructional elements and sequence Stage Characteristics Implications for Instructional Design 1. Declarative Knowledge about, that, what why; Can be stated verbally; Conceptual network Conscious Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures 2. Procedural Knowledge how Sequence of if-thens Potential to become unconscious Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance 3. Automated Fluency Expert Unconscious 10,000 hours Design opportunities for repeated frequent practice on the job and monitoring of speed and accuracy Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition. Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007) 23 Much research to guide us Learning Events (hidden - inside students minds) Student Performance (observable -indicates knowledge) Instructional Events (in the learning environment) Knowledge Explicit: Information, Explanation, Examples, Demos Implicit: Practice tasks/activities (prompts and response) Diagnosis and feedback Explicit/Declarative/Conceptual/What Implicit/Procedural/How Knowledge Components (Procedures + Facts, Concepts, Principles, Processes) Response accuracy/errors Response fluency/speed Number of trials Amount of assistance (hints) Reasoning Motivation Orientation/Inoculation Monitoring Diagnosis and treatment: Persuasion, Modeling, Dissonance Value beliefs Self-efficacy beliefs Attribution beliefs Mood/Emotion Behavior related to Starting Persisting Mental Effort Self-reported beliefs Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center) 24 4 beliefs influence motivation Sources: Bandura; Eccles & Wigfield; Pintrich & Schunk; Clark; Dweck Beliefs Value Self-Efficacy Attribution Mood Motivated Behavior Starting Persisting Mental Effort Learning/ Performance Practice Test Self-Efficacy EffortHigh Moderate Low Motivation Low High Performance High Low Design materials and interaction to foster positive mood, high perception of value, moderate confidence, and attribution of success and failure to effort Design system for monitoring and guidance (group and individual) 25 Much research to guide us Learning Events (hidden - inside students minds) Student Performance (observable -indicates knowledge) Instructional Events (in the learning environment) Knowledge Explicit: Information, Explanation, Examples, Demos Implicit: Practice tasks/activities (prompts and response) Diagnosis and feedback Explicit/Declarative/Conceptual/What Implicit/Procedural/How Knowledge Components (Procedures + Facts, Concepts, Principles, Processes) Response accuracy/errors Response fluency/speed Number of trials Amount of assistance (hints) Reasoning Motivation Orientation/Inoculation Monitoring Diagnosis and treatment: Persuasion, Modeling, Dissonance Value beliefs Self-efficacy beliefs Attribution beliefs Mood/Emotion Behavior related to Starting Persisting Mental Effort Self-reported beliefs Metacognition Structure Guidance Planning, Monitoring Selecting, Connecting Amount of guidance required/requested Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center) 26 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 27 Instructional design: Engineering from learning science Overviews Information Examples Practice Assessment Learning Outcomes Motivational Guidance Design Deliver Learning science strongly suggests an order to design and delivery Clark, R.E., & Feldon, D. F. (2008). GEL (Guided Experiential Learning), Adaptable Expertise and Transfer of Training. Kirscher, P.A., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41, 75-86. Knowledge Integration 28 Evidence-based instructional principles Accumulation of results from lab studies support: Structure and guidance for novices (Kirschner, Sweller, & Clark, 2006) Demonstrations and worked examples (Paas & van Merrienboer, 1994; Sweller, 2006) Practice and corrective feedback (Mathan & Koedinger, 2005) Prompted self-explanation (Aleven & Koedinger, 2002) Multimedia use that minimizes extraneous cognitive load (Mayer, 2009) Targeting beliefs (value, confidence, and attributions) and emotions (positive feelings) to influence motivation (Clark, 2004; Um et al., 2011) 29 Task-centered instruction Move from simple to increasingly difficult tasks NOT PBL sink or swim Teach everything needed for each task Fade coaching/support over time 30 Knowledge Component Presentation (Prepare) Practice/Assessment (Practice, Perform) Info Example Remember Proxy for Remember Use** Proxy for Use ** Procedure When to use; List of action and decision steps Demonstration of when and how to perform Recall when to use; Recall action and decision steps Reorder steps; Recall next or missing steps Decide when to use; Perform the steps (actions and decisions) Critique performance or output of actions and decisions Supportive KnowledgeFact * Statement of fact Statement of fact Recall fact Recognize fact when presented with distractors Recall fact in task context Concepts List of defining attributes Examples; Non-examples List defining attributes verbally or in writing Recognize defining attributes when presented with distractors Classify, identify or generate examples and non-examples Critique someone elses identification or generation of examples Process/ System List of phases, events and causes at each phase Examples; simulations of phases, events, and causes Recall phases, events, and causes Recognize phases, events, and causes; Recall missing phases, events, and causes Identify causes of faults in a process; Predict events in a process Critique someone elses description of causes or prediction of events in a process Principle (cause and effect relationship) Statement of cause and effect relationship Examples, demonstration, simulation of cause and effect relationship Recall the principle Recognize the principle; Recall missing elements of the principle Decide if principle applies; Predict an effect; Apply principle to solve a problem, explain a phenomenon or make a decision Critique someone elses application of the principle to solve a problem, explain a phenomenon or make a decision Knowledge Integration Explain the interconnections among conceptual knowledge components, or the conceptual foundation of procedures, or the procedural implementation of conceptual knowledge components Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or select their reasoning about interconnections among knowledge components, for example the principle(s) that justify the application of a procedure. Knowledge Transfer Multiple and varied contexts for examples Multiple and varied contexts for practice and assessment. Opportunities for students to explain how they would use the knowledge in other contexts *Facts are concepts with single instances ** All Use and Proxy for Use Activities develop/require procedural knowledge Presentation and practice match objectives (knowledge components) 31 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 32 ID can change instructional outcomes at scale Principle Description Effect size (s.d. units) Multimedia Use relevant graphics and text to communicate content 1.5 Contiguity Integrate the text nearby the graphics on the screen avoid covering or separating integrated information 1.1 Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3 Modality Include audio narration where possible to explain graphic presentation 1.0 Redundancy Do not present words as both on-screen text and narration when graphics are present .7 Personalization Script audio in a conversational style using first and second person 1.3 Segmenting Break content down into small topic chunks that can be accessed t the learners preferred rate 1.0 Pre-training Teach important concepts and facts prior to procedures or processes 1.3 Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc. ?? Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008 33 Impact is not small! 50% 1 sd 84%! 34 Redeveloping courses at scale Read, Write, Discuss Outcomes and content not precisely aligned Limited demonstrations, worked examples, and practice General assessment rubrics High reliance on discussion boards Existing courses Prepare, Practice, Perform Outcomes and content aligned One lesson per objective Demonstrations and worked examples Practice, feedback before assessment Detailed scoring guides Less discussion/more practice Standard instructor materials Monitoring and support for motivation Redesigned courses 35 Content Design Items Prepare Practice Perform Seminar Discussion Lessons Sets Overview Course Level Outcome 1 Unit Outcome 1 Unit Outcome 2 Unit Outcome 3 Prepare 1 Practice 1 Perform 1 Prepare 2 Practice 2 Perform 2 Prepare 3 Practice 3 Perform 3 Lesson 1 Lesson 2 Lesson 3 Navigation 36 Narrated demonstrations 37 Explanation and demonstration of concepts 38 Practice with hints and feedback 39 Scenario-based practice 40 Prompted self-explanation 41 Motivation surveys 42 6 Overview Seminar Discussion Journal: A situation in your life where the guidelines for improving nonverbal communication could guard against misinterpretation. Lesson 1 Identify verbal and nonverbal elements in personal and professional situations 1-2 hrs. Lesson 2 Identify nonverbal communication principles in personal and professional situations 1-2 hrs. Lesson 3 Explain instances of effective and ineffective communication in terms of how verbal and nonverbal elements work together 2-4 hrs. UNIT 5 Review: What is Nonverbal Communication? 2. Identify nonverbal communication principles in personal and professional situations 15 UNIT 5 Review: Lesson 2 Practice http://www.youtube.com/watch?v=bg0kSIJZiRQ&feature=related PART 1: Which nonverbal communication principle is predominant in the womans reactions to her blind date? Watch Item 2 video: 21 Overview (including Survey) Seminar Discussion Explain how improving your listening skills can increase the effectiveness of your communication in the workplace and in your personal life. Lesson 1 Identify forms of nonlistening in personal and professional situations 1-2 hrs. Lesson 2 Apply the principles of mindful listening to improve the effectiveness of communication in personal and professional situations 1-2 hrs. UNIT 6 Preview: How Does Listening Enhance Our I.C.? 25 UNIT 6: Lesson 2 Practice Watch the Online Dating video. Answer the three questions, referring to the scoring guide. Online Dating After answering a question, study the Compare with Expert response. 1. From the interaction does it seem to you that Chriss mom is actively listening during the first third of this conversation? Why or why not? 2. Apply the principles of mindful listening to improve communication effectiveness 29 Agenda Minutes Opening 5 P Student Questions 10 P Review Unit 5 10 P View Unit 6 25 P Preview Unit 7 5 P Wrap Up 5 P Poll Question How many of you still have questions? Post your questions in Course Questions discussion board q Yes q No (Link in Course Home menu) Wrap Up Instructor seminar materials more standardized and aligned with online content 43 = Unit 1 = Unit 3 = Unit 6 Provide materials to monitor and support motivation 44 What happened? Research design: quasi-experimental Control Pilot Control Pilot Control PilotInterpersonal Communications 8 7 4 3 237 199Principles of Nutrition 6 4 3 2 148 89Medical Terminology 6 7 4 2 197 220Total 20 18 11 7 582 508n sections n instructors n studentsCourse 1,090 students (508 pilot; 582 control) 87% female, average age 32; average household income $20,000 3 courses 18 instructors 20 sections (assigned to pilot or control) 2 terms (Aug Dec 2011) 45 Analysis Logistic regression to examine effect of course design on student success Success (1 or 0): Defined as: Pass (1 or 0) + Master course objectives (>=4 on 0-5 scale) + Stay (1 or 0) Controlled for variation in Instructor prior student success rates Student background variables Age, prior education, prior GPA, tenure, household income Calendar-based success variation 46 Student success: results controlling for variables 11% higher success rate 28% increase Students in redesigned courses were 1.6 times more likely to be successful Wald Chi-Square: 10.42, df=1, n=895, Sig47 More to do: adjusted results varied by course Odds of success more than doubled (2.5 times more) in Interpersonal Communications 9% difference in Principles of Nutrition (not stat significant due to smaller sample) Small improvement in Medical Terminology due to difficulty level of early units 48 Student quote on benefits of added practice Something I found to be interesting was the degree of understanding between me and another individual that wasnt in this class. A girl I had met in a previous term that has a similar degree plan but ended up in a regular medical terminology course, still we would discuss the differences and similarities between are assigned classes. During our unit 8 test she called me hysterical about all the different elements of the final tests and couldnt seem to grasp the concept of the 1st part of the test i.e., analysis diagram, creating new terms from word roots etc. I was mystified that something that had become 2nd nature to me mainly due to the time spent every week filling out the Analysis Tables was so difficult for her to comprehend. It was at that point I realized all the griping I had done was actually the reason my level of understanding is more evolved than somebody who never experienced it. 49 Quote from a student who previously failed This course was difficult for me to do. I tried to do this course when I attended another school and I failed it. I think the way the course was set up and how it broke everything down really helped me to understand it and pass it this time. I would not change a thing about how this course was set up. 50 Satisfaction (end of term survey, 5 point scale) Student satisfaction Lower on end of course survey in redesigned courses (mean 4.4 vs 4.8), but still greater than 4 on 5 point scale. High positivity scores in motivation survey. Why? Courses more rigorous, more work to complete; this is a common finding in other research (e.g., Clark, 1982) Instructor satisfaction Higher in redesigned courses (mean 4.6 vs 4.1) Why? Detailed scoring guides for assignments Less time in discussions more time to monitor and communicate with at-risk students Standard seminar format and content Student materials: structure, clarity, practice 51 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 52 Employers actually expect job applicants to lack the occupational/technical skills required to do the job Slightly over half of all respondents (52.8%) expected that job applicants would lack occupational skills In healthcare, where occupational certifications and licensures are required, over 68% of respondents expect that job applicants would lack occupational skills Do you expect job applicants to be lacking specific occupational skills or technical skills? March 2011 Workforce Connections, Inc. survey of employers in western Wisconsin. Over 400 employers from all 8 counties responded to the survey. All sizes of businesses were represented with the majority of responses coming from businesses with less than 50 employees. 53 and end up investing significantly in training (if they can afford it) What area of training comprises the bulk of your training budget? Of the companies that have training budgets, 68% of the budget is allocated towards skills training for new workers The cost of losing and replacing an experienced paralegal is roughly $100,000.1 Annual paralegal turnover is nearly 50% a year in many large firms. 2 (and) is about 28% nationally. 3 1 Source: American Bar Association Standing Committee on Paralegals (2001)http://apps.americanbar.org/legalservices/paralegals/update/campbellarticle.html 2 Greene, A. and Cannon, T. (2003) Paralegals, profitability and the future of your law practice. 3 Jordan, P. D. (2001) Paralegal Studies (quoting from Bureau of Labor Statistics). Poorly trained employees drive high turnover rates http://apps.americanbar.org/legalservices/paralegals/update/campbellarticle.html54 CTA lets us do better than letting experts teach Experts are mostly unable, unaided, to express in words to novices more than 30% of their decision-making - They can visualize procedures, but not cognitive decisions CTA gets to 70-80% of expert decision-making - Structured interviews with objectively-determined experts - Refined to a gold standard of decisions and tasks When coupled with well-structured training (see Kaplan Way), takes 20%+ less effort, increases student learning by 25%+ with fewer errors Can influence or change how professions think about themselves but may have to leave some hoops for training to gain acceptance 55 CTA methods can help learning environments work better CTA methods have evidence they unlock 40-50% more of experts skills, reduce training time, and increase students confidence CTA provides the inputs (including task scenarios) to inform high-quality, complete, task-centered instruction From patent examiner CTA: 56 Commanding Generals Financial Analysts Trauma Surgeons Fire Chiefs Geologists Salespeople Pharmacological Researchers Experimental Psychologists Patent Examiners Research Librarians Nuclear Generator Design Engineers Psychotherapists Chemists Radiological Cardiologists Neonatal Nurses Classroom Teachers Fighter Pilots SWAT Teams Emergency Room Teams Football Coaches Blackjack winners (21) Chicken Sexers . . . A wide array of professions have already used CTAs 57 CTA has made real differences in training time and learner success by investing in design up-front Medical school surgical instruction CTA-trained surgeons had greater gains from pretest to post-test in less time Also outperformed control group on patients in every measure of performance Kaplan University Online Faculty CTA-based assessment instrument identified faculty whose students achieved 5% less Urate and .5 higher GPA on average than others teaching the same courses. KU is currently developing hiring tools and training to take advantage of the strategies. Spreadsheet training Scores on post-test problems, and average time to completion: Discovery learning: 34% - 60 minutes Guided demonstration: 64% - 49 minutes CTA: 89% - 29 minutes Emergency and safety procedures New course took half the time with higher scores on the performance posttest CTA required 85% more front-end time for design, development, and PD 58 CTA is a systematic way to document expertise CTA is an interview strategy for capturing how highly successful experts perform complex tasks in a variety of settings Goal is to develop authentic demonstration and practice opportunities for how to perform at expert levels Experts are interviewed who 1) have recent (past 2-3 mo.) experience, 2) are consistently successful, and 3) are NOT trainers. Interviews are done with 3-4 experts to unpack their strategies; these are merged to make an efficient approach suitable for training A range of problem examples or performance scenarios are collected from the experts for use in instruction as well 59 Medical Assistant current course content: X = substantial content; x = ancillary content Pharmacology course Diseases - human body 60 MA CTA: Identifies key tasks/skills performed by experts Tool Skills Clinical Skills Patient Assessment First Aid Medical History Patient exam prep Administering Medication Remove sutures Vital signs Patient Education Prep and clean exam room Collect samples and lab specimens Administrative Skills Medical records Prior authorization Communication Computer use Ethical skills Supply maintenance Laboratory Skills Preparation of medication Preparation of samples and lab specimens Course Content Medical Law and Bioethics Medical Terminology Anatomy and Physiology Pharmacology Diseases of the Human Body Medical Office Management Medical Coding and Insurance Professionalism in Health Care Clinical Competencies Original content New focus Tie to domain tasks as identified by experts 61 MA Program: Skills addressed in new sequence Tool Skills Clinical Skills Patient Assessment First Aid Medical History Patient exam prep Administering Medication Remove sutures Vital signs Patient Education Prep and clean exam room Collect samples and lab specimens Administrative Skills Medical records Prior authorization Communication Computer use Ethical skills Supply maintenance Laboratory Skills Preparation of medication Preparation of samples and lab specimens New focus Tie to domain tasks as identified by experts 62 Task-centered instruction Move from simple to increasingly difficult tasks NOT PBL sink or swim Teach everything needed for each task Fade coaching/support over time 63 MA Program: Skills addressed in new sequence Tie to domain tasks as identified by experts Repeated use of skills across courses B: Begin; A: Advanced; R: Reinforce 64 MA Program: New courses include previous content Original Course Content Proposed Course Sequence Admin Skills 1 Clinical Skills 1 Admin Skills 2 Clinical Skills 2 Lab Skills Medical Law and Bioethics X x X x x Medical Terminology X X X X X Anatomy and Physiology X X X Pharmacology X X X Diseases of the Human Body X X x Medical Office Management X X X x X Medical Coding and Insurance X X X Professionalism in Health Care X X x x X Clinical Competencies X X X X X X = substantial content; x = ancillary content Tie to domain tasks as identified by experts Repeated use of skills across courses Original concepts spread across task instruction, not confined to courses 65 Agenda What evidence says about learning What this means for the design of instruction What happens when you do this for real [How to get outcomes aligned with real expertise] [Teaching & Learning in the 21st Century thoughts] 66 67 Diversity? Absolutely An expectation that a wider variance of already-mastered skills has to be accommodated Much more flexibility around the logistics of mastery schedules, life-stage, life changes, etc. New work to help the diverse array of learners really understand what works for learning (Carol Dweck) Arguably, similar issues for faculty learners too! 68 More innovative pedagogies and experiential learning? Yes but need to be taking into account how our learning machinery actually works (medical analogy) -Whether you do or dont the real world awaits. . . Guided/structured experiential learning for novices Closer tie between each concept and applications both must get taught and practiced in close proximity Challenge: How to balance individual instructor innovation with lessons learned from thousands? 69 More collaborative? Collaborating to solve a hard problem at work is not the same as collaborating to learn Learning to collaborate as one does at work is a terrific goal but is just as hard as domain-specific objectives For novices, collaboration to learn too soon can be too much cognitive load may block mastery of key objectives Group activities after mastery are terrific for cementing, extending, generalizing 70 Supported by technology? Technology does NOT solve learning problems per se (or any other problem) Technology takes a good (or bad) solution, and makes it more affordable, reliable, available, data-rich, etc. So it is and will be a critical component of educational systems much more than now But if learning does not take precedence. . . A key is much more systematic use of data to evaluate measures of learning, processes for learning, to personalize learning to students needs Technology (well-deployed) should help us find what works and deploy it systematically and well 71 Changing student expectations? Students should have the right to expect that what they learn is deeply tied to what experts really decide and do variable now, most careers! Students expectations of a good environment are not always correct especially for novices in a domain Indeed, research shows students often think environments that work better for them work worse, and vice-versa (Steve-Jobs-like lesson!) Culture can get in the way, too beliefs about talent/learning Students brains are NOT rewired now - Its the same machinery (narrow working memory supported by fast long term memory) -There are new things driven into long-term memory, however -The system works as it has neurons dont follow Moores law! -E.g.: multi-tasking can produce, but not become better 72 A tool were using: An evidence-based checklist specifications for design and quality assurance Is the course/lesson designed for effective knowledge acquisition and transfer? Learning outcomes/objectives Assessments Practice Presentation: Examples Presentation: Information Content chunking and sequencing Does the course provide support for motivation? Does the course provide opportunities for knowledge integration? Are media used appropriately and efficiently? Does instruction adapt to student's level of knowledge and motivation? The checklist Categories on the checklist 73 Where do items on checklist come from? Principle Design Actions Task-centeredness Include authentic tasks that represent the domain/learning outcomes Activation Connect to learners prior experience/knowledge/larger knowledge structure Demonstration Demonstrate and give examples of correct performance Application Provide part-task and whole-task practice with corrective feedback Integration Deepen knowledge with opportunities for reflection, discussion, public performance, exploration of real life uses Merrill, M. D. First Principles of Instruction, In C. M. Reigeluth & A. Carr (Eds.), Instructional Design Theories and Models III (Vol. III), 2009 First Principles of Instruction Principle Design Actions Multimedia Use words and graphics rather than words alone Contiguity Place printed words near corresponding graphics; Synchronize spoken words with corresponding graphics Modality Present words as audio narration rather than on-screen text Redundancy Explain visuals with words in audio OR text, not both Coherence Avoid interesting but unnecessary material; avoid extraneous audio, graphics, words Personalization Use conversational rather than formal style; Use effective on-screen coaches; Make the author visible Segmenting Break content into bite-size segments Pre-training Teach key concepts prior to procedures or processes Examples Transition from worked examples to problems via fading; Promote self-explanation of worked-out steps; Supplement worked examples with explanations Practice Mirror the job; Provide explanatory feedback; Adapt the amount and placement of practice to job performance requirements; Transition from examples to practice gradually Collaboration Insufficient evidence for guidelines on social learning Learner Control/ Navigation Give experienced learners control; Make important instructional events the default; Consider adaptive control; Give pacing control Build Thinking Skills Use job-specific cases; Make thinking processes explicit; Define job-specific problem-solving processes Games and Simulations Match game type to learning goals; Make learning essential to progress; Build in guidance; Promote Reflection on correct answers; Manage complexity Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008 E-Learning and Multimedia Design Principles 74 Evidence-based checklist: Objectives, Assessment, Practice 1 Learning Outcomes/Objectives1.1 Learning objectives are stated. 0.01.2 Learning objectives are stated as performance objectives, i .e., what learners will be able to DO, not what they will know. 0.01.3 Lesson, module, units, course, and program objectives are aligned. 0.01.4Learning objectives map to certification requirements or competencies or domain taxonomies/standards from professional or accreditation bodies. 0.01.5 Learning objectives are based on cognitive task analysis of expert performance in the domain or profession. 0.0SECTION SCORE 0.02 Assessment2.1 Assessment tasks match learning outcomes/objectives. 0.02.2Assessment tasks measure mastery/acquisition of knowledge components: procedures, facts, concepts, principles, processes (one assessment may cover multiple objectives). 0.02.3 Rubrics (scoring guides) guide scoring and performance of assessment tasks with open-ended response formats. 0.0SECTION SCORE 0.03 Practice3.1 Practice matches assessment. 0.03.2Practice tasks elicit performance to develop procedural knowledge and supportive knowledge components (facts, concepts, principles, processes). 0.03.3 Rules/rubrics diagnose errors and misconceptions. 0.03.4 Feedback/adaptation/guidance corrects errors and misconceptions. 0.03.5 Practice matches transfer context, e.g., job situation. (Prompt is contextually authentic. Response is cognitively authentic). 0.03.6 Part-task practice precedes whole task practice. 0.0SECTION SCORE 0.075 Presentation: Examples and Information 4 Presentation: Examples4.1 Examples (demonstrations, worked examples) match practice. 0.04.2 Demonstrations (or worked examples) i l lustrate task performance (procedures). 0.04.3 Examples, stories, cases il lustrate concepts, principles, processes. 0.0SECTION SCORE 0.05 Presentation: Information5.1Descriptions and explanations cover steps to perform a task (when and how), and related knowledge components - facts, concepts, principles, processes (what and why). 0.05.2 Information needed to do practice tasks is emphasized; "nice to know" information is excluded or minimized. 0.05.3 Information is integrated (interwoven) with examples. 0.0SECTION SCORE 0.076 Chunking, Sequencing, Overviews Do course, unit, and lesson overviews support learning and motivation?9 Overviews9.1Overviews include orientation (description of where each component - course, unit, lesson - fits in a larger curriculum, program, course, process, or hierarchy of objectives). 0.09.2 Goals are clarified (description of the learning outcomes or objectives - what learner will be able to do by end). 0.09.3 Value/Reasons/Benefits/Risks are explained (inoculation against low perceived value of content and/or methods). 0.09.4 Connection to prior knowledge/something familiar is made or activated (story, example, analogy, questions). 0.09.5 Outlines describe what is to come (in the course, unit, or lesson). 0.0SECTION SCORE 0.06 Content Chunking and Sequencing6.1 Content is broken into manageable chunks/segments. 0.06.2 Outcomes/objectives are presented in order of application, difficulty, with prerequisites first. 0.06.3 For each outcome/objective, the learning sequence is Overview, Presentation (Information and Examples), Practice, Assessment. 0.06.4 Navigation from section to section is simple and not confusing - there is a clear sequence and clear directions. 0.0SECTION SCORE 0.077 Multimedia Are media used appropriately and efficiently?10 Multimedia10.1 Graphics and media are relevant and not distracting (Coherence Principle). 0.010.2 Text and graphics are positioned close together without scrolling (Contiguity Principle). 0.010.3 Visuals are explained with text or audio, not both (Redundancy Principle). 0.010.4 For complex graphics, audio narration is used instead of on-screen text (Modality Principle). 0.010.5 Students control pace of media to play, pause, forward, rewind (Pacing Control Principle). 0.010.6Media use is consistent with Section 508 of the Americans with Disabilities Act (e.g., non-text has text equivalent; images are tagged with text, audio has text alternative). 0.010.7 Look and feel are polished; media quality is adequate. 0.0SECTION SCORE 0.078 Motivation, Knowledge Integration, Personalization Does the course provide support for motivation?7 Motivation7.1 References are made to importance/reasons for the content and/or instructional methods used. 0.07.2 The tone is positive, encouraging, conversational. 0.07.3 Statements or stories are included to attribute success and failure to effort, not innate ability. 0.07.4 Statements or stories are included to prevent under-confidence/anxiety and over-confidence. 0.0SECTION SCORE 0.0Does the course provide opportunities for knowledge integration?8 Integration8.1Techniques to promote deep processing and integration of knowledge are included, for example, prompted self-reflection, self-explanation, discussions, student presentations. 0.0SECTION SCORE 0.079 References: Why Students Dont Like School, Daniel Willingham highly readable! ;-) Talent is Overrated, Geoffrey Colvin highly readable! ;-) E-Learning and the Science of Instruction, Clark and Mayer, 2nd ed. First Principles of Learning, Merrill, D., in Reigeluth, C. M. & Carr, A. (Eds.), Instructional Design Theories and Models III, 2009. How People Learn, John Bransford et al, eds. Design factors for educationally effective animations and simulations, Plass, J.L., Homer, B.D., Hayward, E.O., J Comput High Educ (2009) 21:3161 The Implications of Research on Expertise for Curriculum and Pedagogy, David Feldon, Education Psychology Review (2007) 19:91110 Cognitive Task Analysis, Clark, R.E., Feldon, D., van Merrienboer, J., Yates, K., and Early, S.. in Spector, J.M., Merrill, M.D., van Merrienboer, J. J. G., & Driscoll, M. P. (Eds.), Handbook of research on educational communciatinos and technology (3rd ed., 2007) Lawrence Erlbaum Associates