“Learning)engineering”:) The)Artof)Using)Learning… · “Learning)engineering”:)...

“Learning  engineering”:  The  Art  of  Using  Learning  Science  at  Scale  to  Li7  Performance  

October  2015  

2

•  Kaplan University •  Kaplan Legal Education •  Kaplan Professional

Education •  KU Nursing

•  Kaplan Higher Ed – Europe •  Kaplan Professional – Europe

•  Kaplan Higher Ed – Asia •  Kaplan Professional – Asia •  Kaplan Higher Ed – Australia •  Kaplan Professional –

Australia •  In Country Pathways – China •  BEO - HO

Kaplan education spans domains and geography

Kaplan University Group

Kaplan Test Prep Kaplan Performance Solutions

Kaplan Asia Pacific Kaplan United Kingdom •  Kaplan Int’l Colleges

•  Global Pathways •  Dublin Business School

Kaplan International Colleges

•  KTP Grad •  KTP Pre-College •  KTP Med •  KTP Nursing •  KTP Bar Review •  Dev Boot Camp •  KTP International

3

We know a lot about how expertise works

4

This allows for a “learning engineering” approach

Cognitive Task Analysis

5

From expertise to instructional design

Evidence- Based Instructional Design

Cognitive Task Analysis

6

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

7

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

8

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

Procedural Build varied Practice tasks, and rich feedback/coaching

9

How  exper)se  gets  acquired  with  prac)ce  

Stage Implications for Instructional Design

Declarative Clear information displays, e.g., job aids, examples, reference material

Procedural Build varied Practice tasks, and rich feedback/coaching

Automated Repeated frequent practice to build speed and accuracy

10

Knowledge Component

Practice/Assessment S

uppo

rtive

Kno

wle

dge

Research gives us clearer guides to help with hard outcomes

11

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Research gives us clearer guides to help with hard outcomes

12

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Research gives us clearer guides to help with hard outcomes

13

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Research gives us clearer guides to help with hard outcomes

14

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Process Identify causes of faults in a process; predict events in a process

Research gives us clearer guides to help with hard outcomes

15

Knowledge Component

Practice/Assessment

Procedure Decide when to use; perform the steps

Sup

porti

ve K

now

ledg

e Fact Recall fact in task context

Concept Classify, identify or generate examples and non-examples

Process Identify causes of faults in a process; predict events in a process

Principle Decide if principle applies; predict an effect; apply principle to solve a problem, explain a phenomenon or make a decision

Research gives us clearer guides to help with hard outcomes

16

We  also  know  more  about  mo)va)on.  .  .  Motivation Beliefs

•  Value •  Self-Efficacy •  Attribution •  Mood

Motivation Actions

•  Starting •  Persisting •  Mental Effort

Learning/Results

•  Practice results •  Test results •  Fluency/ease •  Work results

Self-Efficacy

Lear

ning

Effo

rt

High Moderate Low

17

And about specific ways to make lessons work better

Principle Description Effect size (s.d. units)

Multimedia Use relevant graphics and text to communicate content 1.5

Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating integrated information

1.1

Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3

Modality Include audio narration where possible to explain graphic presentation 1.0

Redundancy Do not present words as both on-screen text and narration when graphics are present

.7

Personalization Script audio in a conversational style using first and second person 1.3

Segmenting Break content down into small topic chunks that can be accessed at the learner’s preferred rate

1.0

Pre-training Teach important concepts and facts prior to procedures or processes 1.3

Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc.

??

Source: E-learning and the Science of Instruction, Clark and Mayer, 3rd ed., 2011

18

The impact is not small!

50% 1 sd

84%!

19

We  know  there  is  much  poten)al  to  improve*  

*Example: Ratings from Kaplan Way for Learning checklist, applied to 9 Kaplan products

20

Have to be careful – what we think is “good” may not be

•  Comparison of course teacher view vs. independent teachers’ markings

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

4"

4.5"

course"teacher" Teacher"1" Teacher"2"

Based on 10 randomly selected papers from a writing course

21

The evidence also shows our intuitions aren’t the best guides LSAT Logical Reasoning example

0"

1"

2"

3"

4"

5"

6"

7"

Study"8"worked"examples"

Study"15"worked"examples"

Use"Exis>ng"Kaplan"On"Demand"Instruc>on"

Test"Only"–"No"instruc>on"

N 153* 148* 107 84 Time (mins) 8.15 12.8 99.32 NA

* Significant difference from “No Instruction”

22

All this changes how courses should be developed

Read, Write, Discuss •  Outcomes and content not precisely aligned •  Limited demonstrations, worked examples,

and practice •  General assessment rubrics •  High reliance on discussion boards

Existing courses

Prepare, Practice, Perform •  Outcomes and content aligned •  One lesson per objective •  Demonstrations and worked examples •  Practice, feedback before assessment •  Detailed scoring guides •  Less discussion/more practice •  Standard instructor materials •  Monitoring and support for motivation

Redesigned courses

23

Result: much greater student success

•  11% higher success rate

•  28% increase

•  Students in redesigned courses were 1.6 times more likely to be successful

Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.

39%

50%

20%

30%

40%

50%

60%

70%

Control Pilot

Adj

uste

d st

uden

t suc

cess

rate

Adjusted student success rates with 95% confidence limits

24

Faculty supports do help students – but need to check

60%

65%

70%

75%

80%

85%

90%

Attendance Number of assignments*

LMS minutes*

Ret

entio

n

Control Treatment

* Improved learning outcomes

§ §

N: 653 498 625

Impact of faculty dashboards on first year college social studies course

25

At  scale,  we  can  pursue  mul)ple  lines  of  inves)ga)on  

Adaptive

SmartBook

MyLab

Assessment & CLA

CLA Re-Scoring

Assessment

Learning Strategies

Self-Explanation

Self Summary

Self Test

Media Principles

Advanced Organizers

Coherence

Continuity

Modality

Multimedia

Redundancy

Motivation & Self

Efficacy

Judgments of Learning

Motivational Priming

Self Efficacy

Social Norming

Badging

Iconographs

Worked Examples

Early

Later

Others

Section Size

Orientation

Faculty Dashboard

Open Courses

Challenge Exams

26

Running dozens of trials requires careful tracking

1 Study In Prep

Adaptive Learning (Mcgraw Hill Connect AB219)

3 Studies In Development

Attribution I (SOFI) Standing Out Fitting In (AB/MT140) Indigo Live Seminar Delivery (AB/MT203)

19 Studies In Term

Adaptive Learning (McGraw-Hill Connect CM107) Adaptive Learning (SOOMO PS124) Attribution II (KU160) CLA Anchor Paper (PA106) Competency (CS113) Campus LearnLab (CM220 Blended)

Mind Set Challenge (CM220) PERTS Mind Set Challenge (MM150) Science Center Evaluation Study (SC156) AS Section Size (PS124) Self Regulated Learning (CE100) Indigo Live Seminar Delivery (GB512) StereoType Threat Surveys (CJ100, CJ101, MM150, MM207, MM212, MM255)

8 Studies In Assessment (5 Analysis in Progress; 2 Data Requested; 1 Awaiting Data Availability)

Attribution I (HS100 Pre-Intervention) Harvard Study Supporter (CM107 and MM150) CLA Blind Scoring (CJ101 and GB512) Faculty Dashboard (SS310) Social Norming (HU300, MM150, IT301) Worked Examples (CJ130)

34 Studies Completed Advanced Organizers (HN144) CLA Reliability CLA Assignment Order (CJ100) CLA Training (CS204) Orientation Facilitation Bus Section Size Seminar Delivery (LS) Stereotype Threat Historical Review Library Use Historical Review Judgments of Learning (HS101) V4 Learning Platform (IT133) Math Center Feedback (MM207) Motivational Priming (HS100 and PS124) Self Test (HS200) Adaptive Learning (SmartBook HS120) VoiceThread (PA205 and PA201) Worked Examples (14 instances)

27

It’s real work to alter how a large number of IDs build. . .

Online Course

Wor

ksho

p

Analysis Micro Design Development

-8 0 1 3 2 4 5 6 7 8 9 10 11 12

Deliverables

Coaching

Production

Week 13 14 15 16

Delivery

2hr 2hr 2hr 2hr

POs Specs Draft Scripts

C C C

Bug List

Moodle

Revise

Rebuild

Team review / preview ...

2hr

Cohort review ...

Final Scripts

Author

2-month online self-study (40 hours)

2-day workshop Team Projects

4 months coaching

Transfer to products

28

But it matters if you’re after good “learning engineering”

29

Where  to  find  out  more?  

•  LocaAon  of  course  on  using  (and  downloading)  the  checklist:  

                                                           hHp://goo.gl/f1RCAu  

•  Bror’s  Blog  for  more  on  “learning  engineering”:  

                                       hHp://www.kaplan.com/brorsblog    

•  Contact  me:                                                            bror.saxberg@kaplan.com    

30

April 20, 2015 Why We Need Learning Engineers Chronicle of Higher Education