Studying and achieving robust learning with PSLC resources

1

Studying and achieving robust learning with PSLC resourcesPrepared by: Ken Koedinger

HCII & Psychology, CMUCMU Director of PSLC

Presented by: Vincent AlevenHCII, CMUMember PSLC Executive Committee

2

7th Annual Pittsburgh Science of Learning Center Summer School

• 11th overall– ITS was focus in

2001 to 2004

• Goals:– Learning science &

technology concepts & tools

– Hands-on project => poster on Fri

3

Vision for PSLC

• Why? Chasm between science & practiceIndicators: Ed achievement gaps persist,Low success rate of randomized controlled trials

• Underlying problem: Many ideas, too little sound scientific foundation

• Need: Basic research studies in the field

=> PSLC Purpose: Identify the conditions that cause robust student learning– Field-based rigorous science – Leverage cognitive & computational theory,

educational technologies

“rigorous, sustained scientific research in education” (NRC, 2002)

4

Builds off past success: Intelligent Tutors Bringing Learning Science to Schools!

• Intelligent tutoring systems– Automated 1:1 tutor– Artificial Intelligence– Cognitive Psychology

• Andes: College Physics Tutor– Replaces homework

• Algebra Cognitive Tutor – Part of complete course

Students: model problems with Students: model problems with diagrams, graphs, equationsdiagrams, graphs, equations

Tutor: feedback, help, Tutor: feedback, help, reflective dialogreflective dialog

5

Tutors make a significant difference in improving student learning!

• Andes: College Physics Tutor– Field studies: Significant

improvements in student learning

• Algebra Cognitive Tutor – 10+ full year field

studies: improvements on problem solving, concepts, basic skills

– Regularly used in 1000s of schools by 100,000s of students!

0

10

20

30

40

50

60

Iowa SAT subset ProblemSolving

Represent-ations

Traditional Algebra Course

Cognitive Tutor Algebra

6

President Obama on Intelligent Tutoring Systems!“we will devote more than three percent of our GDP to research and development. …. Just think what this will allow us to accomplish: solar cells as cheap as paint, and green buildings that produce all of the energy they consume; learning software as effective as a personal tutor; prosthetics so advanced that you could play the piano again; an expansion of the frontiers of human knowledge about ourselves and world the around us. We can do this.”

How close to this vision are we now?What else do we need to do?

7

Overview

• PSLC Background– Intelligent Tutoring Systems– Cognitive Task Analysis

• PSLC Methods & Tech Resources– In vivo experimentation– LearnLab courses, CTAT, TagHelper,

DataShop

• PSLC Theoretical Framework

Next

b

CognitiveTutorTechnology

Algebra I GeometryEquationSolver

Algebra II

Cognitive Tutors

ArtificialIntelligence

CognitivePsychology

Research base

Math InstructorsMath EducatorsNCTM Standards

Curriculum Content

Cognitive Tutor Approach

9

• Cognitive Model: A system that can solve problems in the various ways students can

Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d

Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c =

d/a

Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d

Cognitive Tutor Technology

10

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9



If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

11

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9



If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.”

Bug message: “You need tomultiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%

b

CognitiveTutorTechnology

Algebra I GeometryEquationSolver

Algebra II

Cognitive Tutors

ArtificialIntelligence

CognitivePsychology

Research base

Math InstructorsMath EducatorsNCTM Standards

Curriculum Content

Cognitive Tutor Approach

14

Difficulty Factors Assessment:Discovering What is Hard for Students to Learn

Which problem type is most difficult for Algebra students?

Story Problem

As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?

Word Problem

Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?

Equation

x * 6 + 66 = 81.90

15

Algebra Student Results:Story Problems are Easier!

70%61%

42%

0%

20%

40%

60%

80%

Story Word Equation

Problem Representation

Percent Correct

Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.

Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science.

17

“The Student Is Not Like Me”

• To avoid your expert blind spot, remember the mantra:

“The Student Is Not Like Me”

• Perform Cognitive Task Analysis to find out what students are like

Use Data!

19

Prior achievement:Intelligent Tutoring Systems bring learning science to schools

A key PSLC inspiration:Educational technology as research platform to generate new learning science

20

Overview



DataShop

• PSLC Theoretical Framework

Next

21

PSLC Statement of Purpose

Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning.


22

What is Robust Learning?

• Achieved through:– Conceptual understanding & sense-making

skills– Refinement of initial understanding– Development of procedural fluency with basic

skills

• Measured by:– Transfer to novel tasks– Retention over the long term, and/or – Acceleration of future learning

23



24

In Vivo Experiments: Laboratory-quality principle testing in real classrooms

25

In Vivo Experimentation MethodologyImportant features of different research

methodologies:• What is tested?

– Instructional solution vs. causal principle

• Where & who?– Lab vs. classroom

• How?– Treatment only vs. Treatment + control

• Generalizing conclusions:– Ecological validity: What instructional activities work in real classrooms?– Internal validity: What causal mechanisms explain & predict?

33

In Vivo Experimentation MethodologyLab

ExperimentsDesign

ResearchRandomized Field Trials

In vivo Experiments

What?

Instructional solution

√ √

Causal principle √ × × √

Where & who?

Lab √

Classroom × √ √ √

How?

Treatment only √

Treatment + control

√ × √ √

Generalizes how?

Internal validity √ × +/– √

Ecological validity

× √ √ √

34

LearnLabA Facility for Principle-Testing

Experiments in Classrooms

35

LearnLab courses at K12 & College Sites

• 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English

• Data collection– Students do home/lab work

on tutors, vlab, OLI, …– Log data, questionnaires,

tests DataShop

Researchers Schools

Learn Lab

Chemistry virtual labChemistry virtual lab

Physics intelligent tutorPhysics intelligent tutor

REAP vocabolary tutorREAP vocabolary tutor

36

PSLC Technology Resources

• Tools for developing instruction & experiments– CTAT (cognitive tutoring systems)

• SimStudent (generalizing an example-tracing tutor)

– OLI (learning management)– TuTalk (natural language dialogue)– REAP (authentic texts)

• Tools for data analysis – DataShop– TagHelper

37



38

Overview



DataShop

• PSLC Theoretical Framework Next

39

KLI Framework: Designing Instruction for Robust Learning

The conditions that yield robust learning can be decomposed at three levels:Knowledge componentsLearning eventsInstructional events

Get framework report at learnlab.org

40

Instructional events

Explanation, practice, text, rule, example, teacher-student discussion

Assessment eventsQuestion, feedback, step in ITS

Learning events

KC accessible from long-term memory

Immediate performance

Robust performance

KEYOvals – observableRectangle - inferredSolid line – causeDashed line – inferencesKC = Knowledge Component

State test, belief survey

KLI Event Decomposition

• Decompose temporal progress of learning

• Observe/control instructional & assessment events

• Infer learning events & changes in knowledge

41

KLI Framework: Designing Instruction for Robust Learning

The conditions that yield robust learning can be decomposed at three levels:Knowledge componentsLearning eventsInstructional events

42

What’s the best form of instruction? Two choices?• More assistance vs. more challenge

– Basics vs. understanding– Education wars in reading, math, science…

• Researchers like binary oppositions too.We just produce a lot more of them!– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving (Sweller,Renkl)– Direct instruction vs. discovery learning (Klahr)– Re-explain vs. ask for explanation (Chi, Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …

Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review.

43

How many options are there really? And what works best when?

What’s best?

Focused practice

Distributed practice

Study examples

Test onproblems

50/50Mix

Concrete AbstractMix

Delayed No feedback

Immediate

Block topics in chapters

Interleave topics

Fade

Explain Ask for explanations

Mix

Gradually widen

Study Test50/50 Study Test50/50Study

Concrete Mix Abstract

ImmediateNo

feedbackDelayed

Block topics in chapters

Fade Interleave topics

Explain Ask for explanations

Mix

More helpBasics

More challenge Understanding

44

205,891,132,094,649“Big Science” effort needed to tackle this complexity

Cumulative theory development Field-based basic research with microgenetic data collection

Derivation:15 instructional dimensions 3 options per dimension 2 stages of learning=> 315*2 options

45

An example of some PSLC studies in this space

• Researchers like binary oppositions too.We just produce a lot more of them!– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving (Sweller,Renkl)– Direct instruction vs. discovery learning (Klahr)– Re-explain vs. ask for explanation (Chi, Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …

46

Learn by doing or by studying?• Testing effect (e.g., Roediger & Karpicke, 06)

– “Tests enhance later retention more than additional study of the material”

Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work:…failure of… problem-based…teaching. • Worked example effect

– “a worked example constitutes the epitome of strongly guided instruction [aka optimal instruction]”

– Paas and van Merrienboer (1994), Sweller et al. (1998), van Gerven et al. (2002), van Gog et al. (2006), Kalyuga et al. (2001a, 2001b), Sweller et al. (1998), Ayres (2006), Trafton & Reiser (1993), Renkl and Atkinson (2003), … the list goes on …

• Theoretical goal: Address debate between desirable difficulties, like “testing effect”, and direct instruction, like “worked examples”

• Limitation of past worked example studies have weaker control, untutored practice– PSLC studies compare to tutored practice

47

Worked Example Experiments within Geometry Cognitive Tutor (Alexander Renkl, Vincent Aleven, Ron Salden, et al.)

• 8 studies in US & Germany– Random assignment, vary single principle– Over 500 students– 3 in vivo studies run in Pittsburgh area

schools

• Cognitive Science ’08 Conference IES Best Paper Award

48

Ecological Control = Standard Cognitive Tutor Students solve problems step-by-step & explain

49

Worked out steps with calculation shown by Tutor

Treatment condition: Half of steps are given as examples

Student still has to self explain worked out step

50

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Pretest Transfer:Procedural

Transfer:Declarative

Percentage correct

Example

Problem

d = .73 *

Worked examples improve efficiency & understanding

Lab results• 20% less time

on instruction• Conceptual

transfer in study 2

In Vivo• Adaptively fading examples to problems yields

better long-term retention & transfer

51

Worked example effect generalizes across domains, settings, researchers• Geometry tutor studies• Chemistry tutor studies in vivo at High School & College

(McLaren et al.)

– Same outcomes in 20% less time• Algebra Tutor study in vivo (Anthony et al.)

– Better long term retention in less time • Theory: SimStudent model (Matsuda et al.)

– Problems provide learning process with negative examples to prune misconceptions

• Research to practice– Influencing Carnegie Learning development– New applied projects with SERP, WestEd

52

Processes of Learning within the KLI Framework

The conditions that yield robust learning can be decomposed at three levels:Knowledge

componentsLearning eventsInstructional events

Learning eventsFluency building, refinement, sense making

53

Learning Events in the Brain & reflected in Dialogue

• Fluency buildingMemory, speed, automaticity

• Refinement processesClassification, co-training, discrimination, analogy, non-verbal explanation-based learning

• Sense-making processesReasoning, experimentation, explanation, argument, dialogue

Some PSLC Examples• ACT-R models of spacing,

testing effects & instructional efficiency (Pavlik)

• SimStudent models of learning by example & by tutoring– Inductive logic prgrming,

probabilistic grammars (Matsuda, Cohen , Li, Koedinger)

• Transactivity+ analysis of peer & classroom learning dialogues (Rose, Asterhan, Resnick)

Ken Koedinger

Use big picture and elaborate with these.

54

Knowledge components carry the results of learning• Knowledge component =

an acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks.

• Used in broad sense of a knowledge base– From facts to mental

models, metacognitive skills

55

Example KCs with different features• Chinese vocabulary KCs:

const->const, explicit, no rationale– If the Chinese pinyin is “lao3shi1”, then the English word is “teacher”– If the Chinese radical is “ 日” , the English word is “sun”

• English Article KCs: var->const, implicit, no rationale

– If the referent of the target noun was previously mentioned, then use “the”

• Geometry Area KCs: var->var, implicit & explicit, rationale

– If the goal is to find the area of a triangle,and the base <B> and the height is <H>,then compute 1/2 * <B> * <H>

– If the goal is to find the area of irregular shape made up of regular shapes <S1> and <S2>, then find area <S1> and <S2> and add

Integrated KCs for mental models, central conceptual structures, strategies & complex planning

56

Kinds of Knowledge Components

• Other kinds of KCs– Integrative, probabilistic, metacognitive, misconceptions or “buggy”

knowledge

57

Knowledge components are not just about domain knowledge• Examples of possible domain-general KCs• Metacognitive strategy

– Novice KC: If I’m studying an example, try to remember each step– Desired KC: If I’m studying an example, try to explain how each

step follows from the previous• Motivational belief

– Novice: I am no good at math– Desired: I can get better at math by studying and practicing

• Social communicative strategy– Novice: When an authority figure speaks, remember what they

say. – Desired: Repeat another's claim in your own words and ask

whether you got it right

Can these be assessed, learned, taught? Broad transfer?

58

Learning curves: Measuring behavior on tasks over time• Data from flash-card tutor• Tasks present a Chinese word & request the English translation• Learning curve shows average student performance (e.g., error rate, time on

correct responses) after each opportunity to practice

6 secs

3 secs

59

Empirical comparison of KC complexity

Example KC typesChinese vocabulary KCs const->const, explicit, no rationale

English Article KCs var->const, implicit, no rationale

Geometry Area KCs var->var, implicit/explicit, rationale

Time 6 -> 3 secs

10 -> 6 secs

14 -> 10 secs

6 secs

3 secs

10 secs

6 secs

14 secs

10 secs

60

Which instructional principles are effective for which kinds of knowledge components?

Do complex instructional events aid simple knowledge acquisition?

Do simple instructional events aid complex knowledge acquisition?

61

Prompted self-explanation studies across domains• Physics Course - field principles

– Better transfer than providing explanations

• Geometry Course - properties of angles– Better transfer than just practice despite

solving 50% fewer problems in same time

• English Course - article use– Pure practice appears more efficient;

self-explanation may help for long-term retention & for novices

• Cross-domain hypoth: Type of KC determines when self-explanation will be effective.

Var->Var, explicit

Var->Const, implicit

62

Summary• Obama: “learning software as effective as a

personal tutor”• How close to this vision are we now?

– Many fielded Intelligent Tutors– Students learn as much or more

• What else do we need to do?– Expand to more areas => CTAT– More sophisticated interaction => CSCL– Use tutors to advance science & improve

educational practice => In Vivo & EDM

In other words …Take the PSLC Summer School!

63

64

To do • Other possibilities

– Reduce in vivo slide by deleting redundancy between text and table

– Define ITS as about the intelligence in the design – Add an opener?

• Not techy enough– TEDx talk => KDD Cup => learning curves

• Lots of issues -- gaming vs. not (Shih)• Tie to WE and SE

– Get from HCI AB talk (CSCL example) Too much listing at the end

65

LearnLab Products

Infrastructure created and highly used• LearnLab courses have supported over

150 in vivo experiments

• Established DataShop: A vast open data repository & associated tools– 110,000 student hours of data

• 21 million transactions at ~15 second intervals – New data analysis & modeling algorithms– 67 papers, >35 are secondary data analysis not

possible without DataShop

66

Novice Expert

Learning

Instruction

Pre-test Post-test

Typical Instructional Study• Compare effects of 2 instructional conditions in lab • Pre- & post-test similar to tasks in instruction

67

Post-test

Novice Expert(desired)

Learning

Instruction

Pre-test Post-test: Long-term retention, transfer, accelerated future learning, or desire for future learning

• Macro: Measures of robust learning

PSLC Instructional Experiments

68

Post-test


Learning

Instruction




Assessment Events

Knowledge:Shallow percepts & concepts

Knowledge: Deep percepts & concepts, fluent

• Micro analysis: knowledge, learning, interactions

Instructional Events

69

Post-test

Coursegoals &cultural context

• Studies run in vivo as part of existing courses


Learning

Instruction




Assessment Events

Knowledge:Shallow percepts & concepts

Knowledge: Deep percepts & concepts, fluent

• Micro analysis: knowledge, learning, interactions

Instructional Events

70

Develop a research-based, but practical framework• Theoretical framework key goals

– Support reliable generalization from empirical studies to guide design of effective ed practices

Two levels of theorizing: • Macro level

– What instructional principles explain how changes in the instructional environment cause changes in robust learning?

• Micro level– Can learning be explained in terms of what knowledge

components are acquired at individual learning events?

71

Example study at macro level: Hausmann & VanLehn 2007

• Research question– Should instruction provide explanations and/or elicit

“self-explanations” from students?

• Study design– All students see 3 examples & 3 problems

• Examples: Watch video of expert solving problem• Problems: Solve in the Andes intelligent tutor

– Treatment variables: • Videos include justifications for steps or do not• Students are prompted to “self-explain” or paraphrase

72

Para-phrase

Self-explain

Explan XNo

explan

73

Para-phrase

Self-explain

Explan

No explan X

74

Self-explanations => greater robust learning

0.670.670.90 0.980

0.2

0.4

0.6

0.8

1

1.2

Paraphrase,With just.

Paraphrase,Without just.

Self-explain,With just.

Self-explain,Without just.

(hints+errors) / steps

• Justifications: no effect!

• Immediate test on electricity problems:

• Transfer to new electricity homework problems

0.450.370.69 1.040

0.2

0.4

0.6

0.8

1

1.2

1.4






• Instruction on electricity unit => accelerated future learning of magnetism!

0.911.17 0.75 0.680

0.2

0.4

0.6

0.8

1

1.2

1.4






75

Key features of H&V study

• In vivo experiment– Ran live in 4 physics sections at US Naval

Academy – Principle-focused: 2x2 single treatment

variations– Tight control manipulated through

technology

• Use of Andes tutor => repeated embedded assessment without

disrupting course

• Data in DataShop (more later)

76

Develop a research-based, but practical framework• Theoretical framework key goals

– Support reliable generalization from empirical studies to guide design of effective ed practices

Two levels of theorizing: • Macro level

– What instructional principles explain how changes in the instructional environment cause changes in robust learning?

• Micro level– Can learning be explained in terms of what knowledge

components are acquired at individual learning events?

77

Knowledge Components

• Knowledge Component– A mental structure or process that a learner uses,

alone or in combination with other knowledge components, to accomplish steps in a task or a problem-- PSLC Wiki

• Evidence that the Knowledge Component level functions in learning …

79

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

Problem1 Problem2 Problem3

(hints+errors)/steps

Instructionalexplanation

Self-explanation

Example1 Example2 Example3

Back to H&V study: Micro-analysisLearning curve for main KCSelf-explanation effect tapers but not to zero

80

PSLC wiki: Principles & studies that support them

Points to Hausmann’s study page (and other studies too)

Instructional Principle pages unify across studies

81


With links to concepts in glossary Hausmann’s study description:

82


~200 concepts in glossarySelf-explanation glossary entry

83

Research Highlights• Synthesizing worked examples &

self-explanation research – 10+ studies in multiple 4 math & science domains– New theory: It’s not just cognitive load!

• Examples for deep feature construction, problems & feedback for shallow feature elimination

This work inspired new question: Does self-explanation enhance language learning? Experiments in progress …

• Computational modeling of student Learning – Simulated learning benefits of examples/demonstrations vs.

problem solving (Masuda et al., 2008)• Theory outcome: problem solving practice is an important source of

negative examples • Engineering: “programming by tutoring” is more cost-effective than

“programming by demonstration”– Shallow vs. deep prior knowledge changes learning rate (Matsuda et

al., in press)

84

Research Highlights (cont) Computational modeling of instructional assistance

Assistance formula: Optimal learning (L) depends on right level of assistance

Relevant to multiple experimental paradigms & dimensions of instructional assistance Direct instruction (worked examples) vs. constructivism (testing effect)

Concrete manipulatives vs. simple abstractions

Formula provides path to resolve hot debates

P*Sb+(1-P)FbP*Sc+(1-P)Fc

L =

Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist

Kaminski, Sloutsky, & Heckler (2008). The advantage of learning abstract examples in learning math. Science.

L

PAssistance

85

Research Highlights (cont) Synthesis paper on computer tutoring of metacognition

Generalizes results across 7 studies, 3 domains, 4 populations Posed new questions about role of motivation

Lasting effects of metacognitive support Computer-based tutoring of self-regulatory learning

Technologically possible & can have a lasting effect Students who used help-seeking tutor demonstrated better learning

skills in later units after support was faded Spent 50% more time reading help messages

Data mining for factors that affect student motivation Machine learning to analyze vast student interaction data

from full year math courses (Baker et al., in press a & b) Students more engaged on “rich” story problems than standard Surprise: Also more engaged on abstract equation exercises!

Koedinger, Aleven, Roll, & Baker. (in press). In vivo experiments on whether supporting metacognition in intelligent tutoring systems yields robust learning. In Handbook of Metacognition in Education.

86

Social Communication

Social context of classroom Teacher

Interaction

Thrusts investigate overlapping factors

Novice Expert

Learning

Instruction

Knowledge:Shallow, perceptual

Knowledge: Deep, conceptual, fluent

THRUSTSCognitive Factors

MetacognitionMotivation

MetacognitionMotivation

Metacognition & Motivation

Motivation Metacognition

Comp Modeling & Data Mining

Learning

Instruction

Knowledge Knowledge

87

Thrust Research Questions• Cognitive Factors. How do instructional events affect

learning activities and thus the outcomes of learning?• Metacognition & Motivation. How do activities

initiated by the learner affect engagement with targeted content?

• Social Communication. How do interactions between learners and teachers and computer tutors affect learning?

• Computational Modeling & Data Mining. Which models are valid across which content domains, student populations, and learning settings?

88

4th Measure of Robust Learning • Existing robust learning measures

– Transfer– Long-term retention– Acceleration of future learning

• New measure:– Desire for future learning

• Is student engaged in subject? • Do they chose to pursue further math, science, or

language?

89

END

Documents

Studying and achieving robust learning with PSLC resources