Assessing instructional and assessment practice:What makes a lesson formative?

CRESST conference, September 2004UCLA Sunset Village, CA

Dylan WiliamETSdwiliam@ets.org

www.dylanwiliam.net

What makes a lesson formative?

• Moving from “I know it when I see it”

• Expanding ideas of formative assessment– Feedback– Elicitation and feedback– The student’s role– Systems thinking

Preliminary assumptions

• The primary purpose of educational research is the improvement of education.

• The purpose of education is the improvement of student achievement

• The improvement of student achievement will be achieved primarily through changes in what happens in classrooms

• The role of the teacher is not to teach but to create situations in which students learn

Functions of assessment

• For evaluating institutions

• For describing individuals

• For supporting learning– Monitoring learning

• Whether learning is taking place

– Diagnosing (informing) learning• What is not being learnt

– Forming learning• What to do about it

Effects of feedback

• Kluger & DeNisi (1996)• Review of 3000 research reports• Excluding those:

– without adequate controls– with poor design– with fewer than 10 participants– where performance was not measured– without details of effect sizes

• left 131 reports, 607 effect sizes, involving 12652 individuals

• Average effect size 0.4, but– Effect sizes very variable– 40% of effect sizes were negative

Effects of formative assessment

• Several major reviews of the research– Natriello (1987)– Crooks (1988)– Black & Wiliam (1998)– Nyquist (2003)

• All find consistent, substantial effects

Kinds of feedback (Nyquist, 2003)

• Weaker feedback only– Knowledge or results (KoR)

• Feedback only– KoR + clear goals or knowledge of correct results (KCR)

• Weak formative assessment– KCR+ explanation (KCR+e)

• Moderate formative assessment– (KCR+e) + specific actions for gap reduction

• Strong formative assessment– (KCR+e) + activity

Effects of formative assessment (HE)

No. Effect

Weaker feedback only 31 0.16

Feedback only 48 0.23

Weaker formative assessment 49 0.30

Moderate formative assessment 41 0.33

Strong formative assessment 16 0.51

Formative assessment

• Classroom assessment is not (necessarily) formative assessment

• Formative assessment is not (necessarily) classroom assessment

Formative assessmentAssessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged.

Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs.

Black et al, 2002

Bad items/good items 1

Which is larger: 0.33 or 1/3?A) 0.33 is larger than 1/3

B) 1/3 is larger than 0.33

C) They are the same

D) You need more information to be sure

Bad items/good items 2

What is the general rule for the following

pattern: 3, 7, 11, 15, …?A) 3 + n

B) n + 4

C) 3n + 4

D) 4n - 1

E) 4n + 3

Bad items/good items 3

If f + g = 8, then f + g + h = ?A) 9

B) 12

C) 15

D) 16

E) 8 + h

Bad items/good items 4

Which of the following statements are true ?A) A square has two 90° angles

B) A square is a trapezoid

C) A kites are squares

D) A trapezoid is a rectangle

E) A rectangle is a trapezoid

Bad items/good items 5

Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes dissolve?

A) The level of the water drops

B) The level of the water stays the same

C) The level of the water increases

D) You need more information to be sure

Bad items/good items 6

What is the pH of gasoline?A) Less than zero

B) Zero

C) Greater than zero but less than 7

D) 7

E) Greater than 7

F) None of the above

Bad items/good items 7

In what year did World War II begin?A) 1919

B) 1937

C) 1938

D) 1939

E) 1941

F) 1942

Good formative items are different…

• Can have more (or less!) than one correct answer• Items need to be generative

– of learning– of insights into learning– of insights into how to promote learning

• Increasing ability may reduce your chance of getting the item correct

• Distractors must be explicitly connected to incorrect or incomplete conceptions (facets)

• Item responses must provide clues to effective action• Some items support both summative and formative

functions well; most do not

How do we make good items?

• Good items are not universal– In some domains, facets of thinking are reasonably

universal (e.g. physics)

– In most domains, facets of thinking are dependent on instruction

– So there is no ‘universal’ set of facets

– And we have to rely on teachers to do it for themselves

• Good items have to be used– Requires a model of effective, scalable teacher

professional development

Why research hasn’t changed teaching

• The nature of expertise in teaching

• Aristotle’s main intellectual virtues– Episteme: knowledge of universal truths

– Techne: ability to make things

– Phronesis: practical wisdom

• What works is not the right question– Everything works somewhere

– Nothing works everywhere

– What’s interesting is “under what conditions” does this work?

• Teaching is mainly a matter of phronesis, not episteme

Countdown

25 3

9 4

Target number: 127

1

Aspects of formative assessment

Where the learner is

Where they are going

How to get there

TeacherEvoking

informationCurriculum philosophy

Feedback

PeerPeer-

assessmentSharing criteria

Peer-tutoring

LearnerSelf-

assessmentSharing criteria

Self-directed learning

A formative lesson

• Grade 9 Algebra lesson• Weather balloon set off, rising at 8fs-1

• 2s later, a flare is launched, initial velocity 80fs-1

• Students complete a table and a graph to answer three questions:– Maximum height of flare– How long to max height– Average velocity of flare

Regulation of learning• Teaching as engineering learning

environments

• Key feature: well-regulated– Teaching vs. learning– Regulation of activity vs. regulation of learning– Long feedback cycles vs. variable feedback

cycles– Quality control vs. quality assurance in learning

Regulation of learning

• Proactive (upstream) regulation– Planning regulation into the learning environment– Planning for evoking information

• Interactive (downstream) regulation– ‘Negotiating the swiftly-flowing river’– ‘Moments of contingency’– Tightness of regulation (goals vs horizons)

• Retrospective regulation– Structured reflection (e.g. lesson study)