Upload
hatuong
View
223
Download
0
Embed Size (px)
Citation preview
Science Teachers Learning from Lesson Analysis (STeLLA)
Chris Wilson , Joe Taylor, Kathleen Roth
NARST Annual Conference, Pittsburgh PA, April, 2014
STeLLA is:
STeLLA Professional
Development Program
Student Science Content
Knowledge
Teacher Science Content
Knowledge
Teacher Pedagogical
Content Knowledge
Teaching Practice
In the context of videocase-based analysis-of-practice
A one-year professional development program for elementary teachers:
STeLLA is also: a Scale-Up Study
Population: 153 4th and 5th grade teachers from 80 schools, ~4500 students
Cohort 1 2011-2012 45 Schools
STeLLA 22 Schools
37 Teachers
Comp 23 Schools
47 Teachers
Cohort 2 2012-2013 35 Schools
STeLLA 23 Schools
42 Teachers
Comp 12 Schools
27 Teachers
STeLLA I
Lines of Research
Overview of STeLLA PD
Content Deepening Program STeLLA PD Program
Summer Institute, 2 weeks
Full-day science content sessions with university science faculty
• 4th grade: food webs, Earth’s changing surface
• 5th grade: water cycle, Sun’s effect on seasons
Half-day science content sessions with university science faculty
4th grade: food webs, Earth’s
changing surface
5th grade: water cycle, Sun’s effect
on seasons
Half-day lesson analysis study groups with BSCS PD Leaders
4th grade: Earth’s changing surface 5th grade: Sun’s effect on seasons
Academic Year, 30 hours
Content sessions with university science faculty
Lesson analysis study groups Fall: Content Area 1 Winter/Spring: Content Area 2
Key STeLLA Components Video analysis tasks: Teachers analyze videocases from other teachers’ and each other’s classrooms
Videocases include:
sequence of lesson videos
student and teacher interview videos
student written work
student pre- and posttests
Key STeLLA Components Analysis of Practice Cycles and Tools
FOCUS ON STUDENT THINKING AND
SCIENCE CONTENT STORYLINE
Key STeLLA Components STeLLA lesson plans and analysis tools
Key STeLLA Components Student pre-post tests and analysis tools
Theory of Change and Focus Today
STeLLA Professional
Development Program
Student Science Content
Knowledge
Teacher Science Content
Knowledge
Teacher Pedagogical
Content Knowledge
Teaching Practice
STeLLA Intervention Effects
Level 1 (student)
POSTij = π0j + π1jk (PRE)ij + eij
Level 2 (school)
π0j = β00 + β01 (TREAT)j + r0j
Participants Grade
Level
Effect
Coefficient (β01)
Standard
Error t-ratio d.f. p-value
Effect Size
(Hedges g)
Students 4th & 5th 2.96 0.61 4.88 77 <.001 0.40
MEASURE PERSON - MAP - ITEM
<more>|<rare>
66 .####### T+
65 .# + PO23WC
64 .# +
63 .### + PO1SEC PO3FW
62 .## + PO24WC
61 .#### +T
60 .#### + PO14WC PO7ECS
59 .### S+
58 .##### +
57 .####### + PO16ECS PO23SEC
56 .###### + PO5FW PO9ECS
55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC
54 .######## + PO11SEC PO21ECS PO6FW
53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW
52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC
51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS
50 .######### + PO17SEC PO21WC PO7WC PO9SEC
49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC
48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC
47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS
46 .#### + PO16SEC PO18SEC PO18WC PO8FW
45 .####### + PO10FW PO12WC PO14SEC
44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS
43 .### + PO23FW PO24FW PO8SEC PO9WC
42 .#### + PO12FW PO13FW PO5ECS
41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW
40 .### + PO17ECS PO22FW PO22WC PO2FW
39 .# +
38 .# + PO19FW PO6WC PO8WC
37 .# T+ PO1WC PO9FW
36 .# + PO3WC
35 . +T PO17FW
34 . + PO7FW
33 . +
32 +
31 . +
30 + PO11WC
29 . +
<less>|<frequent>
EACH "#" IS 20: EACH "." IS 1 TO 19
MEASURE PERSON - MAP - ITEM
<more>|<rare>
66 .####### T+
65 .# + PO23WC
64 .# +
63 .### + PO1SEC PO3FW
62 .## + PO24WC
61 .#### +T
60 .#### + PO14WC PO7ECS
59 .### S+
58 .##### +
57 .####### + PO16ECS PO23SEC
56 .###### + PO5FW PO9ECS
55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC
54 .######## + PO11SEC PO21ECS PO6FW
53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW
52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC
51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS
50 .######### + PO17SEC PO21WC PO7WC PO9SEC
49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC
48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC
47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS
46 .#### + PO16SEC PO18SEC PO18WC PO8FW
45 .####### + PO10FW PO12WC PO14SEC
44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS
43 .### + PO23FW PO24FW PO8SEC PO9WC
42 .#### + PO12FW PO13FW PO5ECS
41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW
40 .### + PO17ECS PO22FW PO22WC PO2FW
39 .# +
38 .# + PO19FW PO6WC PO8WC
37 .# T+ PO1WC PO9FW
36 .# + PO3WC
35 . +T PO17FW
34 . + PO7FW
33 . +
32 +
31 . +
30 + PO11WC
29 . +
<less>|<frequent>
EACH "#" IS 20: EACH "." IS 1 TO 19
Pre Mean = 43
Post Mean = 52
MEASURE PERSON - MAP - ITEM
<more>|<rare>
66 .####### T+
65 .# + PO23WC
64 .# +
63 .### + PO1SEC PO3FW
62 .## + PO24WC
61 .#### +T
60 .#### + PO14WC PO7ECS
59 .### S+
58 .##### +
57 .####### + PO16ECS PO23SEC
56 .###### + PO5FW PO9ECS
55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC
54 .######## + PO11SEC PO21ECS PO6FW
53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW
52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC
51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS
50 .######### + PO17SEC PO21WC PO7WC PO9SEC
49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC
48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC
47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS
46 .#### + PO16SEC PO18SEC PO18WC PO8FW
45 .####### + PO10FW PO12WC PO14SEC
44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS
43 .### + PO23FW PO24FW PO8SEC PO9WC
42 .#### + PO12FW PO13FW PO5ECS
41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW
40 .### + PO17ECS PO22FW PO22WC PO2FW
39 .# +
38 .# + PO19FW PO6WC PO8WC
37 .# T+ PO1WC PO9FW
36 .# + PO3WC
35 . +T PO17FW
34 . + PO7FW
33 . +
32 +
31 . +
30 + PO11WC
29 . +
<less>|<frequent>
EACH "#" IS 20: EACH "." IS 1 TO 19
BaU Post = 49.73
STeLLA Post = 53.22
MEASURE PERSON - MAP - ITEM
<more>|<rare>
66 .####### T+
65 .# + PO23WC
64 .# +
63 .### + PO1SEC PO3FW
62 .## + PO24WC
61 .#### +T
60 .#### + PO14WC PO7ECS
59 .### S+
58 .##### +
57 .####### + PO16ECS PO23SEC
56 .###### + PO5FW PO9ECS
55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC
54 .######## + PO11SEC PO21ECS PO6FW
53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW
52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC
51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS
50 .######### + PO17SEC PO21WC PO7WC PO9SEC
49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC
48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC
47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS
46 .#### + PO16SEC PO18SEC PO18WC PO8FW
45 .####### + PO10FW PO12WC PO14SEC
44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS
43 .### + PO23FW PO24FW PO8SEC PO9WC
42 .#### + PO12FW PO13FW PO5ECS
41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW
40 .### + PO17ECS PO22FW PO22WC PO2FW
39 .# +
38 .# + PO19FW PO6WC PO8WC
37 .# T+ PO1WC PO9FW
36 .# + PO3WC
35 . +T PO17FW
34 . + PO7FW
33 . +
32 +
31 . +
30 + PO11WC
29 . +
<less>|<frequent>
EACH "#" IS 20: EACH "." IS 1 TO 19
BaU Post = 49.73
STeLLA Post = 53.22
The primary source of energy for a food web comes from:
Where does water in a lake get most of the energy it needs to evaporate?
What is the connection between clouds and rain?
Why is the equator hotter than the poles?
MEASURE PERSON - MAP - ITEM
<more>|<rare>
66 .####### T+
65 .# + PO23WC
64 .# +
63 .### + PO1SEC PO3FW
62 .## + PO24WC
61 .#### +T
60 .#### + PO14WC PO7ECS
59 .### S+
58 .##### +
57 .####### + PO16ECS PO23SEC
56 .###### + PO5FW PO9ECS
55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC
54 .######## + PO11SEC PO21ECS PO6FW
53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW
52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC
51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS
50 .######### + PO17SEC PO21WC PO7WC PO9SEC
49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC
48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC
47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS
46 .#### + PO16SEC PO18SEC PO18WC PO8FW
45 .####### + PO10FW PO12WC PO14SEC
44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS
43 .### + PO23FW PO24FW PO8SEC PO9WC
42 .#### + PO12FW PO13FW PO5ECS
41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW
40 .### + PO17ECS PO22FW PO22WC PO2FW
39 .# +
38 .# + PO19FW PO6WC PO8WC
37 .# T+ PO1WC PO9FW
36 .# + PO3WC
35 . +T PO17FW
34 . + PO7FW
33 . +
32 +
31 . +
30 + PO11WC
29 . +
<less>|<frequent>
EACH "#" IS 20: EACH "." IS 1 TO 19
STeLLA Post = 53.22
BaU Post = 49.73
Some Conclusions
This study is yielding strong evidence that the STeLLA Lesson Analysis approach is efficacious for students, even when non-developers deliver the intervention.
Effects are statistically and practically significant.
Next Steps
• Fold in Colorado state science assessment data and student demographics.
• Add in teacher content knowledge, teacher PCK, and teacher practice measures.
• Examine how teacher learning predicts student learning.
• Look at which STeLLA strategies were most predictive of student learning.
• Qualitative analyses.
An Issue
“Education reform usually arrives with fanfare, great expectations, and overconfidence. Truth be known, typical education-reform effects tend to be small. Evaluations, if done at all, burst the reform balloon, having difficulty finding effects.”
Ruiz-Primo et al., 2002
Ab
ility
Expert Novice
The Measurement Challenge: 1. An Instrument used to Assign Grades
Pretest Posttest
Growth
The Measurement Challenge: 2. An Instrument to Measure Change over Time
Pretest Posttest
*Fidelity of Implementation and Achieved Relative Strength
Control for Pretest
Pretest Posttest
The Measurement Challenge: 3. An Instrument to Differentiate between the Effects of Two Treatments over Time
Treatment Comparison
The Measurement Challenge: 4. An Instrument to Differentiate between the Effects of Two Treatments over Time of a Treatment at the Teacher Level on Learning at the Student Level.
Pretest Posttest
Treatment Comparison
Trickle Down
Teacher Level Student Level
PD Program Student
Outcomes Teacher Learning
Pretest Posttest Pretest Posttest Pretest Posttest
Treatment Comparison
We clearly need very precise measurement instruments…
But due to the scale of many efficacy studies, we are often limited to more crude measures.
(e.g. 200 teachers, pre and post, 4 topics, 30 student per class = ~48,000 student tests)
Instructional Sensitivity
• Learning goal specification and monitoring
• Developing construct models
• Item selection and development
• Pilot testing and item/instrument analyses
Learning Goals Learning Goals Curriculum Materials
Backward Design (Wiggins and McTighe, 1998)
Curriculum Mapping (Ruiz-Primo et al., 2010)
1
2
Item Number
Difficulty Pre
Difficulty Post
Item Discrimination
Item-Total Correlation
Pre-Post Difference
Instructional Sensitivity
Flag
7 .32 .65 .77 .51 .33 .49 8 .12 .53 .33 .52 .41 .47 9 .21 .29 .21 .19 .08 .10 X
NOVICE EXPERT
Increase in Sensitivity Example
To download this session please go to bscs.org/sessions