Upload
amy-cassata-phd
View
15
Download
0
Tags:
Embed Size (px)
Citation preview
Measuring educa-onal interven-ons in context: A system for iden-fying how and why instruc-onal materials are effec-ve
Amy Cassata, PhD and Jeanne Century, PhD Flash Presenta5on at the Third Biennial Global Implementa5on Conference Dublin, Ireland, May 29, 2015
Measuring educa-onal interven-ons in context: A system for iden-fying how and why instruc-onal materials are effec-ve
Amy Cassata, PhD and Jeanne Century, PhD Flash Presenta5on at the Third Biennial Global Implementa5on Conference Dublin, Ireland, May 29, 2015
Educa&onal interven&ons are curricula, tools, teaching approaches, programs, or prac5ces designed to improve student outcomes.
Instruc&onal materials are one type of educa5onal interven5on. They are resources designed for use by teachers in classrooms to guide their instruc5on.
To help guide selec5on decisions, instruc5onal materials are evaluated based on evidence of their effec5veness in improving student achievement.
The evidence is organized and summarized to help school and district leaders decide which instruc5onal materials to adopt in their seLng.
The expecta5on is that teachers’ use of these materials will lead to large-‐scale changes in prac5ce and in turn, increased levels of student achievement.
Evidence-‐based Instruc5onal materials
Large-‐scale changes in teacher prac5ce
Improved student achievement
But how do schools and districts get from point A, to point B, to point C?
D Evidence-‐based Instruc5onal materials
Large-‐scale changes in teacher prac5ce
Improved student achievement
Con5nued posi5ve outcomes
TIME
And, how do they ensure that any changes in teacher prac5ce, and effects on student outcomes, will be sustained over 5me?
We developed a theory of implementa5on to understand and measure interven5ons in prac5ce, the factors that influence their use, and their rela5onships to outcomes.
We then consider the context in which the interven5on is used, which also has many elements.
THE CONTEXT THE CONTEXT
Characteris5cs of individual end-‐users include demographics, disposi5ons, and percep5ons.
Percep5ons
Individual beliefs
Mo5va5on
Self-‐efficacy
Innova5veness
Resourcefulness
Experience
Demographics
THE CONTEXT
Characteris5cs of the organiza5on include people, structures, and strategies.
THE CONTEXT THE CONTEXT
Ongoing improvement structures
Organiza5onal innova5veness
Time
Locus of decision making
Clarity of
organiza5onal goals
Learning opportuni5es
Leadership
Elements of the environment include policies, poli5cs, and community characteris5cs.
THE CONTEXT
Poli5cal environment
Extraneous events
Community
characteris5cs
Community beliefs
Learning
opportuni5es
THE CONTEXT
Networks are systems of interconnected people or organiza5ons.
THE CONTEXT THE CONTEXT Network structure
Member
rela5onships
Trust
Communica5on
Change and evolu5on
The interven5on is enacted in a context over 5me, which means that elements of the context change.
THE CONTEXT THE CONTEXT THE CONTEXT
And in turn, interven5on components may be adapted to fit the changing context.
THE CONTEXT THE CONTEXT THE CONTEXT
All of these elements together make up the IMPLEMENTATION PROCESS.
THE CONTEXT THE CONTEXT THE CONTEXT
D Evidence-‐based Instruc5onal materials
Large-‐scale changes in Teacher prac5ce
Improved student achievement
Con5nued posi5ve outcomes
TIME
Based on this theory, we created a suite of instruments to measure the implementa5on of instruc5onal materials in school systems.
D Evidence-‐based Instruc5onal materials
Large-‐scale changes in Teacher prac5ce
Improved student achievement
Con5nued posi5ve outcomes
TIME
Part of this work involved a research study to understand the implementa5on of the Everyday Mathema&cs curriculum in five school districts.
“poten5ally posi5ve effects” on math achievement -‐ What Works Clearinghouse (2010)
D Evidence-‐based Instruc5onal materials
Large-‐scale changes in Teacher prac5ce
Improved student achievement
Con5nued posi5ve outcomes
TIME
We focused in par5cular on measuring teacher varia5on in EM use, and understanding why teachers used EM in par5cular ways.
We administered two ques5onnaires to 387 K-‐5 teachers to collect data on which EM components they were using, and why.
EM essen5al components
The first ques5onnaire asked teachers to report the extent to which they used each of the EM components in the most recent unit they taught.
Items measured teachers’ use of EM structures (ac5vi5es, procedures, materials, and content), their interac5ons with students, and level of student par5cipa5on.
EM essen5al components
The second ques5onnaire asked teachers to report on selected teacher and school-‐level factors.
THE CONTEXT
Percep5ons about EM
Beliefs about student ability & behavior
Extrinsic mo5va5on
for EM
Self-‐efficacy for EM
Innova5veness
Resourcefulness
Years of experience
Ongoing improvement structures
Organiza5onal innova5veness
Time
Locus of decision making
Clarity of
organiza5onal goals
Facilita5on of Small Group Par5cipa5on
Facilita5on of Cogni5vely Demanding Work
Facilita5on of Students Taking Risks
Facilita5on of Student Interest
Use of Assessment to Inform Instruc5on Use of Differen5a5on
Through confirmatory factor analysis, we iden5fied six specific EM instruc5onal prac5ces.
We conducted mul5ple regression analysis to see which factors were most strongly related to which EM prac5ces.
THE CONTEXT
Beliefs about student ability Beliefs about student behavior Perceived effec5veness for EM Extrinsic mo5va5on for EM Self-‐efficacy for EM Innova5veness Resourcefulness Years of experience
Ongoing improvement structures Clarity of organiza5onal goals Organiza5onal innova5veness Locus of decision making Time sufficiency
? EM prac5ces
?
We found that teacher characteris5cs were more influen5al than school characteris5cs.
THE CONTEXT
Beliefs about student ability Beliefs about student behavior Perceived effec5veness for EM Extrinsic mo5va5on for EM Self-‐efficacy for EM Innova5veness Resourcefulness Years of experience
Ongoing improvement structures Clarity of organiza5onal goals Organiza5onal innova5veness Locus of decision making Time sufficiency
? EM prac5ces
?
D Evidence-‐based Instruc5onal materials
Large-‐scale changes in Teacher prac5ce
Improved student achievement
Con5nued posi5ve outcomes
TIME
Next, we were interested in which of these EM prac5ces were associated with changes in student math achievement.
We conducted a path analysis to predict student math achievement using data from one district (N=125 teachers).
We es5mated a path model to predict math achievement for students in grades 2-‐5.
Variables controlled for Independent variables Dependent variable
The model contained the 6 EM instruc5onal prac5ces we iden5fied in CFA.
EM Instruc-onal prac-ces
Facilita5on of small group par5cipa5on
Facilita5on of cogni5vely demanding work
Facilita5on of students taking risks
Facilita5on of student interest
Use of assessment to inform instruc5on
Use of differen5a5on
The model also contained 6 EM structures.
EM Structures
Number lessons taught in order
Number of lessons with parts in order
Number of lessons omiked
Number of lesson with parts omiked
Amount of content covered in the unit
Non-‐EM materials supplementa5on
Significant, posi5ve direct effects on student achievement included: ✔ Non-‐EM supplementa5on ✔ Prior achievement
Significant, posi5ve indirect effects on student achievement included: ✔ Facilita5on of cogni5vely demanding work ✔ Facilita5on of student interest ✔ Number of lessons taught in order ✔ Prior achievement
For more informa5on, contact Amy Cassata at [email protected] Century, J., & Cassata A. (2014). Conceptual founda5ons for measuring the implementa5on of educa5onal innova5ons. In L.M.H. SaneL & T.R. Kratochwill (Eds.), Treatment Integrity: A Founda&on for Evidence-‐Based Prac&ce in Applied Psychology (pp. 81-‐108). Washington, DC: American Psychological Associa5on.