Large Scale, Randomized Cluster Large Scale, Randomized Cluster Design Study of the Relative Design Study of the Relative

Effectiveness of Reform-Based and Effectiveness of Reform-Based and Traditional/Verification Curricula in Traditional/Verification Curricula in

Supporting Student Science Learning Supporting Student Science Learning and Teachers’ Knowledge and Beliefsand Teachers’ Knowledge and Beliefs

E.M. Granger and T.H. Bevis, E.M. Granger and T.H. Bevis,

Office of Science Teaching Activities, College of Arts & SciencesOffice of Science Teaching Activities, College of Arts & Sciences

Florida State UniversityFlorida State University

  

Y. SakaY. Saka

Zonguldak Karaelmas University, Eregli Egitim Fakultesi, TurkeyZonguldak Karaelmas University, Eregli Egitim Fakultesi, Turkey

S.A. SoutherlandS.A. Southerland

School of Teacher Education/FSU-Teach, Florida State UniversitySchool of Teacher Education/FSU-Teach, Florida State University

Recent Reform Efforts’ Recent Reform Efforts’ Emphasis on Learning Emphasis on Learning

Student understanding of the discipline’s core Student understanding of the discipline’s core concepts, theories, and models; concepts, theories, and models;

Student understanding, in general, of how Student understanding, in general, of how scientific knowledge is generated, tested, and scientific knowledge is generated, tested, and accepted; and accepted; and

Students being able to use the first two to extend Students being able to use the first two to extend understanding into new areas.understanding into new areas.

To achieve this, student’s learning of science should To achieve this, student’s learning of science should echo the way science is conducted by scientists. echo the way science is conducted by scientists.

(Karplus & Thier, 1967; Bybee, 1997)(Karplus & Thier, 1967; Bybee, 1997)

National Reform EffortsNational Reform Efforts

Have largely failed to shift classroom practices Have largely failed to shift classroom practices as as

reforms require fundamental shifts in the way reforms require fundamental shifts in the way teachers understand both their content and the teachers understand both their content and the appropriate ways to teach this contentappropriate ways to teach this content

new practices are perceived by many teachers new practices are perceived by many teachers and administrators as being in conflict with and administrators as being in conflict with student achievement in standardized content student achievement in standardized content testingtesting

Rationale of the ResearchRationale of the Research

Central issues for successful enactment of Central issues for successful enactment of science education reformscience education reform

Teacher professional developmentTeacher professional development Well-designed instructional and curricular Well-designed instructional and curricular

materialsmaterials Knowledge of how best to combine Knowledge of how best to combine

professional development and curriculum to professional development and curriculum to achieve desired student outcomes achieve desired student outcomes

This Research Aims toThis Research Aims to

Explore the influence of both professional Explore the influence of both professional development and the enactment of reform-development and the enactment of reform-based science curriculum on students based science curriculum on students learning and on teachers’ knowledge and learning and on teachers’ knowledge and beliefs about science and the teaching and beliefs about science and the teaching and learning of science. This paper will report learning of science. This paper will report the initial results on what we have learned the initial results on what we have learned about these topics.about these topics.

Research QuestionsResearch Questions

Does the use of such curricula plus professional development work? Does the use of such curricula plus professional development work?

If it does work, then for whom does the use of such curricula plus If it does work, then for whom does the use of such curricula plus professional development work, that is, what student subgroups professional development work, that is, what student subgroups (student effect moderators) can we identify from the data collected? (student effect moderators) can we identify from the data collected?

Under what teacher conditions do such curricula plus professional Under what teacher conditions do such curricula plus professional development work, that is, in this case what teacher subgroups development work, that is, in this case what teacher subgroups (teacher effect moderators such as teachers with more or less years of (teacher effect moderators such as teachers with more or less years of experience, those with high or low self efficacy, etc.) can we identify as experience, those with high or low self efficacy, etc.) can we identify as influential from the data collected? influential from the data collected?

How does it work, that is, what possible causal mechanisms How does it work, that is, what possible causal mechanisms (mediators) can we identify that indirectly transmit the effect of the (mediators) can we identify that indirectly transmit the effect of the treatment? treatment?

The StudyThe Study Design: A randomized-cluster experimental field trialDesign: A randomized-cluster experimental field trial

Participants: Participants: 66 Experimental Treatment teachers (PD and GEMS)66 Experimental Treatment teachers (PD and GEMS)

1418 students from their classes1418 students from their classes 59 Control teachers (District’s Text)59 Control teachers (District’s Text)

1176 students from their classes1176 students from their classes Control/Experimental group matching: within schools if possible, based on Control/Experimental group matching: within schools if possible, based on

student demographics where not possiblestudent demographics where not possible

Time: 2007-2008 and 2008-2009 school years (two cohorts)Time: 2007-2008 and 2008-2009 school years (two cohorts)

Location: 29 participating schools in a district located in central FloridaLocation: 29 participating schools in a district located in central Florida

Curricula:Curricula: Experimental: GEMS Experimental: GEMS Space Science Curriculum SequenceSpace Science Curriculum Sequence

Control: District Adopted Text’s Space Science UnitControl: District Adopted Text’s Space Science Unit

Student DemographicsStudent Demographics

4th Grade 39.4%

Grade Level 5th Grade 60.6%

Female 50%

Gender Male 50%

Free & Reduced Non FRL 68.6%

Lunch FRL 31.4%

Caucasian 62.5%

Race Other 37.5%

English Language Non ELL 97%

Learners ELL 3%

Professional DevelopmentProfessional DevelopmentExperimental GroupExperimental Group

A four day, intensive professional development prior to school yearA four day, intensive professional development prior to school year

• activities, discussions, and reflections centered around the GEMS activities, discussions, and reflections centered around the GEMS curriculum and the pedagogies that underpin itcurriculum and the pedagogies that underpin it

• explicit/reflective experiential approach throughout with respect to explicit/reflective experiential approach throughout with respect to both the curriculum and the pedagogy (pedagogy included: learning both the curriculum and the pedagogy (pedagogy included: learning cycle approach to science instruction, questioning/discussion cycle approach to science instruction, questioning/discussion strategies, evidence circles, assessment strategies, nature of strategies, evidence circles, assessment strategies, nature of science teaching strategies)science teaching strategies)

Three-hour follow up immediately prior to teaching the GEMS unitThree-hour follow up immediately prior to teaching the GEMS unitThree-hour mid-teach meetingThree-hour mid-teach meetingRegular observations and/or videotapingRegular observations and/or videotapingClassroom support by a science coach as requested Classroom support by a science coach as requested

Curriculum: Experimental GroupCurriculum: Experimental Group

Great Exploration in Math and Science (GEMS) Great Exploration in Math and Science (GEMS) Space Science Curriculum Sequence Space Science Curriculum Sequence (2007)(2007)

Designed to address the age-appropriate core concepts in space Designed to address the age-appropriate core concepts in space science (NSES, 1996) and common misconceptions that students science (NSES, 1996) and common misconceptions that students harbor about them.harbor about them.

Inductive, activity based approach to space science concepts Inductive, activity based approach to space science concepts (learning cycle format lessons).(learning cycle format lessons).

Explicit focus on the role of models and evidence in science.Explicit focus on the role of models and evidence in science. Students encouraged throughout to evaluate alternative Students encouraged throughout to evaluate alternative

explanations, to use evidence to support explanations, and to explanations, to use evidence to support explanations, and to critique the merits of an explanation. critique the merits of an explanation.

Clear directions for the teacher (material preparation, background Clear directions for the teacher (material preparation, background content information, classroom discourse dialog suggestions, content information, classroom discourse dialog suggestions, pedagogy notes).pedagogy notes).

Curriculum: Control GroupCurriculum: Control Group

District adopted textDistrict adopted text Features more didactic presentation of space science Features more didactic presentation of space science

concepts including direct instruction, reading of text, concepts including direct instruction, reading of text, with students answering very focused questions. with students answering very focused questions.

The activities included in the text typically are verification The activities included in the text typically are verification of the content already presented in the text or, more of the content already presented in the text or, more commonly, activities that are peripherally associated with commonly, activities that are peripherally associated with the topic, but that do not address the core concepts. the topic, but that do not address the core concepts.

Teachers asked to adhere to the text and to this Teachers asked to adhere to the text and to this instructional model (even if this is not their normal instructional model (even if this is not their normal science teaching practice)science teaching practice)

Teacher Construct to be Measured

Tools for Data Collection and Analysis

Space Science Content Collaboration for Astronomy Education Research (CAER) (1999). Astronomy Diagnostic Test.

Self-efficacy/Outcome expectancy/Attitude about reform practices

Teaching Science as Inquiry (TSI) (Smolleck, Zembal-Saul, Yoder, 2006).

Attitudes toward science Test of Science-Related Attitudes (TOSRA) (Fraser, 1981)

Teaching beliefs/concepts Beliefs About Reformed Science Teaching and Learning (BARSTL) (Sampson & Benton, 2006)

Science Inquiry Views of Science Inquiry (VOSI) (Lederman & Lederman, (2005)

Fidelity to the assigned instructional methodology

Reformed Teaching Observation Protocol (RTOP) (Sawada , Piburn, Kyle, &Alport, 2002)

Data AnalysisData Analysis

Exploratory statistical analysis without controlling family-wise error.Exploratory statistical analysis without controlling family-wise error.

Teacher Data:Teacher Data:

Standard single-level multiple regression models on the effects of the Standard single-level multiple regression models on the effects of the treatment on teacher outcomes from the teacher assessments treatment on teacher outcomes from the teacher assessments (Space Science Content, TSI, VOSI, BARSTL, and TOSRA)(Space Science Content, TSI, VOSI, BARSTL, and TOSRA)

Piece wise approach with multiple regression using SPSS and path Piece wise approach with multiple regression using SPSS and path analytic approach using MPlus (for mediation models)analytic approach using MPlus (for mediation models)

Student Data:Student Data:

Hierarchical Linear Modeling (HLM-6) was required to account for all Hierarchical Linear Modeling (HLM-6) was required to account for all the inter-dependencies of student outcomes within teachers the inter-dependencies of student outcomes within teachers (Raudenbush & Bryk, 2002; Raudenbush et al., 2004)(Raudenbush & Bryk, 2002; Raudenbush et al., 2004)

Multilevel Path Analysis using Mplus SoftwareMultilevel Path Analysis using Mplus Software

Findings (from Broader Study)Findings (from Broader Study)

••Student achievement on the post-tests for space Student achievement on the post-tests for space science content, knowledge about models and science content, knowledge about models and evidence, and VOSI-E, for GEMS group as evidence, and VOSI-E, for GEMS group as compared with the control group was positive and compared with the control group was positive and statistically significant. statistically significant.

••Student achievement on the delayed post-tests (5 Student achievement on the delayed post-tests (5 months ± 2 weeks) for each of these dimensions months ± 2 weeks) for each of these dimensions was in the positive direction for the GEMS group was in the positive direction for the GEMS group as compared with the control group, but only as compared with the control group, but only those for VOSI-E and knowledge about models those for VOSI-E and knowledge about models and evidence were statistically significant. and evidence were statistically significant.

FindingsFindings

Outcome UnstandardizedGEMS Effect

Standard Error

p-value Standardized GEMS Effecta

Achievement (Space Science Content)

1.93* 0.476 < 0.001 0.546*

Self Efficacy 0.243* 0.105 0.023 0.426*

Beliefs About Reform Science

Teaching & Learning

0.182* 0.044 < 0.001 0.655*

Confidence 0.161 0.147 0.275 0.202

Attitude 0.120 0.122 0.327 0.085

Views of Science Inquiry

0.131 0.169 0.439 0.140

a Standardized effects have been obtained by dividing the unstandardized GEMS effect by the outcome standard deviation.* Statistically significant at the 0.05 level

GEMS Pooled Effects for Teacher Outcomes

TeacherTeacher Moderator VariableModerator Variable

Outcome Pre-MeasureLow on

Moderator (-1 SD below the

Mean)

Average on Moderator

(Mean)

High on Moderator

(+1 SD above the Mean)

Self Efficacy 1.098* 0.446* -0.207 (NS)

Pre_Efficacy

2.0 2.5 3.0 3.5 4.0 4.5 5.0

Pre

dic

ted E

ffic

acy O

utc

om

e

3

4

5

6

7

8

9

GEMS Group

Control Group

Predicted self efficacy outcome as a function of pre-test teacher self efficacy

Standardized effects of the pre-test self-efficacy score as moderator of the GEMS Self Efficacy Effect

Estimate of the standardized simple effect of GEMS on the self efficacy outcome as Estimate of the standardized simple effect of GEMS on the self efficacy outcome as a function of pre-test self efficacya function of pre-test self efficacy

Pre_Efficacy

2.0 2.5 3.0 3.5 4.0 4.5 5.0

GE

MS

Eff

ect

on E

ffic

acy

-3

-2

-1

0

1

2

3

4

5

Region of Significance

Pre-test self efficacy (pre-Efficacy) = middle straight line. The two curved lines represent the 90% confidence band on the effect estimate. The estimated effects are statistically significant at the 0.10 level for all values of pre-test self efficacy where the confidence interval does not capture zero.

Relationship between independent, dependent Relationship between independent, dependent and mediating variablesand mediating variables

Independent Variable (A)

Dependent Variable (B)

MediatingVariable (M)

The effect of variable A on variable B exists because variable A leads to a change in a mediating variable (M), which in turn effects the dependent variable (B)

Mediation Model for Post-Test Beliefs about Reform Mediation Model for Post-Test Beliefs about Reform Science Teaching and LearningScience Teaching and Learning

sach_pr-m sach_po_m

GEMS bar_po

bar_pr

.827*

.034*.475*

.181*

.501*

Effect of GEMS on post-test teacher beliefs about reform science teaching mediated by the average post-test student space science achievement

ach_pr ach_po

GEMS bar_po

bar_pr

.781*

.010 (p = 0.11)1.93*

.161*

.527*

Effect of GEMS on post-test teacher beliefs about reform science teaching mediated by the average post-test teacher space science achievement

Discussion of Teacher DataDiscussion of Teacher DataWell-designed, reform-based curriculum combined with professional Well-designed, reform-based curriculum combined with professional

development centered around that curriculum can support the development centered around that curriculum can support the development of elementary teachers’ content knowledge as well as development of elementary teachers’ content knowledge as well as their beliefs about teaching. their beliefs about teaching.

The GEMS curriculum materials and professional development were The GEMS curriculum materials and professional development were more effective in supporting teachers who had more to learn--those more effective in supporting teachers who had more to learn--those with lower self efficacy at the outset of the study. This effect with lower self efficacy at the outset of the study. This effect lessened for teachers who had high self efficacy at the outset. lessened for teachers who had high self efficacy at the outset.

Mediation models indicate that the increase in beliefs about reform Mediation models indicate that the increase in beliefs about reform teaching and learning for the experimental (GEMS) group is teaching and learning for the experimental (GEMS) group is influenced (mediated) by two variables, student post-test space influenced (mediated) by two variables, student post-test space science content knowledge and teacher post-test space science science content knowledge and teacher post-test space science

content knowledge.content knowledge.

Student InstrumentsStudent Instruments

Construct to be Measured Tools for Data Collection and Analysis

Space Science Content MOSART resources (Sadler, Coyle, Cook-Smith, & Miller, 2007) http://www.cfa.harvard.edu/smgphp/

mosart/site_map.html

Attitudes toward science Homerton Science Attitudes Survey (Warrington, Younger, & Williams, 2000)

Models and evidence content knowledge

Knowledge about Models & Evidence (Granger, Saka, & Bevis, designed for this study, 2007)

Science Inquiry Views of Scientific Inquiry—Elementary (VOSI-E) (Schwartz, Lederman, & Lederman, 2008)

Student Outcome FindingsStudent Outcome FindingsOutcome Unstandardized

GEMS Effectp-value Standardized

GEMS Effecta

Post-Test OutcomeSpace Science Achievement

0.488* 0.002 0.171*

View of Science Inquiry

0.083* 0.002 na

Models and Evidence

No FRL 0.354* <0.001 0.682*FRL 0.261* <0.001 0.503*

Attitude Toward Science

0.009 0.753 0.014

Delayed Post-Test OutcomeSpace Science Achievement

0.187 0.193 0.067

View of Science Inquiryb

0.054* 0.047 na

Models and Evidence

0.285* <0.001 0.573*

Attitude Toward Science

0.020 0.530 0.029

* Statistically significant at the 0.10 level.a Standardized effects have been obtained by dividing the raw score GEMS coefficient by the outcome standard deviation.b The student variable of Views of Scientific Inquiry is dichotomous. The associated GEMS effect, assuming a linear probability model, is the difference in the probability of having the “Traditional + Informed” outcome for those in the experimental group versus that of “naïve” for those in the control group.

Mediation model for post-test space science Mediation model for post-test space science student achievementstudent achievement

Teacher Level

Student Level

te_pr te_po

GEMS

sach_pomod_pomod_pr

sach_pr

.767*-.799*GEMS

3.27*-.799*te_pr

.355*

-.067 (NS)

.332*

.534*

.591*

1.190*

Te_pre=pre-test teacher self efficacy; te_po=post-test teacher self efficacy; mod_pr=pre-test student models and evidence; mod_po=post-test student models and evidence; sach_pr=pre-test student space science achievement; sach_po=post-test student space science achievement

Direct, Indirect, and Total Effects of GEMS on student Direct, Indirect, and Total Effects of GEMS on student post-test space science achievement score with post-test space science achievement score with

ModerationModerationEffect Unstandardized Effect at

Pre-test teacher Self Efficacy equal to:

Standardized Effect at Pre-test Teacher Self

Efficacy equal to:

M – 1 SD

M M + 1 SD

M – 1 SD

M M + 1 SD

Total Indirect 0.619 0.485 0.352 0.219 0.171 0.124Indirect via post-test teacher self efficacy

0.224 0.090 -0.043 0.079 0.032 -0.015

Indirect via post-test student Models & Evidence

0.395 0.395 0.395 0.140 0.140 0.140

Direct -0.067 -0.067 -0.067 -0.024 -0.024 -0.024Total 0.552 0.418 0.285 0.195 0.148 0.101

a The pre-test teacher self efficacy values shown are based on a mean of 3.77 and a standard deviation of 0.47. The standardization of the effects is based on a post-test student space science achievement standard deviation of 2.83.

Predicted student post-test space science achievement Predicted student post-test space science achievement (sach_po) as a function of the teacher pre-test self (sach_po) as a function of the teacher pre-test self

efficacy measure (te_pr)efficacy measure (te_pr)

te_pr

2.5 3.0 3.5 4.0 4.5 5.0

Pred

icte

d sa

ch_p

o

7.0

7.5

8.0

8.5

9.0

9.5

10.0

GEMS

Control

[Note: The student pre-test space science achievement and student pre-test models and evidence variables have been set equal to their means.]

Mediation model for post-test space science Mediation model for post-test space science student achievementstudent achievement

Teacher Level

Student Level

te_pr te_po

GEMS

sach_pomod_pomod_pr

sach_pr

.767*-.799*GEMS

3.27*-.799*te_pr

.355*

-.067 (NS)

.332*

.534*

.591*

1.190*

Te_pre=pre-test teacher self efficacy; te_po=post-test teacher self efficacy; mod_pr=pre-test student models and evidence; mod_po=post-test student models and evidence; sach_pr=pre-test student space science achievement; sach_po=post-test student space science achievement

Discussion of Student DataDiscussion of Student Data

Well-designed, reform-based curriculum combined with professional Well-designed, reform-based curriculum combined with professional development centered around that curriculum can support the development development centered around that curriculum can support the development of elementary students’ disciplinary content knowledge, knowledge about of elementary students’ disciplinary content knowledge, knowledge about models & evidence, and views of science inquiry (cognitive outcomes). models & evidence, and views of science inquiry (cognitive outcomes).

StudentsStudents in the classrooms of in the classrooms of teachers who were low on their self efficacy test teachers who were low on their self efficacy test score score before beginning the study showed a significant increase in their post before beginning the study showed a significant increase in their post content test scores.content test scores.

Student performance on the models and evidence post-test was increased for Student performance on the models and evidence post-test was increased for non-FRL students over that of those enrolled in the FRL program, though non-FRL students over that of those enrolled in the FRL program, though both had improved performance over the control group.both had improved performance over the control group.

Mediation model indicates that the improved student content knowledge Mediation model indicates that the improved student content knowledge outcome for the experimental (GEMS) group is influenced (mediated) by outcome for the experimental (GEMS) group is influenced (mediated) by two variables, one at the student level, student post-test models and two variables, one at the student level, student post-test models and evidence performance, and one at the teacher level, teacher post-test self evidence performance, and one at the teacher level, teacher post-test self efficacy. efficacy.

How do we explain our results in light How do we explain our results in light of mixed results from previous studies?of mixed results from previous studies?

Strength of curriculum: GEMS Strength of curriculum: GEMS SSCS SSCS is is certainly a well-designed and easy to enact certainly a well-designed and easy to enact curriculum.curriculum.

Tight control of curriculum and instructional Tight control of curriculum and instructional methodology of both experimental methodology of both experimental and and control groups.control groups.

Monitoring and assessment of fidelity to Monitoring and assessment of fidelity to the assigned curriculum and instructional the assigned curriculum and instructional methodology.methodology.

ImplicationsImplicationsOur results suggest that what teachers learn from Our results suggest that what teachers learn from professional development and teaching with the professional development and teaching with the curriculum varies according to what their knowledge, curriculum varies according to what their knowledge, beliefs, and/or attitudes are entering the experience. beliefs, and/or attitudes are entering the experience.

Reform-based instruction using a specified curriculum in Reform-based instruction using a specified curriculum in combination with professional development results in combination with professional development results in positive effects on student cognitive outcomes as positive effects on student cognitive outcomes as compared with more traditional instruction. compared with more traditional instruction.

We further suggest that at least some of the mixed We further suggest that at least some of the mixed results found in previous research may be due to results found in previous research may be due to experimental design, especially in assessing fidelity to experimental design, especially in assessing fidelity to the curriculum and/or instructional methodology in the curriculum and/or instructional methodology in bothboth the treatment the treatment andand control groups. control groups.

Future ResearchFuture ResearchAre there mediation models for other student outcomes besides Are there mediation models for other student outcomes besides improved content achievement?improved content achievement?

How does the use of models and evidence by the curricula mediate the How does the use of models and evidence by the curricula mediate the increase in student content test scores?increase in student content test scores?

How do low socioeconomic students need to be supported to close the How do low socioeconomic students need to be supported to close the achievement gap?achievement gap?

What are the individual roles of professional development and What are the individual roles of professional development and curriculum enactment in the student outcomes of this study? curriculum enactment in the student outcomes of this study?

What is the reason behind the result that teachers with low pre-test self What is the reason behind the result that teachers with low pre-test self efficacy scores garnered such an increase in the test scores of their efficacy scores garnered such an increase in the test scores of their students?students?

How many classroom samples are needed to reliably assess fidelity?How many classroom samples are needed to reliably assess fidelity?

Would similar findings be found in secondary-school classrooms Would similar findings be found in secondary-school classrooms (where teachers usually have an increased science content knowledge (where teachers usually have an increased science content knowledge and higher teaching self-efficacy)?and higher teaching self-efficacy)?

AcknowledgementsAcknowledgements We are deeply indebted to Dr. Richard Tate, our We are deeply indebted to Dr. Richard Tate, our

statistician, for his talent, help, instruction, and advice with statistician, for his talent, help, instruction, and advice with the statistical analyses in this paper; to our research the statistical analyses in this paper; to our research assistants, Francesca Gerbino and Katrina Dial for their assistants, Francesca Gerbino and Katrina Dial for their assistance with data management; to Dr. Victor Sampson assistance with data management; to Dr. Victor Sampson for reviewing the manuscript; and to Claudia Zimmerman for reviewing the manuscript; and to Claudia Zimmerman for her clerical support. The staff of the Florida Center for for her clerical support. The staff of the Florida Center for Research in STEM Education was essential in helping us Research in STEM Education was essential in helping us to administer this project. We wish to acknowledge the to administer this project. We wish to acknowledge the significant contribution of our science coach, Tracie significant contribution of our science coach, Tracie Ragland, to the project. Finally, the project is indebted to Ragland, to the project. Finally, the project is indebted to the teachers who agreed to constrain their teaching to the the teachers who agreed to constrain their teaching to the assigned methodology and curriculum in order to further assigned methodology and curriculum in order to further knowledge about their profession.knowledge about their profession.

Copies of this paper may be Copies of this paper may be downloaded from:downloaded from:

http://ret.fsu.edu/papers_10.html http://ret.fsu.edu/papers_10.html