A Process for Analyzing Coherence in Science Instructional Materials Presenter: April Gardner

Collaborators: Joseph A. Taylor, BSCS; Rodger W. Bybee, BSCS; Liu Enshan, Beijing Normal University

2 May 2010 AERA Annual Meeting Denver, CO

Project Background

•  Asia Society convened meetings of American and Chinese math and science education leaders in mid-2000s

•  This group encouraged collaborative work to examine and compare curriculum standards and materials in the two countries

•  In 2006, BSCS received a grant from the Office of Science Education at NIH to collaborate with Beijing Normal University biology educators to compare U.S. and Chinese biology textbooks

Initial project goals

Compare U.S. and Chinese secondary biology programs for: •  Foundational concepts •  Role of scientific inquiry •  Role of educational technology

Refining the goals

Goals were refined and expanded based on our recognition that: •  both the U.S. and China had developed and adopted national

science curriculum standards in the recent past (1996 for U.S. and 2003 for China),

•  both countries had popular, pre-standards programs in use as well as less-popular, post-standards programs in use, and

•  it would be interesting to compare not only distinctions between American and Chinese instructional materials, but also distinctions between pre- and post-standards curriculum materials.

An emphasis on curricular coherence

As we continued this work, we further recognized that: •  international comparison studies identified curricular coherence as a

key distinction between high- and low-performing countries, •  focus and rigor have been described as two components of overall

coherence (Schmidt et al., 2005; Schmidt & Prawat, 2006), and •  focus and rigor are key features of science curriculum standards in

both the U.S. and China.

Thus, our emphasis changed to •  analysis of the coherence (focus and rigor) of one pre-standards

and one post-standards biology program from each country

A more interesting story

When we completed this work, we recognized that we had developed and tested a method for analyzing the coherence of instructional materials, and

that’s the more interesting story I’ll relate today . . .

Four programs reviewed

Process for evaluating coherence: Background

•  Adaptation of part of the Analyzing Instructional Materials (AIM) process (Powell et al., 2002)

•  Uses criteria based on research about how students learn science •  These criteria address both focus and rigor

Process for evaluating coherence: Analyzing focus

1.  Construct and score a conceptual flow graphic (CFG) •  Write the overarching concept

of chapter and major concept(s) of each section

•  Depict strength of connections among these using arrows of varying thicknesses

•  Score by giving each connecting link a score based on its strength: 2 for strong, 1 for moderate, and 0 for weak or no link

•  CFG score = sum of all links, divided by sum if all links are strong

•  Transform % score to a score of 5, 3, or 1

Example:

Percent Score

33% weakly focused

1

33-66% moderately focused

3

>66% strongly focused

5

Process for evaluating coherence: Analyzing focus, continued

2.  Search instructional materials for evidence of focus based on sequencing, context, and key concepts.

3.  Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 (strongly, moderately, or weakly address criterion, respectively).

Criterion Abbreviated Description

Sequencing -organized to promote student understanding -links facts and concepts explicitly -builds & extends previously developed concepts -connects concepts strongly to overarching conceptual framework

Context -presented in engaging context related to real world experiences -facilitates assimilation/reorganization of knowledge that builds on students’ prior conceptions and experiences

Key Concepts -structured using global, unifying themes of biology -emphasizes connections across discipline areas

Process for evaluating coherence: Analyzing rigor

4.  Search instructional materials for evidence of rigor based on depth of treatment, engagement of prior knowledge, student metacognition, and abilities and understandings of scientific inquiry.

5.  Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 as for the focus criterion.

Criterion Abbreviated Description Depth of treatment

-treat concepts at developmentally appropriate level -require students to apply & demonstrate understanding in >1 way

Engaging prior knowledge

-help students make current understanding of concept explicit -challenge/confront current thinking about concept

Metacognition -include strategies to help students assess their own learning -include strategies to help students reflect on what & how they learned

Abilities of sci. inquiry

-require students to design & conduct scientific investigations -require students to formulate & revise explanations based on data

Understandings of sci. inquiry

-help students understand that scientists conduct investigations for a variety of reasons -help students understand that scientists use a variety of tools, technology, & methods to enhance their investigations

Process for evaluating coherence: Scoring

6.  Focus, rigor, and overall coherence scores determined as indicated in the table:

Component Criteria Possible Range

Low Moderate High

Focus CFG Sequencing

Context Key Concepts

4 – 20 4 – 8 9 – 15 16 – 20

Rigor Depth of treatment Prior knowledge Metacognition

Abilities of sci. inquiry Understandings of

sci. inquiry

5 – 25 5 – 11 12 – 18 19 – 25

Overall Coherence

Focus & Rigor Subscores

9 – 45 9 – 20 21 – 32 33 – 45

Research question

•  Can the process we devised for assessing coherence distinguish instructional materials developed before the adoption of national science standards from those developed after the adoption of the standards?

Hypothesis: Science instructional materials developed after the adoption of national standards will exhibit greater levels of coherence than those developed before the adoption of national standards.

Procedure followed

1.  Researchers were trained in the use of the modified AIM process using an unrelated program.

2.  Project director selected the genetics and ecology chapter(s) to be analyzed from each of the four programs.

3.  Three researchers independently reviewed the chapter(s), identified the overarching chapter concept and section concepts, and collected evidence to support their scores for the focus and rigor criteria.

4.  The research team met to compare and reach consensus on each overarching chapter concept and the major section concepts.

5.  The team then collaboratively constructed a CFG for each chapter. 6.  The team discussed and revised scores on the focus and rigor

criteria to reach consensus on the scores.

Sample CFG: Program B 50% of possible strong links

Moderately focused (score = 3)

Sample CFG: Program D 89% of possible strong links Highly focused (score = 5)

Example of evaluating rigor for “understandings of scientific inquiry”

•  Weak rigor (score = 1) –  The chapter described few or no scientists and their work. There is no

opportunity for students to connect how the evidence of scientists’ investigations leads them to their conclusions.

•  Strong rigor (score = 5) –  The chapter described how the understanding of inheritance progressed

over time, through the ensuing work of scientists. For example, following a detailed description of Mendel’s experiments, results, and elucidation of the principles of inheritance, the text goes on to explain how some results of genetic crosses conducted by other scientists did not fit Mendel’s patterns. The text presents this puzzle as a rationale for the subsequent work of T.H. Morgan who related the inheritance of genes to newer understandings about chromosomes and their movement. Morgan’s work then led to further studies about gene linkage and mapping.

Findings

Program/type Country Average Focus Score/Level

Average Rigor Score/Level

Overall Coherence

A/pre-standards China 12/Moderate 7/Low 19/Low

B/pre-standards USA 14/Moderate 11/Low 25/Moderate

C/post-standards China 14/Moderate 11/Low 25/Moderate

D/post-standards USA 20/High 23/High 43/High

•  The process distinguished ecology chapters from the programs developed before the adoption of national science standards from the U.S. program developed after the adoption of standards, based on their levels of coherence.

•  The process did not distinguish the U.S. ecology chapters developed before national standards from the Chinese ecology chapters developed after the national standards were adopted.

•  Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.

Ecology chapters:

Findings Program/type Country Average Focus

Score/Level Average Rigor

Score/Level Overall

Coherence A/pre-standards China 10/Moderate 9/Low 19/Low

B/pre-standards USA 4/Low 9/Low 13/Low

C/post-standards China 18/High 17/Moderate 35/High

D/post-standards USA 17/High 21/High 38/High

•  The process distinguished genetics chapters from programs developed before and after the adoption of national science standards, based on their levels of coherence.

•  Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.

Genetics chapters:

Conclusions

•  This adaptation of the AIM process is a promising method for evaluating the coherence of science instructional materials.

•  This method can be used in research studies, curriculum development, and professional development. For example, it could be used: –  to study the relationship between the coherence of instructional

materials and specific student learning outcomes, –  by curriculum developers to strengthen the coherence of

materials they are developing, and –  in curriculum-based professional development to identify areas

in programs where coherence should be strengthened by the teacher.

References cited on slides

Powell, J. C., Short, J. B. and Landes, N. M. (2002) Curriculum reform, professional development, and powerful learning. In R. W. Bybee (ed.), Learning Science and the Science of Learning (Arlington, VA: NSTA Press), 121 -- 136.

Schmidt, W. H. and Prawat, R. S. (2006) Curriculum coherence and national control of education: issue or non-issue? Journal of Curriculum Studies, 38 (6), 641 -- 658.

Schmidt, W. H., Wang, H. C. and McKnight, C. C. (2005) Curriculum coherence: an examination of US mathematics and science content standards from an international perspective. Journal of Curriculum Studies, 37 (5), 525 -- 559.

5415 Mark Dabling Blvd. Colorado Springs, CO 80918 info@bscs.org 719.531.5550

To download this presentation, visit www.bscs.org/sessions