Jinnie Choi Yong Sang Lee Karen Draney University of California at Berkeley

Jinnie ChoiYong Sang LeeKaren Draney

University of California at Berkeley

2009 AERA Annual Meeting, San Diego

April 14, 2009

Learning Progression of Carbon Cycle

Complications in Validation Principle- and Process-based Multidimensionality Rater Effect

Modeling Multidimensionality and Rater Effect

Understanding the Results

Next Steps2009 AERA Annual Meeting






The Carbon Cycle Project A part of the Environmental Literacy Project funded

by National Science Foundation The goal is to integrate Environmental Science

Literacy into contemporary K-12 curriculum

Big Idea: Environmental Literacy Environmentally literate students are expected

to be able to apply fundamental principles to processes in coupled human and natural systems

▪ Systems: coupled human and natural systems▪ Principles for scientific accounts: scale, conservation of

matter (both mass and atoms), conservation of energy and energy degradation

▪ Processes: generation, modification, and oxidation of organic carbon

2009 AERA Annual Meeting

Carbon Cycle Learning Progression Framework

Learning progressions are “descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn about and investigate a topic over a broad span of time” (Duschl, Schweingruber, & Shouse, 2007)

Lower anchor, intermediate, and upper anchor understandings that define levels

Two principle-based dimensions that define progress variables

Six process-based dimensions2009 AERA Annual Meeting

Levels of achievement (performances)


Upper Anchor

LevelsCharacteristics of Responses (Food chain

example)

4: Qualitative model-based

accounts

Responses describe food chains at an atomic-molecular level, identifying key organic and inorganic materials that flow between organisms, and key processes responsible for changes in matter

3: “School science” narratives

Responses attempt to explain food chains at atomic-molecular level by either identifying materials that connect or move between organisms, but does not consistently trace matter at a-m level and may confuse matter and energy flow

2: Force-dynamic with

hidden mechanisms

Responses construct food chains that connect organisms by materials such as "food" or organisms being eaten by other organisms.

1: Force-dynamic

narratives

Responses recognize that living organisms are related because they all exhibit qualities of being alive and depend on each other.

Lower Anchor

Carbon Cycle Learning Progression Framework


Upper Anchor

Lower Anchor

(Process Dimension)

Tracing Matter / Tracing Energy(Principle Dimension)

Photosynthesis

Digestion/Growth

Cellular Respiratio

n

Decompo-sition

Combustion

Cross Process

4: Qualitative model-based

accounts

3: “School science”

narratives

2: Force-dynamic with

hidden mechanisms

1: Force-dynamic

narratives






Principle-based Multidimensionality Tracing Matter and Tracing Energy Progress variables are described separately and

used as scoring rubrics

Process-based Multidimensionality Photosynthesis, Digestion/growth, Cellular

respiration, Decomposition, Combustion, and Cross-process

More like item groups, but crossed with Principle dimension

Rater Effect Multiple raters rated different combinations of

groups of people and sets of items2009 AERA Annual Meeting

Validation of a Learning Progression Levels of performances▪ How are the students distributed across levels of

performances?▪ Do the items capture students’ lower level performances as

well as higher level performances?

Dimensional structure▪ Do dimensions exist? Are the dimensions statistically

distinguishable? ▪ What are the correlations between students’ performances on

different latent dimensions?▪ How consistently/differently do students perform on different

sets of items that measured different latent dimensions?▪ Is a set of items more difficult or easier when measuring one

dimension than others? 2009 AERA Annual Meeting






Multidimensional Random Coefficient Multinomial Logit Model (MRCML; Adams, Wilson & Wang, 1997)

Confirmatory analysis of between-item multidimensionality

Multifaceted Item Response Model (Linacre ,1994)

Examine variation in the harshness or leniency of raters

Examine the fit (or consistency) of individual raters with other raters







Levels of Performances


Principle-based Multidimensionality Relative fit test


Model Deviance # of parameters

UnidimensionMultidimension

22900.0822862.42

110112

Difference 37.66 2

Chi2(37.66, 2) = 5.272e-18

Principle-based Multidimensionality

Item difficultyestimates


Principle-based Multidimensionality Person ability estimates


DimensionDimension

Matter Energy

MatterEnergy

1.2310.959 1.335

EAP Reliability 0.623 0.614


Comparison of rater harshness and weighted fit







More measurement questions…▪ How can we model responses if a rater rates differently for

items that measure different dimensions?▪ How can we model responses if a rater rates differently for

items that requires different scoring criteria?▪ How can we model responses when an item is measuring one

process dimension but doubly scored based on principle dimensions?

▪ How do we model responses when a set of item is differently perceived by different groups of people?

Informing and revising the four building blocks (Wilson, 2004) …▪ Construct map▪ Items design▪ Outcome space▪ Measurement models


Environmental Literacy research groups▪ Michigan State University▪ Long Term Ecological Research (LTER) Network▪ University of California at Berkeley▪ University of Michigan▪ Northwestern University▪ AAAS Project 2061

Visit our websites at…▪ Environmental Literacy website▪ http://edr1.educ.msu.edu/EnvironmentalLit/index.htm

▪ Berkeley Evaluation and Assessment Research (BEAR) Center

▪ http://bearcenter.berkeley.edu


Thank You!

Documents

Jinnie Choi Yong Sang Lee Karen Draney University of California at Berkeley