Compendium of Research Instruments for STEM Education

PART I : Teacher Practices , PCK and Content Knowledge

Written for CADREby Abt Associates (Daphne Minner, et. al.), 2012

In 2011, the National Science and Technology Council reviewed how 13 federal agencies spent $3.4 billion in fiscal year 2010 to support STEM education.

NSF was found to have made the largest investment in STEM education, and its DRK-12 program had the largest budget of its 6 educational research and development programs.

The compendium reviewed here (Part 1 of 2) focuses on 5 cohorts of DRK-12 projects (2008-2012) that utilized instruments designed to assess teacher practices, pedagogical content knowledge, and content knowledge.

The purpose of this compendium is to provide an overview on the current status of STEM instrumentation commonly being used in the U.S and to provide resources useful to research and evaluation professionals.

Synopsis

Research Question: What are the instruments, constructs, and methods being used to study teacher outcomes within the DR-K12 portfolio?

Only extant, named instruments (as opposed to instruments being developed as part of a current proposal) were included.

Two Phases:◦ Phase 1: A review of all proposals funded by DRK-12 2008-

2012 revealed 295 eligible projects.

◦ Phase 2: Data collection was conducted for instrument-specific information about reliability and validity evidence, development and piloting, accessibility of the instrument, administration, and constructs measured.

Methods

Since CADRE is funded as a cooperative agreement rather than a contract, they were unable to access Fastlane files and relied on materials provided by PIs.

For 36 projects, materials were unavailable.

For 8 of the 57 PCK instruments, the actual instruments were unavailable.

6 instruments required purchasing.

Limitations

75 projects proposed to measure teacher practices, PCK, or Content: 71% measured only 1outcome, 24% measured 2 outcomes, and 5% measured all 3.

Instruments Identified

◦Practices: 42

◦PCK: 24

◦Content Knowledge: 27

Cross-Cutting Comparisons (Leah)

5 Categories of Instruments

1. Instructional Practices (Appendix A)

2. Instructional Practices plus Additional Constructs (Appendix B)

3. Instructional Beliefs (Appendix C)

Multidimensional

4. System-wide Reform Focused (Appendix D)

5. Discourse Focused (Appendix E)

Cross-Cutting Comparisons

Need to be more cognizant about providing relevant psychometric information on the tools used and developed in order for others to reliably implement the tools in their own projects.

Instruments developed must go through rigorous reliability and validity

Initial step towards the systematic assessment and improvement of STEM research tools.

Cross-Cutting Comparisons: Conclusions

Eleven instruments that primarily assessed classroom instructional practices:

Seven observation protocols, Three rubrics, One survey

Predominantly designed for pre-k through middle school teachers (6, 55%)

More focused on science (5, 45%) than mathematics (3, 27%) or technology (2, 18%).

The three science observation protocols capture variables ranging from the lesson’s temporal flow and percentage of time students spend in different types of groupings, to the extent of opportunity for students to engage in the various phases of the investigation cycle.

Instructional Practices (Pete)

The two science scoring rubrics are intended to be applied to lesson artifacts and instructional materials that the teacher provides students. They contain codes for student grouping, structure of lessons, use of scientific resources, hands-on opportunities through investigation, cognitive depth of the materials, encouragement of the scientific discourse community, and opportunity for explanation/justification, and connections/applications to novel situations.

Across these eleven instruments, one had low reliability evidence, and four (36%) had acceptable or good evidence. For only two instruments was the team able to find validity evidence.

Instructional Practices

Two Examples Related to Science Instructional Strategies Classroom

Observation Protocol ◦ Identifying sense of purpose; asking account of student

ideas; engaging students with relevant phenomena; developing and using scientific ideas; promoting student thinking about phenomena, experiences, and knowledge

Scoop Notebook – Artifact rubric◦ Portfolio assessment that captures: grouping, structure of

lessons, use of scientific resources, hands-on, inquiry, cognitive depth, scientific discourse community, explanation/justification, assessment, connections/applications

11 instruments that measure instructional practices in addition to one or two other constructs, meaning:◦ physical context◦ demographics◦ teacher content knowledge ◦ an aspect of classroom management

This more comprehensive nature is also reflected in the subject domains being assessed—◦ 2 each, mathematics and science◦ 5 both mathematics and science ◦ 1 technology◦ 1 general teaching skills

Instruments to Determine Teacher Practices Plus One or Two other Constructs (Michele)

Exist for many subjects. Middle School version tests all sciences very generally, whereas high school breaks it apart by specific domains

Sit-down test, 4 hours, 50 multiple-choice questions and 2 constructed-response questions

The test is designed to provide evidence that an examinee has a basic working knowledge of teaching foundations

Instruments to Determine Teacher Practices Plus one or two other Constructs

Ratings are made after at least 3 hours of observation

Ratings for each item are made on a 7-point scale. Behavioral descriptors are present at the 1, 3, 5, and 7 levels.

Assesses the materials and instructional supports for math and science learning present

Instruments to Determine Teacher Practices Plus one or two other Constructs

Instruments to Determine Teacher Instructional Beliefs (Yann)

Name What’s measured Type Validity

EvidenceGRADE LEVEL

Mathematics Teaching Efficacy Belief Instrument (MTEBI, 2000)

Modified from STEBI Riggs (California State U) & Enochs (Kansas State U),1990

Personal math teaching efficacy (13 items) & math teaching outcome expectancy (8 items)

Extent to which teachers believe they have the capability to positively affect student achievement

Survey (Likert 5-point scale)

Construct

MTEBI:21 items, PRESERVICE(STEBI:25 items, ELEMENTARY)

PS: Why is VNOS –C included?

Instruments to Determine Teacher Instructional Beliefs

Name What’s measured Type Validity

EvidenceGRADE LEVEL

Principles of Scientific Inquiry-Teacher (PSI-T)

Campbell, Chapman (Utah State U) & Abd-Hamid (U of Iowa), 2010

Teacher (& student) perceptions of frequency of occurrence when students are responsible for each of 5 principles of scientific inquiry (NRC)

Extent to which students are experiencing inquiry in science classrooms

Survey (5-point, 20 items)(teacher & student version)

ContentConstruct

HIGH

Instruments to Assess System-wide Reform Efforts

Name What’s measured Type Validity

EvidenceGRADE LEVEL

Local Systemic Change Classroom Observation Protocol (LSC)

Inside the Classroom: Teacher Interview Protocol

Overall quality of observed math/science lesson: lesson design; implementation; math/science content; classroom culture; likely impact on students' understanding

Teachers' perceptions of factors that influenced selection of lesson content and pedagogy

Observation (5-point scale)

Content (Good inter-rater % reliability)

K-12

Interview

Horizon Research Inc., 2000

Thirteen instruments looking at instructional practices and social aspects of classroom community (including class management).

Observation protocols Six are non-domain specific Three are math-specific Three are science-specific One measures both Seven demonstrated more than one type of

validity (more than other categories)

STEM Discourse (Matt)

Three scales◦ Lesson design implementation◦ Content ->PCK->Propositional and Procedural◦ Classroom culture (e.g., egalitarian s-t

relationship) High interrater % agreement High Validity

◦ Construct◦ Content◦ Predictive

Reformed Teaching Observation Protocol (RTOP)

Three domains◦ Emotional◦ Classroom organization◦ Instructional support

High internal consistency High interrater % agreement Content validity

Classroom Observation of Early Mathematics Environment and Teaching (COEMET)

25 Content Tests, 12 General tests, 8 Science, 3 Math, 1 Science and Math, 1 Technology.

◦ General Tests: American College testing, GRE, ITBS-Iowa Test of Basic Skills, NAEP, PISA, PRAXIS, WEST-E.

◦ Science: MOSART, FACETS, IL Certification Testing System Study Guide-Science, FCI Force Concept Inventory Assessment, DTAMS-science: Diagnostic Science Assessment for Middle School Teachers, Classroom Test of Scientific Reasoning (Lawson).

◦ Math: MKT, M-SCAN, DTAMS-math: Diagnostic Math Assessment for Middle School Teachers.

◦ Science and Math: TIMSS

◦ Technology: TAGLIT: Taking a Good Look At Instructional Technology

12 of them Student Test, 9 Teacher Test, 2 Survey, 1 observation Tool, 1 Student test and Teacher Tools.

7 K-12 level, 4 elementary and middle, 4 Postsecondary, 3 high school, 2 middle, 1 grades4-9, 1 elementary, 1 middle and high, 1 high and postsecondary, 1 no level indicated.

Survey Instruments to determine Teacher Content Knowledge (Arzu)

Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.

Paper-and-pencil format Pre- and post-tests before and after workshops To determine growth in teachers' content knowledge To be completed by test-takers within an hour. Each assessment has 3-4 science sub-domains. Available for use free of charge. Scored for a fee of $10 per teacher per assessment -includes scores on

individual items, on each science sub-domain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)

• http://louisville.edu/education/centers/crmstd/diag-sci-assess-middle - http://louisville.edu/education/centers/crmstd/diag-sci-assess-middle

DTAMS-Science

Free and can be accessed after completion of four online tutorials that explain test design, use, scoring, and interpretation of results.

A set of multiple-choice items include K–12 physical science and earth science content, and K–8 life science content in the NRC NSES as well as to the research literature about misconceptions concerning science concepts.

MOSART:Misconceptions-Oriented Standards-Based Assessment Resources for Teachers

ASW – Analysis of Student Work: A rubric is used to score teachers’ evaluations of a standardized set of video cases of student problem solving.

LoU – Levels of Use Interviews: An interview determines how a change is being implemented in the classroom.

SEPUP - Group Interaction and Communication of Scientific Information Rubrics: Rubrics are used to grade student work on a variety of measures including how they design and conduct an investigation, analyze data, understand concepts, evaluate evidence and identify tradeoffs, communicate scientific information, and work cooperatively in a group.

Unclassified Instruments (Jenay)

Detailed access information can be found for each instrument in Appendices H & I of the compendium.

Part 2 of the compendium (not covered here) details measurement of students’ content knowledge, reasoning skills, and psychological attributes.

Conclusion