Evaluating the Validity of Student Learning Objectives

KATIE BUCKLEY, HARVARD UNIVERSITY

SCOTT MARION, CENTER FOR ASSESSMENT

NATIONAL CONFERENCE ON STUDENT ASSESSMENT (NCSA)

NATIONAL HARBOR, MDJUNE 22, 2013

Evaluating the Validity of Student Learning

Objectives

Overview of Presentation2

Theory of Action for SLOs

Validity Argument for SLOs tied to ToA

Research Agenda based on Validity Argument

Buckley & Marion. CCSSO 2013

What is an SLO?

Content- and grade/course-specific measurable learning objectives that can be used to document student learning by a teacher over a defined period of time

Designed to involve teachers throughout the process and incentivize good teaching practices

Can be used as both an accountability tool and as an instructional tool

3


Why do we care?


4

SLOs currently used in teacher evaluation systems by for teachers in non-tested grades and subjects (NTSG) Upwards of 20 states/districts are using SLOs Nearly 70% of teachers teach in NTSG

However Very little research currently exists Much evidence is needed to substantiate the implicit

and explicit claims to support the use of SLOs in educator evaluations

Theory of Action as a Starting Point

A theory of action (ToA) is a useful way to structure a validity evaluation in complex contexts…

Researchers have suggested (and we believe) that a theory of action can*: Frame the validity evaluation, Serve as a useful starting point in creating a validity

argument, Address the program evaluation and consequential aspects of

the system not easily addressed through a validity argument

*(Bennett, 2010; Marion & Pellegrino, 2006; Marion, 2010; Ryan, 2002).

5


The Challenges with Creating a ToA for SLOs

SLOs are more than an assessment or program

Student “growth” is embedded within the SLO process rather than measured externally

It is important to convey how teachers should be involved in the SLO process

SLOs are flexible and may be adapted to a variety of systems

6


Components of an SLO


7

SLO is a framework that includes: Meaningful learning expectations (goals) High quality assessments Targets for student performance Targets for aggregate educator performance Instruction from teachers

Therefore, an evaluation of SLOs must address all five components

Theory of Action for SLOs in a Teacher Evaluation System

8


Expanding the ToA with a Validity Argument

We expand the ToA to create a validity argument based on Kane’s 2006 work

We explicate for each condition/input: Propositions Claims Example Evidence

Propositions are organized according to Scoring Generalization Extrapolation Decision Consequence*

9


Input 1: Student learning goal is rigorous yet attainable


10Proposition Claim Example of EvidenceScoring Goals are of high

qualityDocumentation of quality check on the rigor of the rubric

Evidence that goals were evaluated consistently, reliably and accurately

Generalization

Learning goals reflect the relevant content standards and the associated curriculum of the course

Review of the importance and alignment of established learning goals by content experts

Extrapolation The learning goals are representative of the broader construct of student learning within the classroom

Clear documentation on why the learning goal is an important one for students to learn, ways in which the learning goal requires students to demonstrate deep understanding of the big idea being measured, and ways in which the learning goal ties to overarching goals for the course.

Decision Attainment of the learning goals can be used to classify teacher facilitation of student learning

Evidence that higher scores on SLOs reflects greater student learning

Evidence that higher scores on SLOs reflects more focused teaching of important content frameworks

Consequence The SLO does not lead to a narrowing of the curriculum

There is no evidence to suggest that that instruction has narrowed to only the domains covered by the SLO

Input 2: Assessments accurately and reliably measure student learning of goals


11

Proposition Claim Example of EvidenceScoring Student performance is

accurately and consistently scored

Rater accuracy and consistency studies are clearly documented

Generalization Assessment items are representative of broader domain

Reliability/generalizability analyses

Content validity studiesExtrapolation Assessments are/is tailored to

match the specific learning goal

Documentation of content validity studies between assessment content and learning goal

Decision Performance on the assessment(s) can be used to classify student attainment of learning goal

Evaluation of conditional standard errors of measurement around key cutscores

Misclassification analysesConsequence The SLO does not lead to

undesirable behavior such as student cheating on the assessment

SLO scores in the state correlate strongly with other measures of student achievement particularly from low stakes tests measuring similar content

Input 3: Targets set for students are appropriate and valid


12

Proposition Claim Example of EvidenceScoring Determination of whether

student meets target is accurately classified

Business rules for classifying attainment of student targets are clearly defined.

Evidence that student targets are fair and consistent across grades/subjects.

Generalization Student attainment of target provides evidence of student’s ability on learning goal

Evidence that student targets are set in a thoughtful manner based on baseline data and prior student performance

Extrapolation Student attainment of target provides evidence of overall student achievement

Student performance on SLO is strongly correlated with other measures of student performance

Decision Student attainment of target can be used to determine the teacher score on the SLO

Business rules for aggregating student targets to the teacher level are clearly defined.

Consequence Student attainment of the target is the result of student learning and not of undesirable behavior such as teaching to the test

SLO scores in the state correlate strongly with other measures of student achievement particularly from low stakes tests measuring similar content

Input 4: Targets set for teachers are appropriate and valid


13Proposition Claim Example of EvidenceScoring Determination of whether

teacher meets target is accurately classified

Evidence that teacher targets are fair and consistent across grades/subjects.

Final teacher SLO ratings are strongly correlated with the percent of students who meet their SLOs.

Generalization Attainment of target provides evidence of teachers’ contribution to student learning in the classroom

Evidence that teacher targets were chosen based on expert review of degree of student learning considered to be sufficient/exemplary for the teacher within specified time frame.

Extrapolation Attainment of target provides evidence of overall teaching effectiveness

Evidence that final teacher SLO rating correlates strongly with other measures of teacher effectiveness

Decision The teacher or teachers being evaluated on the SLO is/are the one(s) primarily responsible for the student’s achievement

Evidence that teacher developing and evaluated on the SLO is the primary teacher of record in the classroom.

Consequence Student attainment of SLO is not due outside factors beyond the teacher’s control and influence

Evidence that student attainment of SLO is no more than weakly correlated with constructs that should be unrelated to SLO attainment.

Input5: SLO is adequately tied to teacher instruction


14

Proposition Claim Example of EvidenceScoring Teachers appropriately connect

standards to learning goals when setting SLOs

Teachers document the curricula standards that are tied to the learning goals and which methods of teaching will be used to teach those standards

Generalization

Teachers have knowledge and skills to reflect on student data as a way to determine students’ mastery of the curriculum

Evidence that educators are able to use assessment results to make data driven decisions regarding their instruction

Extrapolation Teachers modify/alter instruction in direct response to data indicating students are off target from meeting their SLO

Teachers indicate that the SLO process impacts their instructional plans throughout the school year.

Decision Higher scores on SLOs reflects more focused teaching of important content frameworks

Student attainment of SLO is related to expert ratings of teacher instructional practice

Consequence Teacher mobility/attrition among highly effective teachers does not increase as a result of SLO implementation

There is no evidence of teacher mobility/attrition among highly effective teachers

Prioritizing Studies


15

It would be impossible to conduct studies for each claim listed, so how to prioritize? State’s chosen priorities Analyses that provide monitoring results for system-

wide improvement Analyses that address multiple claims

Need to also consider short-term vs. long-term analysesWhat can be done/should be done now based on available

data vs. later when the system has produced more data?

Research Agenda


16

Three Studies 2 based on quantitative data currently available 1 based on qualitative data not yet available

Will help to determine whether: SLOs are being implemented as intended Results that are produced are as hypothesized

Address multiple claimsAnalyses that can be used by GA to respond

to USED’s requests

Research Agenda


17

Study 1: Are there differences in the quality and attainment of SLOs: By district, grade, course, course-type? Across school and classroom

demographics/characteristics? Over time?

Claims addressed Goals are of high quality (Learning Goal input, Scoring

proposition) Student attainment of SLO is not due outside factors beyond the

teacher’s control and influence (Teacher Target input, Decision proposition )

Research Agenda


18

Study 2: What is the relationship between attainment of SLOs and other measures of teacher effectiveness, including: Student growth? Teacher practice?

Claims Addressed: Attainment of the learning goals can be used to classify

teacher facilitation of student performance (Learning Goal Input, Decision proposition)

Student attainment of target provides evidence of overall student achievement (Student Target input, Extrapolation proposition)

Attainment of target provides evidence of overall teaching effectiveness (Teacher Target input, Extrapolation proposition)

Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Decision proposition)

Research Agenda


19

Study 3: How do teachers and school leaders understand the SLO system and use the system for instructional purposes?

Claims addressed: Teachers modify/alter instruction in direct response to data

indicating students are off target from meeting their SLO (Teacher Instruction input, Generalization proposition)

Higher scores on SLOs reflects more focused teaching of important content frameworks (Teacher Instruction input, Extrapolation proposition)

Thank you


20

Questions/comments: Katie Buckley ([email protected]) Scott Marion ([email protected])

Documents

Evaluating the Validity of Student Learning Objectives