Incorporating Contextual Information in Standard Setting...– Three year cycle (2012, 2015, 2018)...

Incorporating Contextual Information in Standard SettingGary W. Phillips, VP, American Institutes for ResearchVince Verges, Assistant Deputy Commissioner, FloridaIrene Hunting, Deputy Associate Superintendent, ArizonaJames Wright, Director, Office of Curriculum and Assessment, Ohio

June 20, 20162016 CCSSO/NCSA

June 2016

AMERICAN INSTITUTES FOR RESEARCH

What is Contextual Information in Standard Setting?• Data used to inform panelist decisions above and beyond

the content standards, performance level descriptions and the test items

Impact Data• State item P-values • Item Maps using Response Probabilities (RPs)• Overall state inverse cumulative percentages• State inverse cumulative percentages by demographic

subgroup

PG1 Phillips, Gary, 6/9/2016

Articulation• Use student frequency distributions to smooth cut-scores

across grades• Use vertical scale to smooth cut-scores across grades

National Benchmarks• ACT/SAT• NAEP• National Norm-Referenced Tests

International Benchmarks• PISA

– Three year cycle (2012, 2015, 2018)– Reading, Mathematics & Science– Age 15

• TIMSS– Four Year cycle (2011, 2015, 2019)– Mathematics and Science– Grades 4 & 8

• PIRLS– Five year cycle (2011, 2016, 2021)– Reading Literacy– Grade 4

Benchmarking Methodology

• Linking–Item Calibration

» Common item linking (e.g., PISA in Delaware, Hawaii and Oregon, 2015)

–Equipercentile» Randomly equivalent groups (e.g., comparing state standards to

SBAC and PARCC standards, 2016)

–Statistical Moderation»Randomly equivalent groups (e.g., comparing state

standards to TIMSS standards, 2010)

When is Contextual Information Important?• When you are trying to reach a policy goal• The earlier you introduce contextual information into the

standard setting workshop the more it is likely to influence the outcome

Literature• Ferrara (2005). Vertically Articulated Performance

Standards (special issue of Applied Measurement in Education)

• McClarty, Way, Porter, Beimer and Miles (2012). Evidence-Based Standard Setting (Educational Researcher)

• Phillips (2011). The Benchmark Method of Standard Setting. In Gregory J. Cizek (ed.), Setting performance standards (2nd edition). New York: Routledge

• Phillips, G. and Jiang, T. (2015) Using PISA as an International Benchmark in Standard Setting. Journal of Applied Measurement, 16(2):161-70

Florida

Florida Standards Assessment (FSA)

• Implemented in spring 2015 (baseline administration)• Assessments administered:

• Grades 3‐10 English Language Arts (includes text‐based writing)

• Grades 3‐8 Mathematics• Algebra 1 EOC• Geometry EOC• Algebra 2 EOC

Standard Setting: A Multi-Stage Process

Four Rounds of Educator Panel Judgment

• Initial judgment based on test content and Achievement Level Descriptors (round 1)

• Articulation – how cut scores appear across grades in Grades 3-10 ELA and Grades 3-8 mathematics (round 2)

• Impact data – how many students would be in each achievement level, and how subgroups would perform based on recommended cut scores (round 3)

• Benchmarking – how students would compare on FSA vs. international assessments (round 4)

Standard Setting ProcessAchievement Level Policy Definitions

• Achievement Level Policy Definitions – describe student achievement of Florida Standards at each achievement level

Level 1 Level 2 Level 3 Level 4 Level 5Students at this level demonstrate an inadequatelevel of success with the challenging content of the FloridaStandards.

Students at this level demonstrate abelowsatisfactory level of success with the challenging content of the Florida Standards.

Students at this level demonstrate a satisfactory level of success with the challenging content of the Florida Standards.

Students at this level demonstrate an above satisfactory level of success with the challenging content of the Florida Standards.

Students at this level demonstrate mastery of the most challenging content of the Florida Standards.

Individual and Group-Level Stakes• Individual

– To be promoted, Grade 3 students must score at or above Level 2 on ELA (“good cause” exemptions exist, however)

– Students must score at or above Level 3 on Grade 10 ELA and Algebra 1 EOC to graduate (retakes and alternative assessments provided for)

– EOCs count as 30% of course grade

• Group– School grades, which include learning gains, acceleration, and improvement of

performance of lowest 25% of students– School recognition dollars– District grades (based on school grades)– Teacher evaluation (Value-added Model, scores count at least 33% toward

evaluation)

ELA – Round 1 (Test Items & ALDs)

ELA – Round 2 (Articulation)

ELA – Round 3 (Impact Data - % in each Level; Subgroup Performance)

ELA – Round 4 (Benchmark to National/Int’l Tests)

ELA – Reactor Panel (Educator Panel + Policy Considerations)

Mathematics – Round 1 (Test Items & ALDs)

Mathematics – Round 2 (Articulation)

Mathematics – Round 3 (Impact Data - % in each Level; Subgroup Performance)

Mathematics – Round 4 (Benchmark to National/Int’l Tests)

Mathematics – Reactor Panel (Educator Panel + Policy Considerations)

Math EOCs – Round 1 (Test Items & ALDs)

Math EOCs – Round 2 (Pseudo-Articulation)

Math EOCs – Round 3 (Impact Data - % in each Level; Subgroup Performance)

Math EOCs – Round 4 (Benchmark to National/Int’l Tests)

Math EOCs – Reactor Panel (Educator Panel + Policy Considerations)

What Florida Educator Panelists Said

How important were the following factors in your placement of the bookmarks?

Very Important

Somewhat Important

Not Important

Achievement Level Descriptions (ALDs)

84% 14% 2%

External benchmark data 37% 53% 10%Feedback data 77% 23%Impact data 70% 29% 1%

What Florida Educator Panelists Said

Strongly Agree

Agree DisagreeStrongly Disagree

The feedback on cut scores was helpful in my decisions regarding placement of my bookmarks.

74% 24% 1%

I found the panelist feedback data and discussion helpful in my decisions about where to place my bookmarks.

83% 17% <1%

I found the impact data and discussion helpful in my decisions about where to place my bookmarks.

69% 28% 3%

I made my recommendations independently and did not feel pressured to set my bookmarks at a certain level.

82% 17% 1%

I believe that the recommended cut scores represent the expectations of performance for the students of Florida.

71% 29% <1%

What Florida Reactor Panelists Said

Usefulness of the following used during the Reactor Panel meeting.

Not at all

useful

Somewhat useful

Very useful

Reviewing external data 25% 75%Reviewing the standard-setting process used by the Educator Panel

6% 94%

Reactor Panel discussions 100%

What Florida Reactor Panelists Said

How important was each of the following factors in rendering your judgments?

Notimportant

Somewhatimportant

Very important

The description of Achievement Level Descriptions 25% 75%

Reactor Panel discussions 6% 94%

External data100%

Impact data 19% 81%

Alignment of cut points across grades/subjects25% 75%

Arizona

Prior AZ Performance Standards• AIMS Performance Standards were not aligned to College

and Career Ready Expectations and not comparable to other tests

• Achieve.org identified Arizona as one of the states with the largest gap (more than 40 percentage points) between state proficiency levels and the state’s NAEP proficiency levels

New AZ Performance Standards• Expected to measure College and Career Readiness, per

law– Arizona Revised Statutes 15-741.01 (D)

Any additional assessments for high school pupils that are adopted by the state board of education after November 24, 2009 shall be designed to measure college and career readiness of pupils

New AZ Performance Standards• Expected to measure College and Career Readiness and

provide comparability, per policy– State Board of Education’s Key Values for new Statewide Assessment

(2014)» Measure student mastery of the Arizona standards and progress toward college and career

readiness» Allow meaningful national or multistate comparisons of school and student achievement

Challenge• Cut scores recommended by Standard Setting panelists

go to State Board of Education for adoption• Previously, State Board of Education has been criticized

when it altered cut scores proposed by Standard Setting panelists

• Establish a standard setting process that would result in recommended cut scores aligned with the Board’s stated expectations and could be adopted without revision

Strategy• Include contextual information in the Standard Setting

process that provide panelists with benchmark information related to college and career readiness and comparability

• Primary consideration for panelists should still be the match between Performance Level Descriptors and the content of the test

Contextual Information• Approximate performance standards for the following were

included at appropriate grades– AIMS (AZ’s previous academic assessment)– Smarter Balanced– NAEP – PISA – ACT college ready

Standard Setting Model• Standard Setting panelists were instructed to place their

bookmarks based on test content and their “just barely” Performance Level Descriptors

• Contextual information provided in the Ordered Item Booklet indicating the general neighborhood where performance standards likely reside

• Contextual information was always available (from Round 1 onward)

Use of Contextual Information• Was the contextual information useful to the panelists? • Did they rely primarily on the contextual information to

make their bookmark decisions?• Did they feel coerced into placing their bookmarks based

on the contextual information?

What Panelists Said

What Observers Said• Arizona invited three independent observers to attend

Standard Setting who had full access to all panelists and ADE and vendor staff

• The following are some excerpts from their report to the State Board of Education

What Observers Said• Teachers were trained to make decisions based on the

Performance Level Descriptors and the content students are supposed to know. They were guided by the Board’s goals of having tests that can be compared to other assessments that reflect college and career readiness.

What Observers Said• (Teachers) were not told that the cut scores have to be at

cut points for other tests but they were there for context. In the training they were told “Your decision should be based on your professional opinion. The related tests are to give you a context for your choice.”

What Observers Said• The cut points were set based on teacher judgment, and

the final decision was theirs. The directions and training made that clear to the teachers.

Results• AzMERIT Performance Standards are quite consistent with

relevant ACT college ready, and the NAEP and Smarter Balanced proficient, benchmarks

Results• Arizona is a “Top Truth

Teller in 2015 for having a proficiency score within five percentage points of NAEP in eighth-grade math.” – HonestGap.org

Performance Level Setting• Purpose of Performance

• Level Workshop

Recommend four performance standards to differentiate the five performance levels for state board consideration

Performance Level Setting

• Ohio Revised Code requires State Board to set scores for 5 levels:

• An advanced level of skill;• An accelerated level of skill;• A proficient level of skill;• A basic level of skill; or• A limited level of skill

Performance Level Setting• Training for Panelist

Take the Assessment Review Student Population Review Performance Level Descriptors Discuss concepts of the book mark method Discuss concept of a student “just barely” in

each performance level

• Ordered Item Booklet

Collection of test items ordered from easiest to most difficult

Each page corresponds to a level of achievement

Panelist used to recommend minimum level of achievement for each level

Performance Level Setting• Benchmark Data Provided

Assessment Consortium performance levels (SBAC and PARCC)

NAEP performance standards for grades 4 and 8 and interpolated for grade 6

ACT for High School End of Course tests Ohio previous OAA and OGT assessments

Performance Level Setting• Workshop Panelists

Reviewed and took the test Received training in performance setting

process Two rounds of performance level setting Table leaders reviewed vertical articulation Completed workshop evaluation

• State Considerations

• Graduation Requirements• Third Grade Reading Guarantee• State Report Cards• Growth measures

Incorporating Contextual Information in Standard Setting...– Three year cycle (2012, 2015, 2018)...

Documents

Pisa. Timss

TIMSS 2007 - Education · TIMSS 2007 Final Report ... Major Accomplishment ... TIMSS 2007 International Mathematics Report • TIMSS 2007 International Science Report

Year 34 Cycle 1

Dubai TIMSS 2015 · TABLE OF CONTENTS 1 Dubai’s Participation in TIMSS 2015 1.1 Introduction 1.1 What is TIMSS? 1.2 Dubai Sample Participating in TIMSS 2015 2 TIMSS 2015 Results:

TIMSS 2015 Completed TIMSS 2019 Underway · TIMSS 2015 Completed TIMSS 2019 Underway Ina V.S. Mullis and Michael O. Martin Executive Directors TIMSS & PIRLS International Study Center

Other information collected by TIMSS · Science Study (TIMSS) eTIMSS 2017 Pilot. Other information collected by TIMSS. TIMSS is more than an assessment of student . knowledge in mathematics

TIMSS 2011 Science Framework - TIMSS and PIRLS …TIMSS 2011 SCIenCe FraMework | 55 Life Science: Life Cycles, Reproduction, and Heredity 1. Trace the general steps in the life cycle

Trends in International Mathematics and Science Study (TIMSS) · Trends in International Mathematics and Science Study (TIMSS) ... 5.2.1 TIMSS and TIMSS Advanced 2015 Benchmark Achievement

TIMSS 2015. Svenska grundskoleelevers kunskaper … › download › 18.6bfaca41169863e6a...6 TIMSS 2015 Sammanfattning TIMSS (Trends in International Mathematics and Science Study)

1 TIMSS Uluslararası Matematik ve Fen Eğilimleri Araştırması TIMSS

TIMSS TIMSS Numeracy 2015 NQCM Classroom Observation Record

TIMSS Physics

Junior Cycle Business Studies – Second Year...Junior Cycle Business Studies – Second Year. Junior Cycle Business Studies – Second Year. ENTERPRISE - ENTERPRISE IN ACTION. TASK

First Cycle Year Activities

Other information collected by TIMSS · 2015 Other information collected by TIMSS TIMSS is more than an assessment of student knowledge in mathematics and science. TIMSS . also considers

TIMSS Advanced

brosur timss

Year 56 Cycle 2

TIMSS Science Items - TIMSS and PIRLS Home

TIMSS 1999