A Model Description By Benjamin Ditkowsky, Ph.D. Student Growth Models for Principal and Student Evaluation

A Model Description ByBenjamin Ditkowsky, Ph.D.

Student Growth Models for

Principal and Student Evaluation

It’s Not Always Easy To See What Our Students Can And Can Not Do

In Addition, Sometimes Some Things Just In Addition, Sometimes Some Things Just Don’t MatchDon’t Match

Can We Just Use The Same Test At Two Points In Time (Use Gain Scores)?

There are some problems with gain scores All tests are made up of a true score and

measurement error xi = xiTRUE +

Error is always present, but we don’t know how much error is present

When we subtract one score from another (i.e., a gain score) error in the final score is greater than in either of the tests from which the error originates

xgain = x2TRUE– x1TRUE + + (true differences are subtracted but error propagates)

Error! There Are Many Reasons Other Than The Test That Student Scores Vary From One Test To Another

Complex Solutions Exist, But

Do Complex Solutions improve teaching and learning?

Do Complex Solutions maintain a focus on the connection on standards?

Do Complex Solutions applied to our data violate basic statistical assumptions?

Do Complex Solutions focus on details or big ideas?

The overly complex statistical models have not been shown to be substantially more effective than less complex models

with complexity comes other concerns

Accountability Should Be

Trustworthy For a solution to be trustworthy it should meet all of the

assumptions upon which it is based

Useable For a solution to be useable, it should help teachers and

principals adjust instruction and intervention in real time

Accessible For a solution to be accessible it should be transparent

We Will Use Student Data For Evaluation

We Have Lots Of Data

Complex solutions have complex algorithms to estimate missing information.

For most students… we have enough information we can infer the rest.

Although, we don’t necessarily have the same data for all students…

Complex Statistical Analysis Is Unnecessary

Given sufficient evidence, complex analysis is unnecessary

looks like a duck

walks like a duck

swims like a duck

quacks like a duck

it probably is a duck

If it …

We Know What We Expect

Cut Scores are Based on Expected Performance

Test performance is categorized as Exceeding Standards, Meeting Standards, Below Standards, Academic Warning

Progress is defined by our assignment of the value or worth of improvement in the categorical performance of test scores from one time to the next

We Use Cut Scores To Categorize Level of Achievement, and Progress

Cut Scores Separate Performance Levels

ISAT Scores are broken down into performance designations (Academic Warning, Below Standards, Meets Standards, Exceeds Standards)

Typical Growth in Language Development on ACCESS has been established (W.I.D.A. Consortium, 2009)is calculated based on entry performance level and change in composite score (typical low, and high range growth)

R-CBM Scores reliably predict state test scores (Silberglitt & Hintze, 2005), cut scores predicting outcomes have been established by aimsweb and within Illinois (i.e., Below Basic, Questionable, Proficient, confidently proficient)

M-CAP scores predict State Test scores, and typical performance is established across the township

Benchmark assessments can divide score into performance categories

We Are Explicit About Our Values

Exceptional Growth (2) as well as maintenance of exceptional performance is highly valued, thus any score-pair that moves up two or more categories or ending in the highest performance category is worth two points.

Proficient Growth (1) is defined as any score-pair that moves up one category or is maintained i.e., scores growth at a rate commensurate with the increasing expectations of the Meeting Expectations category.

Inadequate Growth (-1) is defined as a score-pair in which the performance category at time 1 is higher than the performance category at time 2, or growth from time 1 to time 2 is not sufficient to move a student from the below expectations category into the meeting expectations category.

Unsatisfactory Growth (-2) is defined as a score-pair in which the performance category drops by two categories from time 1 to time 2 or ends in the Academic Warning Category.

Growth is defined as a change in performance category from Time 1 to Time 2

A Value Table Weights Growth

A. Warning Below Meets Exceeds

Academic Warning -2 1 2 2

Below Expectations -2 0 1 2

Meets Expectations -2 -1 1 2

Exceeds Expectations -2 -2 -1 2

Time 1 PerformanceTime 2 Performance Level

Miguel Is A 2nd Grade Student

The Data we have for him include:

Type I ACCESS from grade K and 1 Aimsweb - Fall and Winter

RCBM MCAP (not in example)

Type II Vocabulary Matching Fall and

Winter Avenues Pre-Post Assessments for

ELLs

Type III Pre-Post Classroom Assessments

Name Grade K Grade Level K 1 Fall Winter Fall Winter Pre Post

Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3

R CBM Vocabulary Matching AvenuesACCESS Composite

Name Grade K Growth ValueGrade Level K 1 Pattern Fall Winter Fall Winter Pre Post

Miguel 2 1.8 209 297 Proficient 1 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 Excellent 2 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 Needs to Improve 0 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 Proficient 1 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 Unsatisfactory -1 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3

R CBM Vocabulary Matching AvenuesACCESS Composite Each ACCESS pair is assigned a growth value based on the

comparison of scores and the expectations for growth

(WIDA, March 2009)

Unsatisfactory

Needs Improvement

Proficient growth

Excellent growth

Miguel is in second grade. • Baseline: In K his language proficiency level

was 1.8• Expectations: The low to high range for growth

was 44 to 90• Scores: 297 – 209 = 88

His gain was 88, less than 90 • Category designation: His growth level is

considered in the proficient range





Name Grade K Growth ValueGrade Level K 1 Fall Winter Pattern Fall Winter Pre Post

Miguel 2 1.8 209 297 15 (BB) 66 (PR) Excellent 2 2 (M) 7 (M) 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) Needs to Improve 0 3 (M) 3 (B) 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) Unsatisfactory -1 0 (U) 8 (M) 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) Excellent 2 2 (M) 12 (E) 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) Proficient 1 1 (B) 8 (M) 1 3


In Winter, Miguel’s performance was in the

proficient range

Fall performance

was Below Basic Expectations

Grade Measure Below Basic Proficient Below Basic Proficient Below Basic Proficient

2 R-CBM 30 45 55 65 70 90

Fall Winter Spring

Miguel is in second grade. • Baseline: In Fall his score 15 WRC indicated Below Basic Performance• Expectations: Winter Proficient Score is 65 WRC• Growth: In Winter, Miguel scored in the proficient range, his movement up two

categories (Below to Proficient) is considered excellent growth• Category designation: His growth level is considered in the excellent range

MeasuredEffects.com, 2010 ISAT Cut Scores





Name Grade K Growth ValueGrade Level K 1 Fall Winter Fall Winter Pattern Pre Post

Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) Proficient 1 1 2Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) Excellent -1 4 5Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) Proficient 2 3 5Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) Excellent 2 2 4Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) Proficient 1 1 3

R CBM Vocabulary Matching AvenuesACCESS CompositeRelatively low scores are in the proficient range in the fall, Miguel’s score of 2 is

considered proficient

In Winter, expectations for VM are higher, and

Miguel’s score increased sufficiently to remain on

target.

VMGrade 5 10 25 50 75 90 5 10 25 50 75 90

2 0 0 0 2 5 8 1 2 4 7 10 13Warning Exceeds Warning Exceeds

Winter

Robust Percentile Rank Robust Percentile Rank

Below Meets Below Meets

FALL

Local Normative Values

Miguel is in second grade. • Baseline: In Fall his score 2 WRC indicated his performance was typical in the Meets

category• Expectations: Winter Proficient range is from 7 to 10• Growth: In Winter, Miguel scored in the proficient range, his performance indicated

that his growth was consistent with expectations for his grade level (Meets to Meets)• Category designation: His growth level is considered in the proficient range





Name Grade K Growth ValueGrade Level K 1 Fall Winter Fall Winter Pre Post Pattern

Miguel 2 1.8 209 297 15 (BB) 66 (PR) 2 (M) 7 (M) 1 2 Proficient 1Augustine 2 3.9 287 343 32 (QS) 50 (QS) 3 (M) 3 (B) 4 5 Excellent -1Isabelle 2 2.9 262 307 40 (QS) 40 (BB) 0 (U) 8 (M) 3 5 Excellent 2Fariha 2 3.5 294 345 45 (PR) 72 (PR) 2 (M) 12 (E) 2 4 Excellent 2Nabiha 2 1.8 204 248 12 (BB) 58 (QS) 1 (B) 8 (M) 1 3 Excellent 1


The Avenue’s pre test is broken into 6 levels designated in 3 levels (Beginning,

Intermediate and Advanced)

Miguel is in second grade. • Baseline: In Fall his proficiency level was 1 indicating early beginning language

proficiency• Expectations: Winter Intermediate language range is from 3 to 4• Growth: In Winter, Miguel’s language level increased from level 1 to 2, though not up

to the Intermediate range, his scores demonstrated growth• Category designation: His growth level is considered in the proficient range


A Demonstration Of Proficient Growth

• Each score, for each second grade student in the targeted ELL subgroup has been examined and categorized based on Cut Scores.

• Each available score-pair has been reviewed and growth has been categorized and weighted.

• Individual ratings were calculated

• The overall group rating was calculated

Pattern Value Pattern Value Pattern Value PatternMiguel 2 Proficient 1 Excellent 2 Proficient 1 Proficient 1 1 ProficientAugustine 2 Excellent 2 Needs to Improve 0 Excellent -1 Excellent -1 -1 UnsatisfactoryIsabelle 2 Needs to Improve 0 Unsatisfactory -1 Proficient 2 Excellent 2 1 ProficientFariha 2 Proficient 1 Excellent 2 Excellent 2 Excellent 2 2 ExcellentNabiha 2 Unsatisfactory -1 Proficient 1 Proficient 1 Excellent 1 1 Proficient

Group Rating 1 Proficient

Name Grade Individual Growth

ACCESS Composite R CBM Vocabulary Matching AvenuesGrowth

j

mMiguel = m[1,2,1,1] = 1

Rating = m[1, -1, 1, 2, 1] = 1 1 Proficient Growth1 Proficient Growth

Alternate Illustration: Why Use More Than Two Administrations Of The Same Test From One Source

j

It is possible that different tests indicate different patterns, without changing the overall rating

The differences in patterns by test may be diagnostically important, but insufficient for high stakes evaluative purposes

English Language proficiency is growing at a rate above what

is expected

Automaticity with basic skills in

Reading may need to improve

Automaticity with basic skills

in Math may be on track

Instructional Vocabulary may be insufficient for continued

growth in grade level material

MCAP

Pattern Value Pattern Value Pattern Value Pattern Value PatternMiguel 2 Proficient 1 Excellent 2 Needs to Improve 0 Proficient 1 1 ProficientAugustine 2 Excellent 2 Needs to Improve 0 Needs to Improve 0 Excellent -1 Unsatisfactory -1 0 UnsatisfactoryIsabelle 2 Needs to Improve 0 Unsatisfactory -1 Excellent 2 Excellent 2 Needs to Improve 0 0 ProficientFariha 2 Excellent 2 Needs to Improve 0 Proficient 1 Needs to Improve 0 Proficient 1 1 ExcellentNabiha 2 Excellent 2 Unsatisfactory -1 Needs to Improve 0 Proficient 1 Proficient 1 1 Proficient

Excellent 2 Needs to Improve 0 Proficient 1 Needs to Improve 0 Proficient 1 1 Proficient

Name Grade Individual Growth

ACCESS Composite R CBM Vocabulary Matching AvenuesGrowth

Growth English Language Proficiency is increasing at an adequate rate

While there may be some areas where progress is worth further

investigation, for the purposes of evaluating and categorizing; overall, across measures,

academic growth is occurring at an acceptable rate.

Timmy Is A 4th Grade Student

The data we might have for him include:

Type I ISAT from grade 3 and 4 Aimsweb - Fall and Winter

RCBM MCAP

Type II Vocabulary Matching


What Data Might We Use For Him?

Gertrude Is A 7th Grade Student

The data we might have for her include:

Type I ISAT from grade 3 and 4 Aimsweb - Fall and Winter

RCBM MCAP

Type II Vocabulary Matching Prentice Hall Benchmark Assessments


What Data Might We Use For Her?

Setting Student Data Goals For Principal Evaluation

By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students at administered at two points in time school wide, students at school school namename will demonstrate an increase in the proportion making will demonstrate an increase in the proportion making adequate progress from 35% to 40%. adequate progress from 35% to 40%.

Goals can be set to increase the amount of progress made


By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students identified administered at two points in time school wide, students identified as as define cohortdefine cohort at at school nameschool name will demonstrate an increase in the will demonstrate an increase in the proportion making adequate progress from 68% to 80%.proportion making adequate progress from 68% to 80%.

Goals can be set to increase the proportion of students making adequate progress made for a particular subgroup


By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students identified administered at two points in time school wide, students identified as as define cohortdefine cohort at at school nameschool name will demonstrate a decrease in the will demonstrate a decrease in the proportion making unsatisfactory progress from 8% to 4%.proportion making unsatisfactory progress from 8% to 4%.

Goals can be set to decrease the proportion of students not making adequate progress for a particular subgroup


By February 201By February 201xx, given available Type I and Type II assessments , given available Type I and Type II assessments administered at two points in time school wide, students at administered at two points in time school wide, students at school school namename will demonstrate will demonstrate proficient or excellentproficient or excellent growth as defined by growth as defined by the convergence and magnitude of data classified with district the convergence and magnitude of data classified with district defined value tables.defined value tables.

Goals can be set to achieve an overall rating for the demonstration of student growth

Documents

A Model Description By Benjamin Ditkowsky, Ph.D. Student Growth Models for Principal and Student Evaluation