1

DevelopingDevelopingMeasuresMeasuresin Algebrain Algebra

Anne FoegenAnne FoegenIowa State UniversityIowa State University

Pacific Coast Research ConferencePacific Coast Research ConferenceFeb. 3 - 5, 2005Feb. 3 - 5, 2005

Funded by OSEP Field Initiated award #H324C030060Funded by OSEP Field Initiated award #H324C030060

Session OverviewSession Overview

Project backgroundProject background 5 algebra measures5 algebra measures Initial findingsInitial findings Questions and issuesQuestions and issues

Guidance and suggestionsGuidance and suggestions

2

Why Algebra?Why Algebra?

Challenging academic standardsChallenging academic standards Increased graduation requirementsIncreased graduation requirements Gatekeeper functionGatekeeper function Linked to success in technology fieldsLinked to success in technology fields ““21st century civil right21st century civil right”” (Moses & Cobb, (Moses & Cobb,

2003)2003)

Project AAIMSProject AAIMS

Algebra Assessment and Instruction--Algebra Assessment and Instruction--Meeting StandardsMeeting Standards

Descriptions of current algebraDescriptions of current algebrainstruction for students with and withoutinstruction for students with and withoutdisabilitiesdisabilities

Development and validation of progressDevelopment and validation of progressmonitoring measures in algebramonitoring measures in algebra

3

Identifying MeasuresIdentifying Measures

Compare 5 alternative measures toCompare 5 alternative measures toidentify those that are most promisingidentify those that are most promising

Initial work: examine distributions,Initial work: examine distributions,reliability, criterion validityreliability, criterion validity

Subsequent research activities willSubsequent research activities willexamine the viability of using the mostexamine the viability of using the mostpromising measures for progresspromising measures for progressmonitoringmonitoring

A - Basic/Tool Skills

9 + a = 15

a =

Simplify:

y2 + y – 4y + 3y2

Simplify:

2x + 4 + 3x + 5

12 – 6 = e

e =

12 – e = 4

e =

Simplify:

12 + ( - 8) + 3

Simplify:

(- 5) + ( - 4) – 1

!

z

5= 5

z =

6 • 9 = d

d =

Simplify:

4d + 7 d – 9 + 2

Simplify:

4(3 + s) - 7

!

54

w= 9

w =

2s = 16

s =

Simplify:

8 – (- 6) – 4

Simplify:

4 – (- 2) + 8

6 + 7 = v

v =

8m = 72

m =

Simplify:

6r – 5 – 2r + 6

Simplify:

3 (c + 2) – 2c

f – 7 = 3

f =

!

63

c

= 9

c =

Simplify:

-5 + 6 - 6

Simplify:

17 + 6d + 2d – 9

r – 4 = 4

r =

4

B - Basic ConceptsAlgebra Probe B -1 Page 1

What is the slope?

What is the y

intercept?

Fill in the

empty box:

s 3s

6 18

7 21

8

9 27

Fill in the

empty box:

n 4n+7

1 11

2 15

3

4 23

Fill in the

empty box:

b

5 2

3 0

0 -3

-2 -5

What is the slope?

What is the y intercept?

If y > 9, two

possible values for

y are _____ and

_____

6 • 4 + 1 =

________

Simplify:

7f + (2f + f)

Solve:

n + 3 = 8

n = ________

Evaluate 4 b + 2

when

b = 1 ________

and when

b = 3 ________

Write the

expression for this

phrase:

6 less th an a

number

(-2) • ( -4) =

________

Graph the expression

m > 6

Write a word

phrase for this

expression:

n + 9

8 ÷ 2 + 4 • 3 =

________

23 = ________

Write the expression for

this phrase:

9 times a number

-8 -6 -4 -2 0 2 4 6 8

C - Curriculum Sampling (open)Evaluate 5 y + x2

when y = 3 and x = 2

Simplify:

8(-4)(b)(b)(b)

Solve:

!

x

2

= ( - 6 )

Solve:

2t – 4 = 6

Graph x = 3

Graph the equation:

y + 2x = 4

Write the equation in

slope -

intercept form:

m =( -8) b = 4

Write the equation in

standard form:

(-2y) +

!

1

2

x = 4

5

D - Reform-Oriented

E- Curriculum Sampling (multiple choice)

6

A Tale of TwoA Tale of TwoRounds of DataRounds of Data

CollectionCollection……

Round 1: Spring 2004Round 1: Spring 2004

District 1: small District 1: small ““bedroom community;bedroom community;””predominately white; 16% free/reduced lunchpredominately white; 16% free/reduced lunch

High rate of student participation (82%)High rate of student participation (82%) Included 113 students in 4 types of classesIncluded 113 students in 4 types of classes

8th grade algebra8th grade algebra Algebra 1Algebra 1 Algebra 1A/1BAlgebra 1A/1B Algebra taught by special education teachers for IEPAlgebra taught by special education teachers for IEP

studentsstudents Administered probes A, B, and C and gatheredAdministered probes A, B, and C and gathered

data on criterion measuresdata on criterion measures

7

Data Collection ProceduresData Collection Procedures

Students completed 2 forms of probes AStudents completed 2 forms of probes Aand B, 1 form of C in Mayand B, 1 form of C in May

Probe administration process repeated 1Probe administration process repeated 1week laterweek later

Criterion measures included algebraCriterion measures included algebragrade, GPA, standardized test scores,grade, GPA, standardized test scores,teacher ratings, Iowa Algebra Aptitudeteacher ratings, Iowa Algebra AptitudeTest scoresTest scores

Means and Std. DeviationsMeans and Std. Deviations

N=6N=43N=49N=15N=113

3.0(3.2)

10.8(5.1)

14.7(6.7)

34.5(2.6)

15.3(9.8)

CCurric-open

9.8(4.7)

22.1(6.7)

30.8(6.7)

41.4(5.1)

27.7(10.2)

BBasic

Concepts

12.7(7.7)

14.2(5.1)

19.7(7.2)

26.5(7.4)

18.1(7.7)

ATool Skills

SpEd1A/1BAlg 18th gr.FullProbe

8

Probe DistributionsProbe Distributions

A3MNMEAN

38.0

36.0

34.0

32.0

30.0

28.0

26.0

24.0

22.0

20.0

18.0

16.0

14.0

12.0

10.0

8.0

6.0

4.0

16

14

12

10

8

6

4

2

0

Std. Dev = 7.70

Mean = 18.1

N = 113.00

Mean of all 4 B 6 min probes

50.0

47.5

45.0

42.5

40.0

37.5

35.0

32.5

30.0

27.5

25.0

22.5

20.0

17.5

15.0

12. 5

10.0

7.5

5.0

2.5

20

10

0

Std. Dev = 10.24

Mean = 27.8

N = 113.00

Avg of C across sessions

40.037.5

35.032.5

30.027.5

25.022.5

20.017.5

15.012.5

10.07.5

5.02.5

0.0

30

20

10

0

Std. Dev = 9.82

Mean = 15.3

N = 112.00

Differentiating BetweenDifferentiating BetweenGroups: Probes A, B, CGroups: Probes A, B, C

0

5

10

15

20

25

30

35

40

45

50

Form

A

Form

B

Form

C

Pro

be S

co

re

SpEd

Slower Pace

Typical

Advanced

9

Reliability of Probes A, B, CReliability of Probes A, B, C

.86.85.90†.71†C

Curric-open

BBasic

Concepts

ATool Skills

Probe

.73

.71

Week 1

.92

.86

Week 2

.86

.84

Form 1

.83

.81

Form 2

Criterion Validity for MeansCriterion Validity for Meansfor Teacher 1for Teacher 1

.62.59CCurric-open

BBasic Concepts

ATool Skills

Probe

.54

.32

Teacher Rating

.72

.52

IAAT

10

Conclusions from Round 1Conclusions from Round 1

Reliability is acceptable for probes afterReliability is acceptable for probes afterthe first task administeredthe first task administered

Technical adequacy of forms B and CTechnical adequacy of forms B and Cmore promising than form Amore promising than form A

Concerns about form CConcerns about form C Restricted range of scores for May dataRestricted range of scores for May data Lower Lower interscorerinterscorer reliability rates reliability rates Extremely time consuming to scoreExtremely time consuming to score

Round 2: Fall 2004Round 2: Fall 2004

11

District 1District 1

Lower rate of student participation (38%)Lower rate of student participation (38%)and smaller sample size (N=48)and smaller sample size (N=48)

Students completed 2 forms each ofStudents completed 2 forms each ofprobes D and E on two consecutive weeksprobes D and E on two consecutive weeks

Criterion validity data available forCriterion validity data available forteacher rating, IAAT (other measuresteacher rating, IAAT (other measurescurrently being gathered)currently being gathered)

Means and Std. DeviationsMeans and Std. Deviations

N=4N=8N=30N=5N=47

8.0(3.6)

6.4(3.8)

9.9(3.9)

11.9(3.2)

9.3(4.0)

ECurric-

MC

2.0(1.9)

4.2(2.1)

6.9(4.6)

9.3(3.7)

6.3(4.3)

DReform-oriented

SpEd1A/1BAlg 18th gr.FullProbe

12

Distributions for Probes D and EDistributions for Probes D and E

Mean of all 4 D probes

16.0

15.0

14.0

13.0

12.0

11.0

10.0

9.0

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.0

0.0

7

6

5

4

3

2

1

0

Std. Dev = 4.34

Mean = 6.3

N = 47.00

Mean of all 4 E probes

18.0

16.0

14.0

12.0

10.0

8.0

6.0

4.0

2.0

0.0

12

10

8

6

4

2

0

Std. Dev = 3.98

Mean = 9.3

N = 47.00

Reliability of Single ProbesReliability of Single Probes

ECurric- MC

DReform-oriented

Probe

.41

.40

Week 1

.43

.48

Week 2

.49

.61

Form 1

NS

NS

Form 2

13

Reliability of Aggregated ProbesReliability of Aggregated Probes

ECurric- MC

DReform-oriented

Probe

.45

.62

By Week

.53

.61

By Form

Criterion Validity for Teacher 1Criterion Validity for Teacher 1

ECurric- MC

DReform-oriented

Probe

.54

.46

Teacher Rating

.59

.61

IAAT

14

District 2District 2

Blue collar town; 85% white; 23%Blue collar town; 85% white; 23%free/reduced lunchfree/reduced lunch

64% student participation rate; 81 students64% student participation rate; 81 students 4 sections of Algebra 1A4 sections of Algebra 1A 2 sections of Algebra 1B2 sections of Algebra 1B

Means and Std. DeviationsMeans and Std. Deviations

18.0(6.8)

15.2(5.1)

16.0(5.8)

ECurric- MC

8.6(5.3)

8.9(5.4)

8.1(5.3)

DReform-oriented

N=24N=55N=79

16.1(6.5)

15.3(3.8)

15.5(4.7)

CCurric-open

18.0(6.2)

22.3(5.2)

21.0(5.8)

BBasic Concepts

14.5(4.6)

14.2(5.0)

14.3(4.8)

ATool Skills

Alg 1BAlg 1AFullProbe

15

Reliability of Single ProbesReliability of Single Probes

.56

.63

.92

.68

.70

Form 2

.57.56.46ECurric- MC

.49.72.53DReform-oriented

.44.80†NS†CCurric-open

.68.66.62BBasic Concepts

.73.78.64ATool Skills

Form 1Week 2Week 1Probe

Reliability of Aggregated ProbesReliability of Aggregated Probes

.58

.75

--

.69

.76

By Form

.63ECurric- MC

.70DReform-oriented

--CCurric-open

.72BBasic Concepts

.78ATool Skills

By WeekProbe

16

Criterion Validity for District 2Criterion Validity for District 2

.43

NS

.45

.33

.34Rating

Teacher 2IAATIAATRating

.40.61.41ECurric- MC

NSNSNSDReform-oriented

.38.61.67CCurric-open

.44.68NSBBasic Concepts

NS.46.40ATool Skills

Teacher 1Probe

Summary by Probe TypeSummary by Probe Type

.46-.57

.49-.72

.44-.92

.62-.68

.64-.78Reliability

District 2ValidityValidityReliability

.40-.61.54-.59.41-.49ECurric- MC

NS.46-.61.40-.61DReform-oriented

.38-.67.59-.62.71-.90CCurric-open

.33-.68.54-.72.73-.92BBasic Concepts

.34-.46.32-.52.71-.86ATool Skills

District 1Probe

Data in yellow collected in 2003-4; data in white collected in 2004-05.

17

Conclusions from Round 2Conclusions from Round 2

Serious concerns about probe D andSerious concerns about probe D andmismatch with instruction at these sitesmismatch with instruction at these sites(guessing)(guessing)

Reliability issuesReliability issues Round 2 data lower than Round 1Round 2 data lower than Round 1 D and E probes especially lowD and E probes especially low

Validity estimates differed by teacher inValidity estimates differed by teacher inDistrict 2District 2

Questions and IssuesQuestions and Issues

ReliabilityReliability Probe construction process (problem types,Probe construction process (problem types,

practice activities)practice activities) Student factors (motivation, fatigue, showingStudent factors (motivation, fatigue, showing

work)work) Teacher factors (buy-in)Teacher factors (buy-in)

ValidityValidity Complete analyses for Round 2 withComplete analyses for Round 2 with

additional variables (ITED, grades, IAATadditional variables (ITED, grades, IAATsubtests)subtests)

18

Common QuestionsCommon Questions

Global indicators or curriculum-sampling?Global indicators or curriculum-sampling? No clear pattern favoring one over the otherNo clear pattern favoring one over the other Most promising measures are B (global) andMost promising measures are B (global) and

E (curriculum sampling)E (curriculum sampling) Difficulty levelDifficulty level

No clear patternNo clear pattern B is an easier task; E represents currentB is an easier task; E represents current

instructional levelinstructional level

Next Steps/Future ResearchNext Steps/Future Research Explore strategies to increase reliabilityExplore strategies to increase reliability

(focuses samples of problem types per(focuses samples of problem types perquestion; student motivation; teacherquestion; student motivation; teacherencouragement)encouragement)

Examine additional criterion measures forExamine additional criterion measures forYear 2 to see if diminished magnitude ofYear 2 to see if diminished magnitude ofcorrelations holdscorrelations holds

Replicate static measures study in DistrictReplicate static measures study in District22

Conduct 10 weeks of progress monitoringConduct 10 weeks of progress monitoringin District 1in District 1