Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden

Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional AccountabilitySession 353

Braden J. Hosch, Ph.D.Director of Institutional Research & AssessmentCentral Connecticut State University

AIR Annual Forum, Chicago, ILJune 1, 2010

OverviewVoluntary System of Accountability and

the Collegiate Learning Assessment

Institutional Profile

Methodology and Limitations

Findings

Implications for Accountability

Voluntary System of AccountabilityInitiative among public colleges

led by APLU and AASCU to provide public with comparable information in same format

Pre-emptive response to findings issued by Spellings Commission (2006)

Learning outcomes must be posted by Spring 2011

VSA Learning OutcomesResults from cross-sectional administration

to first-year students and seniors of one of three tests:◦Collegiate Learning Assessment (CLA)◦Collegiate Assessment of Academic Proficiency

(CAAP)◦Measure of Academic Proficiency and Progress

(MAPP)

Template reports scaled assessment scores, SAT/ACT scores of tested students, and an institutional relative-to-expected (RTE) score

Collegiate Learning AssessmentConstructed response test that measures:

◦Critical Thinking◦Problem Solving◦Analytical Reasoning◦Writing

Two tasks:◦Performance task (90 minute time limit)◦Analytic writing task (60 minute time limit)

Scored holistically and converted to scaled score,percentile score, and RTE score.

Central Connecticut State UniversityPublic – part of Connecticut State Univ. SystemCarnegie 2005 Master’s-Larger ProgramsNew Britain, CT (Hartford MSA: ~ 1.2 million pop.)Fall 2009 Enrollment:

◦ 12,461 headcount (9,989 undergraduate, 22% residential);

9,619 full-time equivalent enrollment◦ 52% female; 17% minority◦ Full-time, first-time students: 1,281 (56% residential)◦ Mean SAT score:

1025 (F 2009)

Six-year graduation rates of full-time, first-time students entering in Fall 2003: 49%

940960980

100010201040

974

1025

Me

an

SA

T

CCSU CLA Percentile Scores

FY 2

007

(SAT

= 1

019)

Seni

or 2

008

(SAT

=99

4)

FY 2

008

(SAT

=10

45)

Seni

or 2

009

(SAT

= 1

016)

FY 2

009

(SAT

= 1

019)

0

20

40

60

80

100

51 3767 70

5362 6384 97

Raw Percentile Adjusted Percentile

Perc

en

tile

Am

on

g A

ll C

LA

Takers

Detailed Results by SemesterParticipants

Fall 2007First-Year

Spring 2008

Senior

Fall 2008First-Year

Spring 2009

Senior

Fall 2009First-Year

Participants

105 99 110 134 130

HS Rank 57%ile 64%ile 61%ile 63%ile 62%ile

SAT M+CR

1019 994 1045 1016 1019

CLA Scaled

1057 1133 1127 1248 1098

Performance Task

1066 1113 1110 1214 1087

Analytic Writing Task

1048 1147 1144 1282 1109

CLA %ile 51 37 67 70 53

Cum GPA 2.73 3.13 2.87 3.24 2.88

Minutes -- 45 49 63 44

Test AdministrationTest administration procedures evolved

over time because of difficulty in recruitment.

First-year students recruited through FYE courses◦Incentives varied by instructor

Seniors recruited primarily through email◦Incentive = graduation regalia (~$25 - $40)

First-Year StudentsFall 2007 Fall 2008 Fall 2009

Proctor PT EmployeeOIRA Director

Grad Assistant Grad Assistant

Incentives [Various] [Various] [Various]

FYE Sections 7 7 9

Participants 105 110 130

Pre-Test Survey

N Y Y

SeniorsSpring 2008 Spring 2009 Spring 2010

Proctor PT Employee &

OIRA Director

Grad Assistant Grad Assistant

Incentives Initially none$25 Regalia

Discount

$35 Regalia Voucher

(Total Cost)

$40 Regalia Voucher

(Total Cost)

Senior Capstones

3 0 0

TotalParticipants

99 130 105

Pre-test Survey

N Y Y

Methodological Issues & LimitationsMethodological Issues

◦Cross-sectional design◦Different proctors in 2007-08 than 2008-

09 and 2009-10◦Different incentives for FY students and

seniors Within administrations Across administrations

◦Hand-timingTime on test as an imperfect proxy

for motivation

Minutes Spent on Test by Scaled CLA Score (All Available)

1200+ 1101-1200 1001-1100 Below 10000

10

20

30

40

50

60

70

55.4 50.543.5

33.7

63.456.5 52.1

43.8

First-Year Students Seniors

Scaled CLA Score

Mean

Min

ute

s S

pen

t on

Test

Minutes Spent on Test by Scaled CLA Score (2007-08)

1200+ 1101-1200 1001-1100 Below 10000

10

20

30

40

50

60

70

52.246.1 40.6 38.3


Scaled CLA Score

Mean

Min

ute

s S

pen

t on

Test


1200+ 1101-1200 1001-1100 Below 10000

10

20

30

40

50

60

70

54.4 51.1 47.336.0

65.1 59.6 60.553.5


Scaled CLA Score

Mean

Min

ute

s S

pen

t on

Test


1200+ 1101-1200 1001-1100 Below 10000

10

20

30

40

50

60

70

56.649.8

40.132.6


Scaled CLA Score

Mean

Min

ute

s S

pen

t on

Test

Minutes Spent on Test by Relative-to-Expected CLA Score (All Available)

Wel

l Abo

ve a

nd A

bove

Exp

ecte

d

At E

xpec

ted

Wel

l Bel

ow a

nd B

elow

Exp

ecte

d0

20

40

60

51.3 48.5 36.462.8 54.3 49.2


Relative-to-Expected CLA Score

Mean

Min

ute

s S

pen

t on

Test

Minutes Spent on Test by Relative-to-Expected CLA Score (2007-08)

Wel

l Abo

ve a

nd A

bove

Exp

ecte

d

At E

xpec

ted

Wel

l Bel

ow a

nd B

elow

Exp

ecte

d0

20

40

60

50.8 47.8 34.8



Mean

Min

ute

s S

pen

t on

Test


Wel

l Abo

ve a

nd A

bove

Exp

ecte

d

At E

xpec

ted

Wel

l Bel

ow a

nd B

elow

Exp

ecte

d0

20

40

60

51.8 49.2 41.965.0 58.3 57.5



Mean

Min

ute

s S

pen

t on

Test


Wel

l Abo

ve a

nd A

bove

Exp

ecte

d

At E

xpec

ted

Wel

l Bel

ow a

nd B

elow

Exp

ecte

d0

20

40

60

51.0 47.9 33.1



Mean

Min

ute

s S

pen

t on

Test

Correlations (First-Year Students)

CLA Scaled Score

Minutes Spent on CLA

SATCR

SAT Writing

SAT(Math + CR)

High School Rank

Minutes spent on CLA

.468 -- -- -- -- --

SAT Critical Reading

.333 .178 -- -- -- --

SAT Writing .311 .162 .632 -- -- --

SAT (Math + CR) .326 .186 .807 .610 -- --

High School Rank

.264 .227 .133 .161 .180 --

SAT Math .201 .127 .331 .370 .824 .145

Correlations (Seniors) CLA

Scaled Score

SAT(Math + CR)

SATMath SAT CR

End of term cum GPA

High School Rank

SAT (Math + CR) .505 -- -- -- -- --

SAT Math .479 .886 -- -- -- --

SAT Critical Reading

.409 .890 .576 -- -- --

End of term cum GPA

.400 .271 .285 .263 -- --

High School Rank

.338 .214 .178 .147 .336 --

Minutes spent on CLA

.331 .090 .210 .095 .177 .214

Regressions (First-Year Students)First-Year Student CLA Scaled Score (R2=0.261)

β Std. Err.

t Sig.

(Constant) 652 73.3 8.89 ***

Minutes spent on CLA 3.67 .490 7.49 ***

Combined SAT Score 0.281 .071 3.93 ***

First-Year Student CLA Percentile (R2=0.286)

β Std. Err.

t Sig.

(Constant) -28.9 12.6 -2.28 ***

Minutes spent on CLA .685 .084 8.18 ***

Combined SAT Score .049 .012 3.94 ***

Regressions (Seniors)Senior CLA Scaled Score (R2=0.261)

β Std. Err.

t Sig.

(Constant) 395 93.9 4.20 ***

Minutes spent on CLA 2.50 .535 4.67 ***

Combined SAT Score 0.665 .091 7.34 ***

Senior CLA Percentile (R2=0.286) β Std. Err.

t Sig.

(Constant) -69.0 14.2 -4.85 ***

Minutes spent on CLA .372 .081 4.59 ***

Combined SAT Score .099 .014 7.19 ***

Survey ResultsDifferences in test scores and time

spent on test by self-reported motivation were suggestive but not statistically significant.

Scores and time usage aside, the percentage of students who agreed or strongly agreed they were highly motivated to participate in CLA:◦34% First-Year Students◦70% Seniors

Self-Reported Motivation

First-Year

(Fall 2008 and Fall 2009)Seniors

(Spring 2009 only)

N PctAvg.

MinutesAvg. CLA Score N Pct

Avg. Minutes

Avg. CLA Score

Strongly disagree 10 4% 40 1071 1 1% ^ ^

Disagree 19 8% 45 1103 1 1% ^ ^

Neutral 123 53% 46 1109 37 28% 59 1240

Agree 64 28% 48 1125 70 53% 63 1250

Strongly agree 14 6% 51 1169 22 17% 67 1244

I feel highly motivated to participate in this activity today

Differences are suggestive but NOT statistically significant

Test Modality Preference

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score

Strongly disagree 35 15% 48 1089 9 7% 57 1220

Disagree 73 32% 45 1102 24 18% 61 1222

Neutral 74 32% 48 1127 51 39% 62 1292

Agree 34 15% 46 1129 36 27% 66 1207

Strongly agree 15 6% 45 1155 11 8% 65 1265

I perform better on essay tests than on multiple choice tests.


Assessment Modality Preference

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score


Disagree 30 13% 46 1143 16 12% 62 1248

Neutral 51 22% 45 1122 33 25% 62 1290

Agree 81 36% 48 1103 45 34% 67 1233

Strongly agree 11 5% 42 1083 9 7% 67 1205

I prefer to take a test rather than write a paper.

Test Anxiety

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score


Disagree 54 24% 50 1126 46 35% 67 1271

Neutral 76 33% 48 1114 39 30% 62 1209

Agree 53 23% 42 1092 31 24% 61 1261

Strongly agree 24 11% 45 1112 2 2% ^ ^

I get so nervous when I take tests that I don't usually perform my best work.

Student Responsibility for Learning

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score

Strongly disagree 0 0% ^ ^ 0 0% ^ ^

Disagree 1 0% ^ ^ 2 2% ^ ^

Neutral 19 8% 37* 1022* 11 8% 54 1187

Agree 106 47% 45* 1124* 61 47% 65 1271

Strongly agree 100 44% 51* 1121* 57 44% 63 1233

Students are responsible for learning material assigned by their professors

* Sig. at p<0.05 ANOVA test.

Institution Responsibility for Learning

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score


Disagree 66 29% 46 1097 36 27% 60 1233

Neutral 98 43% 48 1114 40 31% 63 1271

Agree 40 18% 47 1122 36 27% 64 1242

Strongly agree 8 4% 43 1118 14 11% 62 1245

Colleges and universities are responsible if students don't learn what they need to be successful after they graduate.

Mandatory College Exit Tests

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score


Disagree 86 38% 48 1137 63 48% 63 1280

Neutral 61 27% 47 1093 30 23% 61 1226

Agree 31 14% 46 1119 13 10% 69 1176

Strongly agree 5 2% 42 1009 8 6% 61 1197

All college students should be required to pass a standardized exit test in order to graduate


Publications of College Rankings

First-Year


(Spring 2009 only)

N PctAvg.


Avg. Minutes

Avg. CLA Score


Disagree 37 16% 47 1135 33 25% 61 1270

Neutral 118 52% 46 1114 47 36% 62 1231

Agree 62 27% 48 1096 42 32% 63 1260

Strongly agree 3 1% 45 965 5 4% 70 1181

Students should use published college rankings (like US News and World Report) when deciding which school to attend


Overall FindingsTime spent on test MATTERS

What students say about their motivation may not matter (much)

The relationship between time and test scores is generally missing from discussions about accountability

Implications (1)

1. Acknowledge that test scores may be influenced by motivation/time spent on test; support further research into these effects.

2. Longitudinal testing may help control for some of the effects of motivation / time spent on test.

Implications (2)

3. Multi-year moving averages might improve meaningfulness of test-score information.

4. Statistical adjustments based on time spent on test should be explored but may not be technically or politically feasible.

Implications (3)5. Explore portfolio or other contextual

assessment strategies for accountability, esp. among consortia of institutions (not unlike athletics conferences)

6. Recognize that motivation and time effects are also likely present in elementary and secondary education; consider extent to which performance reflects cognitive vs. behavioral/motivational outcomes.

Questions

Session 353

Dr. Braden J. HoschDirector of Institutional Research & AssessmentCentral Connecticut State [email protected]

Paper, handout, and slides online at:http://www.ccsu.edu/page.cfm?p=1973(see “Research and Presentations”)

Contact Information

mailto:[email protected]

http://www.ccsu.edu/page.cfm?p=1973

Documents

Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden