USING THE CLA TO INFORM CAMPUS PLANNING

Anne L. Hafner

Campus WASC Faculty Coordinator

Winter 2008

BACKGROUND

Collegiate Learning Assessment ( CLA) created by Council on Aid to Education, non-profit in NY

Performance assessment of critical thinking, analytic reasoning and written communication (see example)

Assesses “value added” by a collegeValue added: standardized gain score that

controls for students’ academic abilities(and for selectivity and SES)

CLA MEASURES I

Analytic writing task: Make an argument: “in our time, specialists of all kinds are highly overrated. We need more generalists - people who ca provide broad perspectives.”

Critique an argument

CLA Measures - Performance Task

Provide students with a real world scenario. Students have 90 minutes to advise the mayor on crime reduction strategies & evaluate potential policies: a.Invest in a drug treatment program b. Put more police on the street

Students provided w document library including memos, statistics, data tables, news articles, etc.

Performance Task

Students are required to use an integrated set of critical thinking, analytical reasoning, problem solving, and written communication skills

DETAILS I

Common models: cross sectional (freshman and senior cohorts) or longitudinal (follow same kids)

Students’ SAT scores are collected to enable comparison of student’s “expected” CLA score vs. student’s actual CLA score

DETAILS II

Both freshmen & seniors are tested. After seniors were tested, a gain score from freshman to senior class can be estimated. This informs the campus how much “value” was added by attending CSULA

Because sample of 100 drawn may not be representative of the class, CAE makes an adjustment so that a school’s “actual” CLA score is compared to its “expected” CLA score (based on students’ SATs)

INTERPRETING DIFFERENCES

Differences between actual and expected scores are reported in 2 ways: by points on CLA scale and by standard errors in terms of 5 performance levels (well below expected, below expected, at expected, above expected, and well above expected) At expected is between -1 and +1 standard errors from expected.

CSULA Fall 06 Administration

Fall 2006 - 110 students testedProblems with getting students to

participate, paid incentives One half of students took performance

task, one half took writing task CAE sent report back with some

interpretation, also sent data set

ANALYSIS: HOW DID OUR

FRESHMEN SCORE? After taking into account their abilities (SAT),

our freshmen scored 1.4 s.e. higher than expected overall or above expected.

Students performed at expected on performance task and above expected on writing tasks.

CLA sample scored higher than freshman population on SAT and ELM, same on EPT

How CSULA Freshmen Did

Our college performed in the middle of the group of all schools nationwide that participated in CLA in 2006 (5th -6th decile)

Correlations

With dataset, correlations were run CLA scale scores moderately to highly

intercorrelatedScale scores correlated moderately with

EPT, SAT, HSGPAOverall CLA performance level did not

have any positive correlations other than with CLA scale scores

CONCLUSIONS I

CLA sample not equivalent to 2006 freshman class (CAE: not a problem)

CLA sample’s demographics similar to campus demographics

CSULA sample scored higher than expected on tasks

Overall, CLA performance level was not highly correlated with other tests

CONCLUSIONS II

Students with English as primary language do not score significantly higher than non-English speakers on CLA

CLA appears to be measuring something different from knowledge & skills measured by the SAT, EPT, ELM and grades

FINDINGS Summer 2007

In summer 07, we received our senior findings and “gain” score

Seniors scored “at expected” Gain was 55 pts (less than expected) Campus is at

4th decile (between 30th-40th percentile nationwide) CAE can examine if improvement in average student

performance between entry and graduation is in line with similar gains of comparable students at other colleges

FINDINGS II

Seniors did less well than freshmen on performance tasks, and on critique an argument tasks.

CSULA’s 6 year graduation rate was “at expected’ 32%

MAJOR PROBLEMS

Staff time to recruit students and administer the test

High cost of enrollment incentivesTime required to take test (90 minutes)Cost to take the test is inexpensive

($20)

STRENGTHS

Value added is major focus, rather than absolute performance

CLA is well developed and assesses two key skills: critical thinking/reasoning, and writing skills

Strong psychometric properties, valid and reliable, standardized gain scores enable comparisons

Can work with small sample

CRITICISMS

Issue of junior transfers not dealt with in models; different population may need to be also assessed

Uncertainty about validity (consequential) and reliability

Heavy reliance on SAT score as measure of academic ability

THE FUTURE

It may be preferable to embed CLA tasks into a freshman required course such as Intro to College or orientation

Senior level: could be embedded in capstone courses or made graduation requirement

CLA could be a useful measure of some GE outcomes

THE FUTURE II

Pressure on universities from US DOE for accountability led to Voluntary System of Accountability (VSA)

2007: CSU Chancellor mandated that all campuses use CLA and report annually for the VSA

CAE is working to create performance standards for CLA (e.g “Proficient” level); could be useful to colleges

CAE is working with CSU to deal with issue of junior transfers