VOLUNTARY SYSTEM OF ACCOUNTABILITY AND LEARNING

OUTCOMES: AN UPDATE

Teri Hinds Voluntary System of AccountabilityNatasha JankowskiNational Institute for Learning Outcomes Assessment

October 30, 2012 Assessment Institute

Overview NILOA’s work with Transparency Internal and External Communication Voluntary System of Accountability Evaluation of VSA College Portrait Pilot Future of Student Learning Outcomes on

College Portrait VSA as a communication tool

The Heat is On Unprecedented public challenges for higher

education to be more accountable for results. Accompanying external demands for

information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve.(Ewell, 2010)

Overview of NILOANILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts.SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN ● Degree Qualifications Profile

www.learningoutcomesassessment.org

www.learningoutcomeassessment.org

Reports on Transparency

Question What information does your institution

share publically on its website? How are results of student learning

shared internally?

Transparency Framework

http://www.learningoutcomesassessment.org/TransparencyFramework.htm

Transparent Communication

Internal communication External communication Telling an institutional story about

student learning

Questions What types of stories does your

institution tell about its students? What types of stories does your

institution tell about how it assures quality?

Background VSA launched in 2007 Partnership between APLU and AASCU 320 participating institutions Three original objectives

Demonstrate accountability and transparency

College search toolSupport innovation in the measurement of

student learning outcomes

Student Learning Outcomes Pilot

Student learning outcomes page includes link to institution specific learning outcomes data;

Selection of the CLA, CAAP or ETS Proficiency Profile to measure value-added learning gains

Report results no more than four years after joining

Link to information on test administration, sample and response rate

Pilot period ends 12/31/12 What was learned from the pilot?

Evaluate the Effectiveness of SLO Pilot

Focus groups Interviews Surveys Google Analytics College Portrait statistics

Findings related to Transparency Both VSA-participating and nonparticipating institutions

agree that VSA in 2007 was a wise, timely, useful, and necessary response to the accountability and transparency demands of the time

Yet, nearly half of the participants have not yet met expectations of College Portrait student learning outcomes pilot

45% of participating institutions have yet to post student learning outcomes information

Information that was provided on student learning outcomes lacked explanatory or contextual material

0 20 40 60

Student Experiences

Student Learning Outcomes

Majors, Graduation, & Next

Classes & Campus Life

Student Success and Progress

Admissions

Students

College Cost and Financial Aid

College Cost Estimator

Percentage of Total Views

82% of participating institutions had traffic

on their student learning pages but almost 1 in 5 did not have a single visitor to their student learning outcomes page since 2009.

Information posted on the College Portrait may not reflect the needs of prospective students and families or provide the information they actually seek to make decisions about where to attend college.

The authorized standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community – it is thought the tests do not accurately reflect student performance within the institution.

Concerns about displaying results include: Inability to understand what scores

represent Questions regarding what tests measure Issues of reliability and validity Difficulty in using scores to improve

student learning Motivation of students & difficulty to obtain

a sample Questions regarding quality of results

81% of survey respondents stated that the pilot requirements did not align with their assessment practices and that the results were not usable for campus improvement efforts

Participating institutions agree -

expanding the number & nature of student learning measures will:

produce more accurate portrayals of student attainment,

provide more useful information for campuses, make information more meaningful for all

audiences.

Recommendations Focus on specific audiences and

communicating meaningful information to them

Contextualize information Expand range of assessment tools and

approaches Increase traffic to site Include examples of use of evidence to

improve learning Consider creating template which could be

used by all postsecondary institutions

Working Group Deliberations Technical work group considered

alternative measures of learning outcomes to recommend to VSA Board

Confirmed importance of student learning outcomes reporting within the VSA

No perfect of measure of students learning exists for all audiences External accountability Institutional improvement College selection

Working Group Deliberations Make value-add measurement using

existing measures (CAP, CLA, ETS Proficiency Profiler) optional

Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes (written communication and critical thinking)

Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT

Work Group Ratings of Recommended VSA Instruments in Key Areas

Instrument1= lowest rating 10=highest rating

CLA CAAP ETS PP(short form)

ETS PP (long form)

VALUE Rubrics GRE

Allow comparisons, benchmarking across institutions? 6 8 9.5 9.5 6 7

Offer a representative sample? 7 7 7 7 7 2 Usefulness to faculty/students for learning improvement?

inst-2class-7

5 2 2 9 3

Transparency of method, results 6 8 8 8 9 8

External accountability 8 7 7 7 5 5.5

Ease of administration on campus 5 6 7 6 4 9.9

Reasonable costs (time, resources) 5 7 7 6 startup -3 marginal- 8

9

Motivation for students to do well? 3 3 3 3 9 9.9 Likely interest to consumers? 2 2 2 2 5 6

Valid learning outcomes instrument for average student? 5.5 6 6 6 portfolios -9

assign -7 3

VSA Board Decisions College Portrait re-focus from “college selection tool for

prospective students” to “consumer information tool for institutions to demonstrate effectiveness of educational programs.” Primary audiences: state officials, policy makers,

consumers Instrument options:

CAAP, CLA, ETS Proficiency Profile AAC&U VALUE rubrics – critical thinking, written

communication GRE General Test

Reporting options: value-added, benchmarking

Implementation/Next Steps All VSA institutions report results of pilot by

end of 2012 to fulfill reporting requirement 3 part report with tabbed display

Did you participate (or try to)? If so, how’d it go? If not, why not?

Give an example of another institutional-level evaluation

Which of the new options are you considering for the future?

Reporting Format Preview: http://www.collegeportraits.org/slo_preview.html

Implementation/Next Steps Communications Task Force working with College

Portrait Support Team on display of new options Emphasis on more graphical displays Additional guidance on what the scores and results

mean Click-through links for additional details, e.g., subscores

New options released for 2012-13 data cycle (Jan 2013)

Develop outreach strategy to recruit new participants and connect with key audiences – state officials, accreditors

MORE INFORMATIONWebsites www.voluntarysystem.orgwww.collegeportraits.orgwww.learningoutcomesassessment.org

Teri Hinds: thinds@aplu.orgNatasha Jankowski: njankow2@illinois.edu

Let’s Discuss.