Upload
erol
View
43
Download
4
Tags:
Embed Size (px)
DESCRIPTION
The Future of Student Learning Outcomes Assessment within the VSA. Christine Keller VSA Executive Director Teri Lyn Hinds VSA Associate Director. Higher Learning Commission Annual Conference Chicago, IL ▪ April 7, 2013. Presentation Outline. - PowerPoint PPT Presentation
Citation preview
THE FUTURE OF STUDENT LEARNING OUTCOMES ASSESSMENT WITHIN THE VSA
Higher Learning Commission Annual ConferenceChicago, IL ▪ April 7, 2013
Christine KellerVSA Executive Director
Teri Lyn HindsVSA Associate Director
PRESENTATION OUTLINE• Review original goals of VSA, student
learning outcomes pilot project• Discuss selected findings, recommendations
from NILOA evaluation of SLO Pilot, participant reports
• Describe use of evaluation results: technical work group, communications advisory group
• Outline next steps for VSA: new reporting options, refocus/rebranding
VOLUNTARY SYSTEM OF ACCOUNTABILITY
the undergraduate student experience through a common web report – the College Portrait.
www.collegeportraits.org
Initiative by public universities to supply straightforward, comparable information on
No rankings, no
spin
Just the facts
!
VSA OVERVIEW• Developed, launched in 2007
– 70 institutions, 80 leaders– APLU and AASCU partnership
• Three original objectives– Demonstrate accountability,
transparency– College search tool– Support innovation in the measurement,
reporting of student learning outcomes• 300 participating public universities
STUDENT LEARNING OUTCOMES PILOT• Goal: Directly measure, publicly report
student learning gains (value-added) at institution level using a common method– Skills: critical thinking, analytic reasoning,
problem solving, written communication– Tests: CAAP, CLA, ETS Proficiency Profile
• Reporting includes link to institution specific learning outcomes data
• Pilot period ended December 2012
EVALUATE EFFECTIVENESS OF PILOT• Conducted by National Institute for
Learning Outcomes Assessment (NILOA)– Focus groups– Interviews– Surveys– Google Analytics– College Portrait statistics
NILOA’s mission is to document student learning outcomes assessment work, identify and disseminate
best practices, and support institutions in their assessment efforts.
SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN
www.learningoutcomesassessment.org
FINDINGS I• VSA in 2007 was a wise, timely, useful,
necessary response to the accountability , transparency demands of the time
• Many eligible institutions - about 1/3 - do not participate in the VSA
• 50% of participating institutions missing student learning outcomes information
• The student learning outcomes section of attracts little traffic
FINDINGS II• College Portrait: “Information posted may
not reflect the needs of prospective students, families or provide the information they seek to make decisions about where to attend college.”
• Student Learning Outcomes: “The standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community, undermining institutional participation and engagement with the VSA and campus faculty and staff support of the VSA initiative.”
RECOMMENDATIONS
“Mend it, don’t end it”• Expand range of assessment tools and
approaches• Focus on specific audiences and
communicating meaningful information
WORKING GROUP DELIBERATIONS• Technical work group considered alternative
measures of learning outcomes to recommend to VSA Board
• Confirmed importance of student learning outcomes reporting within the VSA
• No perfect of measure of student learning exists for all audiences– External accountability– Institutional improvement– College selection
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument1= lowest rating 10=highest rating
CLA CAAP ETS PP(short form)
ETS PP (long form)
VALUE Rubrics GRE
Allow comparisons, benchmarking across institutions? 6 8 9.5 9.5 6 7
Offer a representative sample? 7 7 7 7 7 2 Usefulness to faculty/students for learning improvement?
inst-2class-7
5 2 2 9 3
Transparency of method, results 6 8 8 8 9 8
External accountability 8 7 7 7 5 5.5
Ease of administration on campus 5 6 7 6 4 9.9
Reasonable costs (time, resources) 5 7 7 6 startup -3 marginal- 8
9
Motivation for students to do well? 3 3 3 3 9 9.9 Likely interest to consumers? 2 2 2 2 5 6
Valid learning outcomes instrument for average student? 5.5 6 6 6 portfolios -9
assign -7 3
WORKING GROUP DELIBERATIONS II• Continue use of value-added
measurement using CAAP, CLA, ETS Proficiency Profile
• Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes
• Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT
VSA BOARD DECISIONSBroader Focus & Outreach
– Evolution from college selection website to consumer information tools
– New audience focus: state officials, accreditors
Expanded Learning Outcomes Options– Original tests - CAAP, CLA, ETS PP – Plus AAC&U VALUE rubrics – critical
thinking, written communication– Both value-added, benchmarked reporting
PARTICIPANT PILOT EXPERIENCE REPORT• Pilot project wrap-up: VSA participants
describe, publish pilot project experiences using three part form
• 92 institutions completed the Pilot Experience Report– 65 participated – administered or tried to
administer one of the instruments– 27 did not participate
PARTICIPANT EXPERIENCES• 27 institutions posted results to their
College Portrait– 16 found the results useful on campus– CLA was the most frequently reported
• 38 institutions tried to administer an instrument– 14 reported difficulty meeting minimum
sample size requirements– 8 reported campus concerns with sample
sizes, even when they met publisher recommendations
NON PARTICIPANT EXPERIENCES• 27 institutions did not participate
– Campus perceptions that the instruments wouldn’t provide useful results at 9 institutions
– 7 institutions has more pressing institutional priorities
– 6 institutions were concerned about the monetary cost of the instruments
FUTURE SLO REPORTING PLANS• Institutions asked to select all options under
consideration by campus – 28 institutions considering more than one
option– Many institutions exploring benchmarking and
value-added for the same instrument• 44 institutions considering the ETS PP
– Either/both value-added, senior-only benchmark administrations
• 34 institutions considering AAC&U VALUE Rubrics
EXPANDED FOCUS & OUTREACH• Form Communication Advisory Group from
public affairs, communications, government affairs
• Promote key elements on College Portrait: success and progress rate, cost of attendance, net price calculator, financial aid, future plans
• Re-connect with key audiences: assessment experts, state/system officials, accreditors
NEW TOOLS FOR USERS• Institutions can publish Success &
Progress Rate graphs and tables for customized student groups– Part-time students, veterans, Pell-
recipients• Augmented cost, financial aid info
– Net price and financial aid by income range, average indebtedness, average loan payment
UPCOMING RELEASES• VSA Affordability Estimator
improvements provide more accurate net price estimates, clearer presentation of results (May)
• Learning Outcomes Pilot Project Report will summarize campus experiences, lessons from 4-year pilot (May)
• At-a-Glance tool for government affairs staff will offer streamlined, summarized view of key indicators (August)
BOTTOM LINE• Unprecedented challenges for public
higher education to be more accountable and more transparent concerning value, outcomes
• Work to position the VSA/College Portrait as a tool to respond to those challenges
College Portrait website: www.collegeportraits.org
Reports & Presentations: www.voluntarysystem.org
Twitter: @VSAvoiceBlog: http://vsavoice.blogspot.com/
Christine KellerExecutive Director [email protected]
Teri Lyn HindsAssociate [email protected]