Upload
umucedu
View
198
Download
0
Embed Size (px)
Citation preview
A Multi-Institutional Study of Student Success Course ImpactBeth Mulherrin, University of Maryland University College
Sandy Daston, PAR Framework
Figuring Out What Works For Student Success
Public university offering online degree programs to a diverse population of working adults
Largest open access public online university in U.S.
Premier provider of higher education to U.S. military since 1949
Part of the University System of Maryland
About UMUC
Jumpstart
Focus on Connection and Entry touchpoints (readiness and motivation, goal setting and commitment)
Problem-solving approach (diagnostics, tools for self-assessment, career goals)
Areas of strength and concern-academic and personal- to support goals and mitigate risks
CAPL 101
Setting the Stage
Online Learning Readiness
Jumpstart Results- RetentionOne Year Retention RatesFall 2013:
◦Successful JS Completers vs. Successful Matched: 96.8% versus 79.3%
Spring 2014:
◦All JS Participants vs. All Matched: 88.5% vs. 58.4%
◦Successful JS Completers vs. Successful Matched: 93.0% versus 57.1%
Jumpstart Results- Re-enrollmentSignificantly higher rates of re-enrollment into next consecutive term for:◦ Fall 2013:
◦ All JS vs. All Matched: 90.0% vs. 70.3%◦ Successful JS vs. Matched: 96.8% vs. 72.4%
◦ Spring 2014◦ All JS vs. All Matched: 68.3% vs. 38.2%◦ Successful JS vs. Matched: 73.2% vs. 36.5%
◦ Fall 2014 (New Students Only)◦ Successful JS vs. Matched: 78.9% vs. 66.1%
PAR ResearchCommon elements for positive first year retention outcomes with reviewed success courses include:
-academic planning/degree planning-information about program of study-connection with advising-study skills, time management-assessment of strengths + possible barriers to success
The Predictive Analytics Reporting (PAR) FrameworkLaunched as a research project at the Western Interstate Commission for Higher Education’s Cooperative for Educational Technologies.
Funded by Bill & Melinda Gates Foundation 2011, 2012, 2014
Supported by 6, then 16, then 22 forward-thinking institutions and managed by WCET to evaluate and test the use of predictive analytics to explore student loss and momentum.
Now a self-sufficient and Independent 501.c.3.
What PAR Does
PAR uses descriptive, inferential and predictive analyses to create benchmarks, institutional predictive models and to inventory, map and measure student success interventions that have direct positive impact on behaviors correlated with success.
PAR by the Numbers2.2 million students and 20 million courses in the PAR data warehouse, in a single federated data set, using common data definitions.
48 institutions
77 discrete variables are available for each student record in the data set. More than 2 dozen additional constructed variables used to explore specific dimensions and promising patterns of risk and retention.
Over 450 institution interventions organized by predictors they address and point in student life cycle.
Structured Readily Available Data
Common data definitions = reusable predictive models and meaningful comparisons.
Openly published via a cc license @ https://public.datacookbook.com/public/institutions/par
Data Inputs- Common Data Definitions
Student Demographics & Descriptive
GenderRace
Prior CreditsPerm Res Zip Code
HS InformationTransfer GPAStudent Type
Student Course Information
Course LocationSubject
Course NumberSection
Start/End DatesInitial/Final Grade
Delivery ModeInstructor Status
Course Credit
Student Academic Progress
Curent Major/CIPEarned Credential/CIP
Student Financial
InformationFAFSA on File – Date
Pell Received/Awarded –Date
Course CatalogSubject
Course NumberSubject LongCourse Title
Course DescriptionCredit Range
Lookup TablesCredential Types Offered
Course Enrollment PeriodsStudent Types
Instructor StatusDelivery Modes
Grade CodesInstitution Characteristics
Possible Additional Placement TestsNSC InformationSES Information
Satisfaction SurveysCollege Readiness Surveys
Intervention Measures
Intervention MeasurementUsual Analysis StepsFind a Reasonable Comparison Group
Ensure Data Understanding & “Completeness”
Define Outcome Measures◦ Term: GPA, >=C Credits Earned, Credit Ratio, Withdrawals, Credits Attempted
◦ Term to Term: Deltas
◦ Retention (if possible)
Iterations◦ Simple binary analysis- no models
◦ Apply models
©PAR Framework 2015
Intervention MeasurementChallengesFind a Reasonable Comparison Group
◦ Often no real controls
◦ Pre implementation vs. Post Implementation
◦ Participated vs. did not (optional interventions/course by course)
◦ Determine control variables
Ensure Data Understanding & Completeness
©PAR Framework 2015
Intervention Measurement–Student Success Courses
ObjectivesDraw comparisons between program elements
◦ existing PAR Framework data set
◦ existing PAR access to information about student success courses
Compare cross institutional results
◦ using PAR available outcomes measures
©PAR Framework 2015
Intervention Measurement –Student Success Courses
Step 1: Identify Participating PartnersOnline Survey to Partners (11 respondents)
◦ Student Success Course Offerings
◦ Academic Planning
◦ New Student Orientation
◦ College Readiness Assessment
©PAR Framework 2015
Step 1: Identify Participating PartnersStudent Success Course/Academic Planning Offerings # of Respondents
Student Success Course Required for All Students 4
Student Success Course Required for Some Students 5
No Student Success Course Offering 2
Have Academic Planning/Degree Mapping Intervention
(varying % of students impacted)
10
No Academic Planning/Degree Mapping Interventions 1
©PAR Framework 2015
Intervention Measurement –Student Success Courses
Intervention MeasurementStudent Success Courses
Step 1: Identify Participating PartnersAvailability of a comparison group
◦ Implemented after 2009 (0)
◦ Required for some, not all (5)
©PAR Framework 2015
Intervention MeasurementStudent Success Courses
Step 1: Identify Participating PartnersSinclair Community College (1 credit)
University of Central Florida (3 credits)
University of North Dakota (2 credits)
Rio Salado Community College (1 or 3 credits)◦ 1- credit short course
◦ 3-credit in-depth course
Excelsior College (3 credits)◦ For Military/Veteran students
◦ For all other students
©PAR Framework 2015
Intervention MeasurementStudent Success Courses
Step 2: Additional Info GatheredWho took the course? (required or strongly encouraged)
◦ Lack preparation/poor prior performance
◦ Special populations
◦ Poor performance at the institution
◦ Certain majors and/or undeclared
Course syllabus
Type of instructor teaching the course
©PAR Framework 2015
Intervention MeasurementStudent Success Courses
Step 3- Analysis Qualitative
Quantitative
©PAR Framework 2015
Intervention MeasurementStudent Success Courses
Step 3- Qualitative AnalysisDetermine Primary Course Components
Sources◦ Survey Responses
◦ Syllabus
©PAR Framework 2015
Student Success Course Components # of courses
Academic Planning/Degree Mapping 8
Information about Programs of Study 8
Connection with Advising Services 8
Study Skills/Time Management 8
Social Engagement Opportunities 8
Assessment of Strengths and Possible Barriers to
Success8
Information about Academic Support Resources 7
Career Planning 7
Financial Aid/Money Management Information 6
Being an Engaged Learner/ Critical Thinking 6
Explicit Goal Setting 6
Student Success Course Components # of Courses
Information Literacy/Research Skills 5
Being a Student HERE (at this institution) 5
Being a College Student (various ways/expectations
for independence)5
Instruction on Technologies Used (Blackboard,
Google Drive, etc.)4
Health & Wellness 3
College Readiness Assessments 2
Community Service Project 1
Common Read 1
Peer Mentor Assigned to the Course 1
Intervention Measurement Student Success Courses
Step 3- Quantitative Analysis (Performed by individual institution)Determine Target & Comparison Groups
◦ Treatment Group: Students taking the Student Success Course, all PAR years
◦ Comparison Group: Students who did not take the Student Success Course
©PAR Framework 2015
Step 3: Quantitative Analysis
Outcomes StudiedRetention: active course records between 12 months and 18
months after first course start date, or if they credentialed during or before that same period
Credit Ratio: # credits earned with a C or better grade in the student’s first 12 months divided by the student’s credits attempted in the same time period (binned into 2 categories around the institutional mean)
©PAR Framework 2015
Step 3: Quantitative Analysis
Identify Control Variables (Per institution)
• Credential Level Sought (Associates, Bachelors, Certificate)
• Development Education Participation
• Delivery Mode (Fully Online, Fully Onground, Mix of both)
• Pell Grant• Full/Part Time• GED • Military or Reservist
• Veteran• Race• Student Age• Prior Credits (Yes or No)• High School GPA (applicable only
for institutions with available data on HS GPA)*
• Gender• Student Type (per the institution’s
definitions)
* When used, students with no data were not included in the analysis
©PAR Framework 2015
Step 3: Quantitative Analysis
Control Variables - Details◦ For 1 institution: used additional info on intervention exposure
◦ For 1 institution: limited analysis to eliminate known extremely high risk students
©PAR Framework 2015
Step 3: Quantitative Analysis
Methods◦ Logistic Regression
◦ Effect of course participation was added after the significant control variables had been entered in the model as predictors
◦ Model selection was based on a balance of model fit and inclusion of appropriate control variables
©PAR Framework 2015
Step 3: Quantitative Analysis
Proportion of students taking vs not taking the student success course varied markedly, from a high of approximately 40% of the student population taking the course at one institution to a low of less than 1% of students at the other end of the range.
©PAR Framework 2015
Step 3: Quantitative Analysis Results
(p values ranged from <.02 to <.0001, with most in the latter highly significant range)
©PAR Framework 2015
Results Summary
Only one of the eight Student Success Courses analyzed showed a statistically significant positive effect on 12 month credit ratio for students taking the course vs. those who did not
For retention, six of seven courses showed a significantly positive effect on retention (1 course not in effect long enough to measure retention)
©PAR Framework 2015
Course Component Summary
©PAR Framework 2015
Course Component Summary
©PAR Framework 2015
Interpretation & Next Steps
Important to consider factors not controlled for (other interventions, past performance)
Optional courses may attract more motivated students
More exploration on factors that make a difference/patterns
Faculty connection
©PAR Framework 2015
Summary & Contact Info