Leveraging analytics to Improve Student Success

Embed Size (px)

Text of Leveraging analytics to Improve Student Success

Leveraging Analytics to Improve Student Success Karen Vignare, University Maryland University College @kvignare Ellen Wagner, PAR Framework @edwsonoma

Leveraging Analytics to Improve Student Success

Karen Vignare, University Maryland University College@kvignare

Ellen Wagner, PAR Framework@edwsonoma

Session DescriptionThis session shows how analytics can be used to identify opportunities for improving student success.By the end of the session, participants will make connections between predictions about risk, and the interventions most likely to work best under varying conditions and with different populations.

Setting the Context: Data Are Changing Everything

But education researchers have always worked with data. We do qualitative research with dataWe do quantitative research with dataWe do evaluations with dataWe develop surveys and instruments and experiments to collect more dataWe pull data from LMSs, SISs, ERPs, CRMs We write reports, summaries, make presentations, develop articles and books and webcasts.

From Hindsight to Foresight


Analytics in Higher EducationLearning AnalyticsBest way to teach and learnLearner Analytics Best way to support students Organizational AnalyticsBest ways to operate a college

Academic Analytics

Analytics is not one size fits all3 major areas of analytics in HE, according to RussLearningThe act & process of learning, Curricular, Best way to teach and learnLearnerDemographics, Behaviors, Best way to support students (Individually is the goal)OrganizationalCapacity, budget, scheduling, Best way to operate a college


Create new insights and opportunities for data in our practicesEnrollment managementStudent servicesProgram and learning experience designContent creationRetention, completionGainful employmentInstitutional Culture

How Are We Doing So Far?Data is the number 1 challenge in the adoption and use of analytics. Organizations continue to struggle with data accuracy, consistency, access.The primary focus of analytics focuses on reducing costs, improving the bottom line, managing risk.Intuition, based on experience, is still the driving factor in data-driven decision-making. Analytics are used as a part of the process.Many organizations lack the proper analytical talent. Organizations that struggle with making good use of analytics often dont know how to apply the results.Culture plays a critical role in the effective use of data analytics. 9

GROUP DISCUSSIONIs your institution using (or planning to use) academic analytics specifically to improve student success?What kinds of questions are you trying to answer?What kinds of data are you planning to use? What kinds of barriers are you encountering?

Getting to the right answer takes workAnalysis and model building is an iterative processAround 70-80% efforts are spent on data exploration and understanding.

SAS Analysis/Modeling Process

Link Predictions to ActionPredictive analytics refer to a wide varieties of methodologies. There is no single best way of doing predictive analytics. You need to know what you are looking for.Simply knowing who is at risk is simply not enough. Predictions have value when they are tied to what you can do about it.Linking behavioral predictions of risk with interventions at the best points of fit offers a powerful strategy for increasing rates of student retention, academic progress and completion.

Collaborative National Multi-institutional Non-profit Institutional Effectiveness +Student Success


What PAR doesPAR uses descriptive, inferential and predictive analyses to create benchmarks, institutional predictive models and to inventory, map and measure student success interventions that have direct positive impact on behaviors correlated with success.

Linking Predictions to ActionIdentify obstacles and remove barriers from student success pathways.Provide actionable information so students and advisors can build informed opportunity pathways.Know where to invest in student success leveraging collaborative insight that determine return on investment in interventions and support.


Benchmarks & Insight Predictive Analytics Intervention Inventory and ROI Tools

Diagnostics PAR analytic toolset

Place Holders for Demo Sections


Benchmarks & Insight Predictive Analytics Intervention Inventory and ROI Tools

Web Tools

Student Success Matrix (SSMx)

Place Holders for Demo Sections


PAR by the Numbers2.2 million students and 24.5 million courses in the PAR data warehouse, in a single federated data set, using common data definitions. 48 institutions, 351 unique campuses.77 discrete variables are available for each student record in the data set. Additional 2 dozen constructed variables used to explore specific dimensions and promising patterns of risk and retention.343 discrete interventions filtered on predictor behaviors, point in student life cycle, student attributes, institutional priorities and ROI factors in the growing SSMx dataset.

Structured, Readily Available DataCommon data definitions = reusable predictive models and meaningful comparisons. Openly published via a cc license @ https://public.datacookbook.com/public/institutions/par


Speak the same language

PAR Puts it All Together

Findings from aggregated dataset Positive PredictorsHigh school GPA (when available)Dual Enrollment HS/CollegeAny prior credit CC GPACredit Ratio Successful Course Completion Positive completion of DevEd Courses

Negative PredictorsWithdrawals Low # of credits attempted

Varies but can be significant PELL Grant Recipient Taken Dev EdAge Fully online student Race


Measurement resources are usually located separately from intervention planning & implementation resourcesLack of connection of predictors to interventions and interventions to outcomes

PAR Framework 2015Common Challenges for Intervention Effectiveness

PAR Student Success Matrix (SSMx)An organizational structure that helps institutions inventory, organize and conceptualize interventions aimed at improving student outcomes.A common framework for classifying interventionsProvides a basis for intervention measurementPAR Framework 2015

SMALL GROUP DISCUSSIONHow Are You Measuring Interventions at YOUR Institution?

Specific Examples of Data Driven ImprovementsUMUC / U of Hawaii replication of community college success prediction studiesU of Hawaii Obstacle coursesUniversity of North Dakota predictives tied to student watchlist dataIntervention measurement at Sinclair CC and Lone Star CCNational online learning impact study on student retention (in press, based on results from >500,000 students taking onground, blended and online courses)

Intervention Measurement Student Success Courses Results

12 month credit ratio: Only 1 of the 8 Student Success Courses analyzed showed a statistically significant positive effect for students taking the course vs. those who did not.

Retention: 7 of the 8 courses showed a significantly positive effect

Retention higher by 14% to 4X

Intervention Measurement Student Success CoursesCourse Component Summary:

Public university offering online degree programs to a diverse population of working adults Largest open access public online university in U.S.Premier provider of higher education to U.S. military since 1949Part of the University System of MarylandAbout UMUC

20th CenturyHistoricalLongitudinalWarehouseSiloed ExternalReporting

21st CenturyPredictiveReal-TimeDashboardsIntegrated Institutional InsightsContinuous Improvements

Evolution of Data for Retention

Institutional ResearchInstitutional EffectivenessBusiness IntelligenceCivitas Learning, Inc.PAR Framework, Inc.Retention Resources at UMUC

Pre-enrollmentDemographicsEnrollmentLMS EngagementStudent PerformanceTransferMilitary

Factors Included in Predictive Model for Retention at UMUC

CampusClass LoadMilitary StatusAcademic PerformancePayment Method

Key Factors for Retention at UMUC

We can give examples from each category34

One year retention (year over year measured with a cohort)Re-enrollment (term to term metric that includes all students)Successful course completion (percentage of students receiving a successful grade)Graduation (1,2,3,4,5, and 10 year rate tracks the graduation status of the starting cohort over time)

Metrics at UMUC

These are commonly used to report retention (as opposed to measuring success)35

Curriculum Redesign (2010)8-week Standard Sessions (2010)Community College Transfer (2010)Registration Policy (2013)Onboarding (2014)Just-in-Time Messages (2014)

Retention Initiatives

DiscussionHow will you begin, or improve, your analytics journey at YOUR institution?

Elements of a Data Model

Use modeling to Test likely impact on retention when new initiatives or planned interventions are undertaken

Create models that build out retention impact by segments, e.g., demographics, academic programs, persistence, etc.

Continual Improvement

Based on PAR42


Thank you for your INterest