View
8
Download
0
Category
Preview:
Citation preview
Redefining “At-Risk” Through Predictive Analytics: A Targeted Approach to
Enhancing Student Success
Amilcah Gomes
Assistant Director, Academic Services Center
Eastern Connecticut State University
Predictive Analytics
• Using real-time data to plan for the future…
• “Predictive analytics helps your organization predict with confidence what will happen next so that you can make smarter decisions and improve business outcomes.” –IBM
Institutions That Use Predictive Analytics
• Predictive Analysis Reporting (PAR) Framework Institutions – American Public University
System – Ashford University – Broward College – Capella University – Colorado Community
College System – Lone Star College System – Penn State World Campus – Rio Salado College – Sinclair Community College – Troy University
– University of Central Florida – University of Hawaii System – University of Illinois
Springfield – University of Maryland
University College – University of Phoenix – Western Governors
University
• Purdue University • Rutgers University • University of South Florida • Eastern Connecticut State
University • Others
Eastern Connecticut State University
• Location: Willimantic, Connecticut • Institution Type: Public, Liberal Arts University • Total Enrollment (2012): 5,237 students, 4,506
undergraduate FTE • Nearly 2/3 residential, with 93% of first-time
students living on campus • 2011 FTFT cohort (N = 931):
– Female 53.4%; Students of Color 22% – Other Characteristics – First Gen Students (32.5%),
Compass Cohort (50.1%), STEP/CAP students (6.8%), Honors Scholars (2.4%), Athletes (9.7%)
• 2012 FTFT cohort (N = 979) • 2013 FTFT cohort (N = 960)
Institutional Changes
• Strategic Plan: Student Success Initiative (2008-2013) – Dual Advising Model
• Project Compass (2008-2012) – Communities of Practice
– Early Identification of At-Risk Students
– Enhancing First-Year Advising Services
– Targeting tutoring services to high-risk subjects
• Title III Student Support Services (2009-2014)
• 2013-2018 Strategic Plan & Targeted Student Success Initiatives
Development of Prediction Models
• 2008 – Multivariate model developed for predicting withdrawal prior to second year – Original assumption: students withdraw due to poor
academic performance – Problem: Original model failed to differentiate
between students who left for academic and non-academic reasons • Did not account for financial reasons, motivational variables,
engagement factors, etc. • Withdrawal risk quintiles were difficult to interpret • Professional advisors were unable to develop advising
strategies specific to a student’s needs
• 2011 – Developed additional multivariate model for academic risk; implemented two-model approach
Variable B S.E. Wald df Sig. Exp(B) Base
male -.087 .122 .508 1 .476 .917 female
black -.723 .278 6.786 1 .009 .485* white
hisp .347 .279 1.540 1 .215 1.414
oth_race -.674 .274 6.069 1 .014 .510
not_east .408 .133 9.401 1 .002 1.503 East CT River
commuter .497 .211 5.545 1 .019 1.644 Campus Pell_yr1 -.245 .161 2.316 1 .128 .783
Not Pell first_gen -.076 .124 .378 1 .539 .927
Not FGEN Athletics -.647 .218 8.816 1 .003 .524 Not Athlete HsGpa_quint1 .734 .189 15.070 1 .000 2.084
Quintile 3 HsGpa_quint2 .242 .184 1.727 1 .189 1.274
HsGpa_quint4 .277 .186 2.228 1 .136 1.319
HsGpa_quint5 .080 .223 .130 1 .719 1.084
admit_rating_le_4 -.057 .160 .125 1 .723 .945 Rate 5, 6, or 7 admit_rating_ge_8 -.573 .203 7.985 1 .005 .564
Vsat_quin1 -.069 .184 .140 1 .708 .934 VSAT Quint 3 Vsat_quin2 -.320 .180 3.157 1 .076 .726
Vsat_quin4 .191 .173 1.216 1 .270 1.210
Vsat_quin5 .172 .186 .854 1 .356 1.188
Stem -.036 .177 .042 1 .838 .964 Not STEM PreEd -.263 .165 2.536 1 .111 .769 Not PreED Undec .121 .132 .837 1 .360 1.129 Declared major ERG_none .484 .201 5.834 1 .016 1.623 ERG DEF ERG_ABC .164 .151 1.181 1 .277 1.178
ERG_GHI .269 .153 3.089 1 .079 1.309
got_schol_yr1 -.022 .165 .018 1 .893 .978 No schol got_FedLoan_yr1 -.227 .125 3.326 1 .068 .797 No Fed Loan Choice -.246 .120 4.235 1 .040 .782 Not #1/No
FAFSA Constant -1.282 .241 28.336 1 .000 .277
*Factors that are significant at the 0.10 level have been highlighted.
Withdrawal Model, 2011 Cohort (2008, 2009 Data)
Academic Risk Model, 2011 Cohort (2008, 2009 Data)
Model 2: for classifying 2011 Cohort into Academic-Risk Quintiles, Based on Data from 2008 and 2009 Cohorts.
B S.E. Wald df Sig. Exp(B)
HsGpa -1.505 .180 69.974 1 .000 .222
ERG_DEF .222 .151 2.165 1 .141 1.249
ERG_GHI .300 .157 3.657 1 .056 1.350
ERG_none .091 .204 .201 1 .654 1.096
black .365 .222 2.714 1 .100 1.441
hisp .429 .275 2.445 1 .118 1.536
oth_race -.194 .245 .629 1 .428 .824
Pell_yr1 .140 .148 .891 1 .345 1.150
first_gen .028 .123 .052 1 .820 1.028
male .344 .121 8.039 1 .005 1.410
Stem .387 .168 5.325 1 .021 1.472
PreEd -.274 .172 2.555 1 .110 .760
Athletics -.641 .216 8.770 1 .003 .527
admit_rating_le_4 .186 .144 1.663 1 .197 1.204
admit_rating_ge_8 -.414 .200 4.271 1 .039 .661
Constant 2.952 .557 28.125 1 .000 19.150
Interpreting the Models: What We Saw
Commuter
Not East
Federal Loans
FAFSA Choice
HS GPA
ERG/DRG (CT HS districts)
Athlete (negative, both)
African American identity (negative withdrawal, positive academic risk)
Admissions Rating ≥ 8 (negative, both)
Males
STEM Majors
Only Significant in Withdrawal Model
Only Significant in Academic Risk Model
Factors Significant in Both Models
Factors that had a significant impact (0.10 level):
Interpreting the Models: What We Didn’t See
• Factors that did not have a significant impact (0.10 level):
Pell eligibility (less significant in
GPA model)
1st Generation Status
Multiracial Identities (added to 2012
model)
Undeclared Majors (significant for 2nd to
3rd year retention)
Math SAT (good predictor of
GPA > 3.0)
Classifying Withdrawal & Academic Risk
Academic Risk Quintile 2011
Total 1.00 2.00 3.00 4.00 5.00
Q_2011 1.00 99 57 35 35 24 250
2.00 45 47 39 33 23 187
3.00 29 49 50 45 45 218
4.00 10 32 30 49 35 156
5.00 2 15 15 26 62 120
Total 185 200 169 188 189 931
TAC 1 = Intensive 172
TAC 2 = Tutoring 205
TAC 3 = Engaged 232
TAC 4 = Monitor 322
Cross-classification of 2011 cohort of entering first-time, full time students by the two types of risk: withdrawal and low academic performance:
Targeted Advising Cohorts (TAC)
• TAC 1 = Intensive (high risk withdrawal & high risk GPA < 2.3)
• TAC 2 = Tutoring (low risk withdrawal & high risk GPA < 2.3)
• TAC 3 = Engaged (high risk withdrawal & low risk GPA < 2.3)
• TAC 4 = Monitor (low risk withdrawal & low risk GPA < 2.3)
Targeted Advising Cohorts (TAC)
TAC 2 Tutoring
Low risk withdrawal
High risk GPA < 2.3
TAC 1 Intensive
High risk withdrawal
High risk GPA < 2.3
TAC 4 Monitor
Low risk withdrawal
Low risk GPA < 2.3
TAC 3
Engaged
High risk withdrawal
Low risk GPA < 2.3
High withdrawal
risk
Low withdrawal
risk
High academic risk
Low academic risk
Fall 2011 Cohort TAC Assignments*
18.5%
25.5%
22.8%
33.2%
TAC1=Intensive
TAC2=Tutoring
TAC3=Engaged
TAC4=Monitor
* 35 students originally in TACs 3 and 4 who did not participate in the library assessment and orientation were reassigned to TAC2.
By using the two-model approach it was determined that 18.5% of the FTFT students entering were at risk of withdrawal and low academic performance (TAC1), and another 25.5% were at risk of low academic performance (TAC2).
Observations
• Gender – Males made up less than half (46.6%) of 2011 cohort,
but represented 67% of TAC 1 & 2 students
• Race/Ethnicity – Students of color, particularly African American/Black
students, were overrepresented in TAC 2 (46.3% vs. 23.3% overall)
• Home of Record – 80.2% of TAC 1 students were “Not East”
(international, out of state, and students living west of the CT river) students (56.9% overall)
• Other Characteristics – 68.8% of Pell eligible and first generation students
were classified in TACs 2 & 4
Summary of Outcomes
• There is very little variation in credits attempted across the four TACs • Students in TAC 1 and TAC 2 on average were slightly less likely to achieve
the same level of academic momentum as their classmates in terms of credits earned
• In addition, the average GPAs for students in TAC 1 and TAC 2 are significantly lower than those in TAC 3 and TAC 4
• TAC assignment is clearly related to retention • TAC 3 and TAC 4 more likely to utilize tutoring services and devote more
hours with math and writing tutors on average
FTFT 2011 General Characteristics Outcomes (Average)
N %
Female % 1st Gen
% Students of Color # Credits Att. # Credits Earned
1st Yr. GPA Lib.Score Retention
927 53.4 32.5 22.0 29.40 26.18 2.79 1.11 75.5 TAC1 171 32.2 27.5 15.7 28.88 24.18 2.34 0.98 67.3 TAC2 235 37.0 40.0 42.4 28.83 23.13 2.38 0.99 74.9 TAC3 210 66.7 23.8 3.4 29.23 27.85 3.10 1.15 71.9 TAC4 306 69.0 35.0 22.6 30.23 28.49 3.14 1.23 83.7
Data-Informed Approach
• Reassess assumptions about student withdrawal patterns and ALANA student performance on campus
• Align institutional priorities to significantly enhance student success
• Better allocate limited staffing and financial resources to support high-impact practices
• Address persistence and performance issues among high-risk groups
Developing Targeted Interventions
• Freshman Preference Registration
• Developmental Advising Focus
• Leading Indicators Project (Library Orientation Score)
• Collaboration with Campus Departments
• Registration Holds and Financial Review
• Diversity Scholars Program
• Strategies for High-Performing Students
• Recognition of ALANA Student Success
Intervention: Major Course Selection
First-Semester GPA for FTFT 2011 & 2012 cohorts by TAC
TAC FTFT 2011 Cohort
Mean CGPA
N FTFT 2012 Cohort “Major Course” CGPA
N FTT 2012 Cohort No “Major Course”
CGPA
N
1 2.35 168 2.54 114 2.48 50
2 2.28 199 2.64 84 2.32 53
2A 2.49 32 2.90 36 3.31 3
3 3.10 205 3.28 121 3.07 55
4 3.15 304 3.25 191 3.22 77
Total 2.78 908* 2.99 546 2.83 238
* 23 students did not complete the Fall 2011 semester and were not included in the first semester GPA analysis
Intervention: Major Course Selection
TAC FTFT 2011 Cohort Returned Spring
2012
FTFT 2012 Cohort Returned Spring
2013
FTFT 2012 Cohort “Major Course” Returned Spring
2013
FTFT 2012 Cohort No “Major
Course” Returned Spring 2013
1 89.0% 90.0% 93.0% 84.9%
2 89.7% 93.0% 95.2% 90.7%
2A 82.9% 93.3% 94.4% 75.0%
3 89.4% 94.0% 95.0% 91.2%
4 94.8% 95.0% 95.3% 94.9%
Total 90.9% 93.3% 94.7% 90.7%
First-Semester Persistence for FTFT 2011 & 2012 cohorts by TAC
Intervention: Major Course Selection Year 1 Persistence to Year 2 Year 1 Academic Performance Outcomes (Average)
Cohort Overall
No Maj Crse ECO 200 BUS 201
Cume GPA
No Maj Crse
Cume GPA ECO 200
ECO 200 grade in 1st Semester
Cume GPA BUS 201
BUS 201 grade in 1st Semester
FTFT 2011 75.6% 72.5% 77.5% 82.4% 2.65 2.54 2.70 2.52 (C+/B-) 3.19 3.05 (B)
FTFT 2012 75.6% 70.0% 77.2% 80.9% 2.86 2.71 2.88 2.46 (C+/B-) 2.92 3.15 (B/B+)
Year 2 Persistence to Year 3 Year 2 Academic Performance Outcomes (Average)
Cohort Overall
No Maj Crse ECO 200 BUS 201
Cume GPA
No Maj Crse
Cume GPA ECO 200
ECO 200 grade in 1st Semester
Cume GPA BUS 201
BUS 201 grade in 1st Semester
FTFT 2011 61.1% 52.5% 65.0% 76.5% 2.72 2.48 2.91 2.52 (C+/B-) 3.16 3.05 (B)
• BUAD majors taking BUS 201 and ECO 200 during first semester via FPR
• Slight improvement in cumulative GPA for students in BUS 201 over ECO 200
• Students consistently performed better in BUS 201, regardless of TAC, etc.
• Some students may be at a disadvantage with ECO 200, particularly African American, Hispanic/Latino, FGEN, PELL, and TAC 1 & TAC 2 students
• Intervention for both at-risk and high-performing students
Intervention: Registration Holds • 36.8% of 2012 cohort had holds
just prior to Spring 2013 advising and registration, mostly financial – 44% of TAC 1 and 48% of TAC 2
students had holds – 45.7% of TAC 2A students had
holds – 76.2% of STEP/CAP students and
51.1% of nonwhite, non-STEP/CAP students had holds vs. nearly 30% of white students
– Average midterm GPA with holds: 2.53 (2.81 without holds)
• Collaboration with Enrollment Management, Fiscal Affairs, and Student Affairs – Financial Review Day – “Intrusive” Advising Strategies – Individual “Day Passes”
• Average cumulative GPA with holds: 2.74 (3.07 without holds)
TAC Returned Spring 2013
N Did Not Return
Total Not Returned
with Holds
N
1 90.0% 21 11 (52.4%) 211
2 93.0% 12 9 (75.0%) 171
2A 93.3% 3 2 (66.6%) 45
3 94.0% 14 8 (57.1%) 232
4 95.0% 16 8 (50.0%) 320
Total 93.3% 66 38 (57.6%) 979
Redefining Perceptions of “At-Risk”
Shifting from deficit-based assumptions…
Cognitive deficit assumptions mischaracterization
marginalization
…toward a success-based model!
Targeted initiatives based on student needs relevant
student support
Diversity Scholars Program
• Spring 2012 – Pilot group for FTFT SOC, TAC 1-2, < 2.0 GPA (N = 28)
• Fall 2012 – services for all FTFT ALANA students (N = 133) – Developmental advising services had positive impact on GPA for TAC 1,
limited impact for TAC 2 when given alone (“high-relational” groups)
– Peer mentoring: participants (2.79) vs. non-participants (2.47) • Hispanic/Latino student participants (3.03) vs. non-participants (2.13)
– Academic & student support interventions (e.g., taking major course in first semester, financial review)
– FTFT students of color had 3-4 times higher increase in GPA from previous year than white students, except TAC 4 African American students
– Reduction in GPA gap experienced across all ethnicities
• Fall 2013 – added targeted developmental advising outcomes and major-related opportunities/workshops (N = 138) – TAC 1-2 DSP participants (2.94) vs. ALANA non-participants (2.34)
– TAC 1-2 Hispanic/Latino students outperformed white students during first semester (2.72 vs. 2.59)
– TAC 3-4 African American students still underperform (2.89 vs. 3.10)
Recommendations for First-Year Success
• Use multiple approaches to assist students, based on individual needs and types of risk
• Consider non-academic barriers when advising students (finances, registration holds, campus integration, family issues, etc.)
• Encourage use of early alert system (APN) and sharing of information across departments
• Connect students to academic departments and major-related opportunities, particularly high performing students
Where do we go from here?
• Explore other non-cognitive measures and/or instruments and administer during the fall semester
• Revisit second-year persistence models developed during Project Compass
• Develop advising strategies for second-year students
• Communicate data findings with more faculty
• Increase access to high-impact practices, particularly for underrepresented students
ANY QUESTIONS?
Recommended