TSLCScholars:Amixedmethodsexamina6onofacomprehensivecollegetransi6onandsuccess

programforlow-incomestudents

November 12, 2016

AdriannaKezar,DarnellCole,Ma1Soldner,Ta4anaMelguizo,RobertReason,RosiePerez,AraceliEspinoza-Wade

PulliasCenterattheUniversityofSouthernCalifornia(USC-Pullias)and

AmericanIns4tutesforResearch(AIR)

Mixed-MethodsDesign

•  Presenta6on1:Thecollegetransi4onprogramandoverviewofthemixed-methodsresearchdesign

•  Presenta6on2:Anoverviewofsurveydesignand

forma4veanalysis•  Presenta6on3:Summa4veresearchdesign•  Presenta6on4:Qualita4veresearchdesign

TheProgramandOverviewofMixedMethodsResearch

Design

AdriannaKezar

TSLCProgram• 

•Sincetheearly1960s,theFounda4onprovidedscholarshipsforlow-incomegraduatesfromNebraskahighschools

•  •2007:TheFounda4onincreasedscholarshipsandcreatedlearningcommuni4esprogram(TSLC)at3UniversityofNebraskacampuses

•  •TSLCprogramincludesfiveyearsoffunding,smalllearningcommunitycourses,mentoring,residen4alcomponent,firstyearexperiencecourse,andprogrammingfromstaff(2yearprogram)

•  •2014:Decisiontoembarkonevalua4on

ProjectGoal

TounderstandhowthekeyelementsoftheThompsonScholarsLearningCommunity

(TSLC)facilitateengagement,thedevelopmentofacademicself-efficacy,

ma1ering/senseofbelonging,persistenceandotheroutcomes.

Framing•PriorresearchonLCsmostlydescrip4ve•Fewoutcomestudies-narrowstudiesofGPAmostly;Learning communi4esandrecentMDRCstudyoflimitedimpact

•One-4melookatoutcomes

•Focusinliteratureissenseofbelonging,self-efficacy,butnotstudied–weendedupfocusingonpsycho-socialoutcomes

•Programismuchmorethanalearningcommunity;amul4-prongedprogram–Howdoweseparateoutimpactofdifferentcomponentparts?

–Moststudiesshownoimpact

•Engagementvs.climatevs.contexts

ModelofEnvironmentalVariablesrelatedtoTSLC

StudentBackgroundCharacteris6cs

RaceGender

FamilyIncomeCollegeReadiness

ExpectedComple4onCampus

FinancialStressTimeUseMajor

Pre-test(outcome)

TSLCEngagement

PeerInterac4onFaculty-StudentAc4vi4es/

EventsFYS/SAC

MentoringGradeCheckStaff-StudentResiden4alOrienta4on

Psycho-SocialOutcomes

ResiliencySenseofBelonging

Ma1eringCollegeKnowledge

Career/MajorSelf-EfficacyAcademicSelf-EfficacySocialSelf-Efficacy

ModeladaptedfromTinto(1997)&Hurtadoetal.(2007)

AcademicSSHealthOfficesCampusSS

CampusServiceUse

ResearchQues4ons1.Howdothekeyprogramcomponentssharedacrossthethreecampuses

facilitateengagement,thedevelopmentofacademicself-efficacy,ma1ering/senseofbelonging,andtheotheroutcomes?

2.Howdodifferencesintheprogramsacrossthethreecampusesfacilitateengagement,shapethedevelopmentofacademicself-efficacy,ma1ering/senseofbelonging,andtheotheroutcomes?

3.Howdodifferencesbyloca4on–asrural,urban,commuterandmoretradi4onalsuburbancampus–shapestudents’programexperienceandoutcomes?

4.HowdoestheTLSCexperienceaffectpar4cularsubgroupsincludingminoritystudentsandstudentswithdifferentlevelsofacademicprepara4on?

5.HowdoTSLCscholarsexperiencethetransi4onoutoftheprogramandhowmightthisexperienceaffecttheircon4nuededuca4onalsuccess(reten4onandgradua4on),academicselfefficacy,andma1ering/senseofbelonging?

ResearchQues4ons6.Howdofacultyandclassroomexperiencesshapestudents’program

engagementandoutcomes?7.HowdodifferentstudentsengagewiththeTSLCprogramandhowdoesthis

engagementappeartoshapetheirexperienceandoutcomes?8.Aretheredifferencesinthepsycho-socialandtradi/onalacademicoutcomes

betweenthestudentswhoexperiencedtheTSLCcomparedtothosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andnon-recipients/control)?

9.Aretheredifferencesinthepsycho-socialandtradi/onalacademicoutcomesbetweenthestudentswhoexperiencedtheTSLCcomparedtothosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andnon-recipients/control)bytypeofTSLC(LC-Kearney,LC-Omaha,andLC-Lincoln)?

10.Aretheredifferencesinthepsycho-socialandtradi/onalacademicoutcomesbetweenstudentswhoreceivedfinancialsupport(i.e.,BuffetScholarship(BS)orCOS)andthosewhodidnot?

ResearchQues4ons11.a)Aretheredifferencesinthepsycho-sociallevels(i.e.,measuresofacademicself-efficacyormeasuresofsenseofbelonging)betweenstudentswhopar4cipatedintheprogram(i.e.,TSLC)andthosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andcontrol)?11.b)Isthereanassocia4onbetweenengagementlevels(mediator)betweenstudentswhopar4cipatedintheprogram(i.e.,TSLC)andthosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andcontrol)?11.c)Aretheredifferencesinthepsycho-socialvariablesbetweenstudentswhopar4cipatedintheprogram(i.e.,TSLC)andthosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andcontrol)anercontrollingforengagementlevels?12.Aretheredifferencesinthepsycho-socialandtradi4onalacademicoutcomesbetweenthestudentswhoexperiencedtheTSLCcomparedtothosewhodidnot(i.e.,CollegeOpportunityScholarship(COS)andnon-recipients/control)byselectedstudentcharacteris4cs(i.e.,minoritystudents)?

ParallelMixedMethodswithEmbeddedSequen4alExploratoryDesignCohort1:2015-2020andCohort2:2016-2020

Casestudy Digitaldiariesandinterviews Socialmedia Focusgroups

2015Focusgroups

• Cohort1Baseline(Fall2015)

• Cohort1Follow-up

(Spring2016)

2016Focusgoups

*Cohort2Baseline(Fall2016)

• Cohort1&2Follow-up

(Spring2017)

2017Focusgroups

• Cohort1&2Follow-up

(Spring2018)

2018Focusgroups

• Cohort1&2Follow-up

(Spring2019)

(Sequen6alExploratoryDesign)

PsychometricPilot

(Summer)2015

Cogni4veInterviews(Spring)2016

Opera4onalPilot

(Spring)2016

*Cohort1&2Follow-up

(Spring2020)

MixedMethodsImplementa4on•  Retreatsthroughouttheyeartodiscuss

quan4ta4veandqualita4veconcepts,designanddatacollec4on

•  Bi-monthlyteammee4ngs–mixedmethodsdiscussed

•  Casestudydatausedtodesignbaselineandfollowupsurveys

–Areasofprogramtoinclude,differencesacrosscampusesandwordingofprogramelementsbyvariouscampuses

*

DesigninginRealTime•Mixedmethodsdesignwherequalita4veinformssurvey,butsurveyneedstostartASAPtoleveragecontrolgroup•TSLCprogramchangeseachyear•Ourunderstandingofprogramchangesover4me•Changeinprogramunderstandingrelatestonewconstructs,theoriesandframing

2015 2016 2017 2018 2019 2020 First 6-8 months

descriptive case study

Ethnographic case study

begins

Case study continues

Spring: Case study ends

Social media analysis

Social media analysis

Social media analysis

Social media analysis

Digital diaries/interviews

begin

Digital diaries/interviews continue

Digital diaries/interviews continue

Digital diaries/interviews continue

Digital diaries/interviews continue

STS survey STS survey STS survey STS survey STS survey STS survey Fall focus

groups Spring/Fall

focus groups Spring/Fall

focus groups Spring/Fall

focus groups Spring/Fall

focus groups

MixedMethodsImplementa4on•Focusgroupsaboutsummerexperienceinformedfollowupsurvey•Casestudyiden4fieduniqueprogramma4celementsbycampuswhichwasusedinsurveydesign•Surveyfindingsusedtodesigndigitaldiaryques4ons•Constructsrevisitedandreconsideredbasedondatainterac4on,e.g.,resiliency

TeamBuilding&Communica4on•On-boardingasteamassembled•Bi-monthlyteammee4ngs•Sub-teams–weeklymee4ngs•Basecamp•Seminars/retreats•Languagedifferences

–methods,theories,paradigms,researchexperience

TeamTraining•Mixedmethods•Constructsandtheories•Differentmembers,needsandengagement:

–Graduatestudents–Postdocs

–Na4onallyrecognizednon-profitwithkeysurveyexper4se

Forma4veResearch

DarnellCole(USC)MarkMasterton&Ma1Soldner(AIR)

DevelopingtheDataset

ScholarsSurvey

Key Design Features ●  Self-administered via the web ●  Optimized for mobile devices ●  Incentives-based ●  Reminders via email and text message

Key Methodological Features ●  Dynamic, allowing targeting of student subgroups based on

pre-loaded administrative data ●  Adaptive non-response follow-up ●  Imputation used to address item-level missingness ●  Weighted to adjust for non-response bias

ScholarsSurvey

Year1SurveyOverviewComponent Purpose Timing Sample and Respondents

Psychometric Pilot

Validate key constructs Evaluate operational protocols

July, 2015

972 prior scholars 350 respondents, RR = 36%

Baseline Survey

Capture baseline data about background, high school experiences and college expectations

August, 2015

1335 scholars in 2015-16 cohort 1074 respondents, RR = 80%

2nd Psychometric Pilot

Validate student engagement measures

January, 2016

1652 prior scholars 332 respondents, RR = 21%

Cognitive Interviews

Refine student engagement measures

March, 2016

11 prior scholars

Follow-up Survey

Capture outcome and engagement data at end of scholars’ first year

May, 2016

1297 scholars in 2015-16 cohort 951 respondents, RR ≈ 73%

ExemplarConstructsBaseline Follow-up

§  High school sense of belonging

§  High school interactions with peers

§  High school interactions with faculty

§  High school time use

§  Expectations about social self-efficacy

§  Expectations about academic self-efficacy

§  Expectations about mattering

§  Expectations about belonging

§  Expectations about likelihood of graduation

§  “Actual” academic self-efficacy

§  “Actual” social self-efficacy

§  Experience of mattering

§  Experience of belonging

§  Engagement with peers and faculty

§  Engagement with program features

§  Financial stress

§  Revised graduation expectations

Takeaway#1:Don’tSkipPilotTes4ng

•  Psychometricproper4esofadaptedmeasuresareonenintheball-parkofpublishedresults,butusingConfirmatoryFactorAnalysistoverifycanaddconfidenceinyourchoices

•  WeusedgradedresponseIRTmodelsasanaddi4onalcheckfollowingCFA,par4cularlywhenwewereconsideringscalereduc4on

24

Takeaway#2:BeWaryofTiming

•  Ourbaselineadministra4onuncoveredatleastonephenomenonthatwasnotdetectedinthepilot,givenitspopula4onandthe4mingofadministra4on:higherthanan4cipatedbaselinescoresonself-efficacymeasures

•  Whatisthetemporalnatureofyourphenomena?Iftheyare4me-variant,howbesttomeasure?

25

Takeaway#3:Don’tSkipCogni4veInterviewing

•  Althoughwethoughtwehadasenseofhowtodesignengagementitemsthatwouldbe“respondentfriendly,”wehadnoideahowhardtheywouldbetoanswerinanyformatun4lwehadconductedcogni4veinterviews.

•  Interviewsbringevidencetodebatesbetweenthosewhoare

arguing“weneeddetailedinforma4on”and“respondentswon’tbeabletonavigatethecomplexity,”sotherightbalancecanbestruck.

26

27

Takeaway#4:AnalyzeDataPromptlytoEvaluateConstructs

•  Givenhighbaselinemeasures,wewereworriedaboutwhetherwecoulddetectasimpletreatmenteffectatfollow-up,controllingforbaselinelevel.Wecould,butdevelopednewitemsforthenextcohortthatwehopewillincreaseourabilitytodiscernbetweenmorenuancedlevelsoftheunderlyingconstruct.

28

Takeaway#5a:Don’tBeAfraidtoExperimentwithMethods

•  Inalongitudinalsurvey,one(ofmany)tricksisensuringhighresponseratesateachstage—withoutbreakingthebank.Experimentscanhelpyouiden4fy“whatworks”withyourpopula4onstokeepresponserateshighandminimizedifferen4alnon-responsebygroups.

•  Inthefollow-upsurvey,AIRconductedanRCTtodeterminewhetherwecouldsafelytailorourapproachtofollow-up,basedonstudents’es4matedpropensitytorespond.

29

Takeaway#5b:Don’tBeAfraidtoExperimentwithMethods

•  Inthefollow-upsurvey,AIRconductedanRCTtodeterminewhetherwecouldtailorfollow-upefforts,basedonstudents’es4matedpropensitytorespond.

•  Ourgoalwastoshineffortfrommostlikelyrespondentstotheleastlikely,makingitpossibletodophonereminders—orphoneinterviews—forthelowestpropensitygroup.

•  Earlyevidence:a5to8percentagepointresponseratebumpinourlowestpropensitygroups.

30

Non-responseFollow-upExperiment

Contact Date Control

Condition

Experimental Condition High

Propensity Medium

Propensity Low

Propensity Very Low

Propensity Invitation e-mail 8/29 X X X X X Reminder e-mail 1 9/1 X X X X X Reminder text 1 9/6 X X X Reminder e-mail 2 9/8 X X X X X Phone reminder (R) / interview (I)

9/12-14 R R R I

Reminder e-mail 3 9/15 X X X X Reminder text 2 9/19 X X Reminder e-mail 4 9/22 X X X

31

ResponseRatebyResponsePropensityGroupandExperimentalCondi4on

Response Propensity Group

Response Rate

Control Experimental Difference Very Low 65.7% 70.3% 4.6% Low 71.9% 80.2% 8.3% Medium 87.6% 82.7% -4.9% High 96.1% 84.3% -11.8%

Summa4veResearch

Ta4anaMelguizoFrancisco(Paco)Martorell

W.EdwardChiElizabethParkAdriannaKezar

OverallResearchGoalSumma4ve

RigorouslyexplorewhethertheTSLCgenerateddifferencesinthesocialandpsychologicaloutcomesrela4vetothe

CollegeOpportunityScholarship(COS)andnon-recipients/control.

USC/AIR’SThompsonScholarLearningCommuni4es(TSLC)SUB-SAMPLE

MustFund

Control

CollegeOpportunityScholarship

(COS)

Buffe1Scholar(BS)

Applicants

EnrolledNUCampus

(3)

EnrolledNECommunityCollege(7)

EnrolledOut-of-State

EnrolledNEStateCollegeCampus(3)

DonotEnroll

Eachgroupcouldfollowoneofthese6paths

USCcanONLY“follow”1ofthese6groupsandcollectsurveydata

USC/AIRSTS

LongitudinalSurvey

EnrolledPrivate

CollegeNE

MethodologicalChallenges•  Twopoten4alsourcesofbias:

–  Studentself-selec4on–  Changesinoriginalgroupingsresul4ngfromtherandomiza4on

–  ShinintoTSLC:studentswhotargetedastateorcommunitycollegeandreceivedtheBuffe1grantcouldenrollatanNUandbecomepartoftheTSLC

–  ShinoutofControl:studentswhoini4allytargetanNUanddidnotreceivetheBuffe1grantoragoodfinancialpackagemightenrollatadifferentcollegeornotenrollincollegeatall

•  Thisisproblema4casthesestudentsmightdifferintermsofobservableandun-observablefactorsfromthestudentsoriginallyrandomized–  Needtoassesstheproblembylookingatdifferencesinbackgroundand

demographiccharacteris4csinresul4nglongitudinalsample

MethodologicalChallenges•  Surveydifferen4alResponseBias:

•  Changesinoriginalgroupingsresul4ngfromsurveydifferen4alresponserate

–  Cohort2015:»  Baselinesurveyresponserateof80%,theresponseratevariedbetweentreatmentsandcontrol(e.g.,TSLC=87%,COS=85%,andControl=66%)

»  First-followupresponserateof73.3%(e.g.,BUFFETT=82.6%,COS=82.39%,andControl=60.87%)

•  Needtoiden4fythemagnitudeoftheproblembyes4ma4ngthepropor4onofstudentswhoremainedinthelongitudinalsamplefromtheoriginalassignment

–  Cohort2015:TSLC=66%,COS=58%,andControl=33%

Cohort2015DatasetsandSample

•  Datasets–  2015STBFScholarshipApplica4onData–  SurveyofThompsonScholars(STS)

•  Cohort2015Baseline•  FirstFollow-upSurveyData

•  Sample–  Studentsfromthe2015cohortwhotargetedoneNUcollegeintheirapplica4onandwerepartofthestra4fiedrandomassignmentthatwasusedtoawardscholarships

AnalysestoAssesstheSelec4onProblems

• Twosidedt-testsofmeandifferences• Mul4plemeancomparisontests,BonferroniAdjusted

• Chi-squaretests• Regressionanalyses•  Latentmeancomparison*

ProposedCorrec4onsforSelec4onProblem

ControlforObservableDifferencesAcrossGroups(RegressionandPropensityScoreWeigh4ng:HeckmanandNavarro,2004;Hirano,Imbens,andRidder,2003)

•  Richinforma4ononapplicantsishelpful•  But,s4llrequiresstrong‘noselec4ononobservables’

assump4onTreatmentEffectBounds

•  Lee“trimming”procedureforcon4nuousoutcomes•  Challengeconver4ngsocialandpsychologicaloutcomes

Likert-scaleintocon4nuousvariables

Takeaway #1 The power and challenges of using simultaneously

qualitative and psychometric methods to measure social and psychological constructs

•  Thoroughreviewoftheoriesandtheliterature•  Leveragecasestudytogainadeepcontextualunderstanding

oftheprogram•  Relateprogramma4celementswithsocialandpsychological

outcomesofinterest•  Partnerandcollaboratewithsurveyexperts•  CommunicateregularlywithProgramDirectorsand

Founda4onOfficials

Takeaway #2 Partner with survey firm to maximize survey response

rates and minimize differential response rates •  SurveyResponserates

-  Needtogetappropriate(80%)andbalanced(similarresponseratesforTsandCgroups)surveyresponserates

•  Psychometricreports

-  Needtocarefullyandsystema4callytestforvalidityandreliabilityofconstructsnotonlyinthepilotbutinthebaselineandfollow-ups

Takeaway #3 Leverage disciplinary methodological expertise

•  Economics-  Studentself-selec4on

•  PsychometricsandEduca4onalPsychology-  Itemresponsetheory(IRT)-  LatentVariableAnalyses-  Structuralequa4onmodels(SEM)

Take Away #4 Summative Research deeply benefited from Case

Study •  Longitudinalcasestudyprovidesdeepcontextualunderstanding

oftheprogramasitwasconceivedandimplementedinthedifferentsites

•  Casestudyinformedlongitudinalsurvey•  Focusgroupsandcogni4veinterviewsarekeyto:

–Iden4fykeyprogramma4celementsasexperiencedbythestudents–Testassump4onsofmeaningofsurveyscalesandindexesforstudents

Take Away #5 Case Study and Formative Benefitted from

Summative Research •  Cohortcategoriza6on

–Earlyanalysesofsurveydatatopresentdescrip4ve portraitsforstudents

•  Impactresults

–Preliminaryimpactfindingscanbeexploredfurthertogetamorenuancedunderstandingofthemechanismsthroughwhichtheprogramisworking(e.g.,promptsfordigital diarypar4cipantsrelatedtosocialself-efficacy)

Qualita4veResearch

RobertReason,RonaldHalle1,RosemaryPerez,AraceliEspinoza-Wade

OverviewofDataCollec4onAc4vi4es

•  Observa4onsandInterviews•  DigitalDiaries•  SocialMedia•  DocumentAnalysis

PurposeswithinMixedMethods

• Exploratory

• Explanatory

LessonsLearnedabouttheResearchDesign

•  DigitalDiaries– Breadthv.Depth– Language– Management

•  SocialMedia– Analyzingsocialmediacontent–  Integra4onintoMixedMethods

LessonsLearnedabouttheResearchDesign

•  LongitudinalQualita4veResearch– Engagement– Reten4on– ResearcherTransi4ons

•  ResearchasInterven4on

Ques4onsLenUnanswered

•  Howdowebalanceethnographicandcasestudymethods?

•  Howdowedevelopacasestudyofprogramsthatcon4nuetoevolve?

•  Howdowemanageandintegratesuchvariedformsofrichdatainqualita4veandmixedmethodanalysis?

51

Q&A

Wewelcomeyourfeedbackandideas!

PulliasCenterforHigherEduca4onRossierSchoolofEduca4on

UniversityofSouthernCalifornia

3470TrousdaleParkwayWaitePhillipsHall701LosAngeles,CA90089

Website:pullias.usc.edu