Eﬀec%veness of PATHWAYSstorage.googleapis.com/cmp-wordpress-public.appspot.com/... · 2017-12-05 · MATH PATHWAYS TECHNICAL REPORT Assessing the Eﬀec%veness ... , and Lumina

CARNEGIEQUANTWAYSUCCESS 1

CARN

EGIEM

ATHPAT

HWAY

S

TECH

NICAL

REP

ORT

AssessingtheEffec%venessofQuantwayAMul&levelModelwithPropensityScoreMatching

October2016

HiroyukiYamada,AngelBohannon,&AliciaGrunow

CarnegieFounda-onfortheAdvancementofTeaching

Stanford,CA

ThisprogramofworkissupportedbyCarnegieCorpora7onofNewYork,TheBill&MelindaGatesFounda1on,TheWilliamandFloraHewle:Founda1on,TheKresgeFounda/on,andLuminaFounda/onincoopera/onwiththeCarnegieFounda-onfortheAdvancementofTeaching.


ABSTRACTQuantwayisaCarnegieMathPathwaysini2a2vewhichredesignsthecontent,pedagogy,andstructureoftradi-onaldevelopmentalmathcoursestosimultaneouslytackletradi-onalbarriersofstudentsuccessandsupportabroaderrangeofdevelopmentalstudentsinachievingtheirmathpoten,al.Specifically,Quantwayisaquan,ta,vereasoningsequencethatiscomprisedofasingletermdevelopmentalmathcoursecalledQuantway1andacollege-levelmathcoursecalledQuantway2.Thisstudyassessedtheeffec4venessofthedevelopmentalmathcourse,Quantway1,duringitsfirst6semestersofimplementa:on.Weusedahierarchicallinearmodelingtechniquetoconductpropensityscorematchingacross37studentcharacteris(csinordertocomparethecourseperformanceofQuantway1studentswithmatchedcomparisonstudentsintradi0onaldevelopmentalmathcourses.Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccessthanmatchedcomparisonstudentsinfulfillingdevelopmentalmathcourserequirements.Addi8onally,Quantway1effectswereposi%veacrossallsexandrace/ethnicitysubgroupsaswellasinnearlyallclassroomsandcolleges.ThisstudyprovidedrobustevidencethatQuantway1increasesstudentsuccessinfulfillingdevelopmentalmathrequirementsandadvancesequityinstudentoutcomes.Direc&onsforfutureworkaresuggested.


Traditionaldevelopmentalorremedialmathsequencesserveasahugeimpedimentforcommunitycollegestudents,oftenpreventingthemfromobtainingtechnicalcredentialsandassociatedegrees,aswellasblockingtheirtransfertofour-yearinstitutions.Nearly60%ofcommunitycollegestudentsnationwidearerequiredtotakeatleastonedevelopmentalmathematicscourse,and80%ofthesestudentsdonotcompleteacollegemathcoursewithinthreeyears(Bailey,Jeong,&Cho,2010).Studentsspendlongperiodsoftimerepeatingcoursesandaccruingstudentloandebt,ultimatelyleavingcollegewithoutadegree.ThiscrisisincompletionhasnegativeramificationsforcommunitycollegestudentearningsandforAmericanworkforcedevelopmentoverall.Studentswhodonotcompletehigherlevelsofeducationhavesignificantlylowerincomes.In2012,forallpeopleaged25orolder,highschoolgraduatesearnedanannualincomeof$33,904,thosewithassociate’sdegreesearned$40,820,andthosewithbachelor’sdegreesearned$55,432(Johnstone,2013).Lowerlevelsofeducationalattainmentalsohaveadetrimentaleffectontheoveralleconomyastechnologicaladvancesandglobalcompetitionhavecreatedapressforanincreasinglyskilledworkforce.TheGeorgetownCenteronEducationandtheWorkforceestimatesthat65%ofallAmericanjobswillrequireapostsecondaryeducationbeyondhighschoolby2020,buttheUnitedStateswillfallshortofmeetingtheserequirementsby5millionworkersatthecurrentrateofproduction(Carnevale,Smith,&Strohl2013).Communitycollegesplayacriticalroleinworkforcedevelopment,servinghalfofthe6.5millionundergraduatesintheUnitedStates.Additionally,traditionallyunderservedstudentsaredisproportionatelylikelytoencounterdevelopmentalmathasastumblingblockontheroadtocollegecompletion.Thecommunitycollegestudentpopulationismoreraciallydiverse,older,andlowerincomethan4-yearuniversitystudents(Bueschel,2004).Minoritystudentsareplacedinmoredevelopmentalmathcoursesandlesslikelytocompletethesecoursestoachievecollege-levelmathcreditthanwhitestudents(Baileyetal.,2010;Chen,2016).Improvingthesuccessratesofstudentsindevelopmentalmathsequencesisakeyleverforadvancinganequityagenda.Severalaspectsoftraditionaldevelopmentalmathsequenceshavebeenproposedascontributorstonegativestudentoutcomes.Traditionally,studentsmusttakelongmulti-coursesequencesofincreasinglevelsofdifficultytofulfilldevelopmentalmathrequirements.Asequenceofmultipledevelopmentalcourses,suchasbasicarithmetic,pre-algebra,elementaryalgebra,andthenintermediatealgebra,leadsintoacollege-leveltransferableclasssuchaspre-calculus.Thisstructuredrasticallyhindersstudentcompletion.Evenwhenstudentscompleteonecourseinasequence,manyfailtoenrollinsubsequentcourses(Cullinane&Treisman,2010;Baileyetal.,2010).Additionally,theinstructioninmanymathclassroomsdoesnotincorporateresearch-basedcurriculumdesignandpedagogicpracticesthatfosterdeeperstudentlearningandengagement(Mesa,2011).Traditionalmathcoursesemphasizetransmissionofcontentoveramoreparticipatoryapproach(Edwards,Sandoval,&McNamara,2015),factualandproceduralknowledgeoverconceptualknowledge(Mesa,2011),anddonotdemonstratetherelevancyofmathematicalconcepts(Carnevale&Desrochers,2003).Furthermore,traditional


developmentalmathcoursesdonotaddresseitherlanguageandliteracyornon-cognitivebarriers(e.g.,mathanxietyandstereotypethreat)thatimpedemanystudents’abilitytolearnmath(Blackwell,Trzesniewski,&Dweck,2007;Haynes,Perry,Stupnisky,&Daniels,2009;Gomez,Rodela,Lozano,&Mancevice,2013).Morerecently,therelevancyofthealgebra-heavycontentoftraditionalmathcurriculumhasalsobeencalledtoquestion.AstudyontheSurveyofWorkplaceSkills,Technology,andManagementPracticesfoundthatonly19%ofemployeesuseanyalgebraintheirwork(Handel,2007).Tospurprogressonthisproblem,CarnegieFoundationfortheAdvancementofTeachingconvenedanetworkedimprovementcommunity(NIC)-anationalcommunityofcommunitycollegeadministratorsandfaculty,andeducationalresearchers(Bryk,Gomez,Grunow,&LeMahieu,2015).Throughanimprovementscienceapproach,theNICredesignedthecontent,pedagogy,andstructureoftraditionalmathsequencestoincreasethenumberofstudentscompletingtheirmathrequirements.Theresultofthisworkwastwoacceleratedalternativestotraditionaldevelopmentalmathsequencesfornon-STEMstudents:StatwayandQuantway.Statwayisanacceleratedyear-longintroductorycollege-levelstatisticscoursethatintegratesdevelopmentalmathcontent.ApreviousstudydemonstratedStatway’sefficacyandimpactonstudentsuccessovertraditionaldevelopmentalmathprograms(Yamada&Bryk,2016).Quantwayprovidesanotheralternativetothetraditionalmathsequencefocusingonquantitativereasoning.AsshowninFigure1,Quantway1isaone-termquantitativereasoningcourseforstudentswhoplacetwolevelsbelowcollege-level,thusenablingthemtocompletetheirdevelopmentalmathrequirementsinasingleterm.StudentswhosuccessfullycompleteQuantway1arepreparedforcollegelevelmathandeligibletoenrollinQuantway21oranothercollege-levelquantitativereasoningcourse.SincetheQuantway1’slaunchin2012,Quantway1’simplementationhasgrownmorerapidlythanStatway,withQuantway1enrollmentgrowingfrom418to1936studentsoverthefirstfouryearsofimplementation(Huang,Hoang,Yesilyurt,&Thorn,2016).BecauseQuantway1fulfillsdevelopmentalmathrequirementsinoneterm,communitycollegescaneasilyintegrateitintotheircurrentdevelopmentalmathofferings.Quantway1hasbeensuccessfulinpromotingstudentsuccessindevelopmentalmathcourses,essentiallydoublingthesuccessratesoftraditionaldevelopmentalmathcoursesinhalfthetime.Notably,Quantway1maintainedthesehighsuccessratesevenasstudentenrollmentmorethanquadrupled.Inthemostrecent2014-2015academicyear,57%ofthestudentsenrolledinQuantway1successfullycompletedthecourseinoneterm.Incomparison,only21%ofabaselinegroupofdevelopmentalmathstudentscompletedthetraditionaldevelopmentalmathcourseinoneyear2(Huangetal.,2016).1Fiveins(tu(onshaveimplementedQuantway2,servingatotalof429studentsover3yearsofimplementa/on(Huangetal.,2016).Forthepurposesofthisstudy,weevaluatedtheefficacyofQuantway1.

2Tocomputethisbaselinesuccessrate,weworkedwithins(tu(onalresearchersfromsixofthefirstQuantwaycolleges.Analysesrevealedthatonly20.6percentofstudentswereabletosuccessfullycompletetheir


Despitethesepromisingresults,itisunclearwhetherQuantway1’sdemonstratedimpactisa"ributabletotheprogramitself,orisduetodifferencesinthestudentsthatselectorareplacedintoQuantway1.Buildinguponourdescrip3veresults,thisstudypursuesamorerigorouscausalanalysisofQuantway1throughapropensityscoretechnique,separa7ngtheprogram’seffectfrompoten0alselec0onbias.Theseresultswouldprovideastrongereviden!arybaseonwhichtobasedecisionsaboutadop!ngandscalingQuantway1.

DESIGNOFQUANTWAYQuantway1isdesignedaroundthepremisethatallstudentsarecapableoflearningambitiousmathematicsandsucceedingindevelopmentalmathcourseswiththerightsupports.AssummarizedinFigure2,Quantway1sharesaworkingtheoryofimprovementwithitssisterprogramStatway,aimingtoincreasestudentsuccessthroughworkingonsixkeydrivers:(1)accelerationofdevelopmentalmathrequirements,(2)implementationofaresearch-basedinstructionalsystem,(3)socioemotionalsupports,(4)languageandliteracysupports,(5)facultydevelopment,and(6)participationinaNIC(formoreinformationonthetheoryofimprovement,seeYamada&Bryk,2016).ThewayQuantway1addressesthesedriversdiffersfromthatofStatwayinordertofilladifferentnicheincommunitycollegemathdepartmentsandmeettheneedsofspecificstudents.Below,wewillelaborateonthefeaturesthatdistinguishQuantway1fromotheraccelerateddevelopmentalmathprograms.First,Quantway1acceleratesstudents’abilitytocompletedevelopmentalmath,gettingthemtocollegelevelmathmorequickly.Quantway1focusesonquantitativeliteracy,whichisdescribedas“theabilitytoadequatelyuseelementarymathematicaltoolstointerpretandmanipulatequantitativedataandideasthatariseinanindividual’sprivate,civic,andworklife”(Gillman,2004,p.5).Thesequantitativeliteracyconceptsarecodifiedinasetofrigorouslearningoutcomesthatwerecollaborativelyestablishedandvettedbyacommitteethatincludedrepresentativesfromseveralmathematicalprofessionalsocieties.3BecauseQuantway1’slearningoutcomesprovidestudentswithastrongfoundationinnumericalandquantitativereasoningconcepts,ithasservedaspreparatoryquantitativereasoningcourseformanynon-STEMpathwaysandmajorsandastheculminatingmathcoursefortechnicalcertificateprograms.SomecollegesofferQuantwayasapathwaythroughcollegelevelmathbycombiningQuantway1withQuantway2oranothercollegelevelquantitativereasoningcourse.Quantway’sflexibledesigncanbeeasilyintegratedintocurrentinstitutionalstructuresandmeetsavarietyofcommunitycollegeneeds.

developmentalmathsequencewithinafullyear.Addi8onally,28.5percentachievedthisgoala&ertwoyears,31.6percenta)erthreeyears,and33.3percenta)erfouryears.Formoreinforma7on,seeHuangetal.,2016.3Thesemathema)calsocie)esincludetheNa)onalNumeracyNetwork,AmericanMathema)csAssocia)onofTwo-YearColleges,andtheMathema,calAssocia,onofAmerica.


Second,Quantway1’sinstructionalsystemisdesignedtogroundunfamiliarmathconceptsinfamiliarsituationsthroughcontextualization.Quantway1’slessonsuseauthentic,relevantcontextsandrealdatatoincreasestudentmotivationtolearn.Quantway1isorganizedaroundthreeintentionalthemes(citizenship,healthcare,andfinancialliteracy)thatreflecteverydayconceptsandarecriticallyimportantinengaginginsociety.Byillustratingtherealworldapplicationsofmathconcepts,Quantway1cancontributepreviouslyunsuccessfulstudentscanhavemeaningfulandpositiveinteractionswithquantitativereasoningcontent.LikeStatway,theQuantway1instructionalsystemisdesignedtofosterrobustandsustainedmathematicallearning,emphasizingtheteachingofconceptstoimprovebothproceduralandconceptualunderstanding(Hiebert&Grouws,2007).TheQuantway1instructionalmodelisorganizedaroundthreeresearch-basedlearningopportunities–productivestruggle,explicitconnections,anddeliberatepractice.Inproductivestruggle,facultyengagestudentsinsubstantivemathematicaltasksthatencouragestudentstostrugglewithkeymathematicalconceptsandsolveproblemsthatarechallengingbutstillwithinreach(Hiebert&Grouws,2007).Byproductivelystruggling,studentscanmakemeaningofthemathematicalcontentforthemselvesanddevelopstrategiesforengagingwiththecontent.Explicitconnectionsrefertoinstructionthatcreatesopportunitiesforstudentstomakeconnectionsbetweenmathematicalproceduresandunderlyingconceptualknowledge.Deliberatepracticeaimstoimprovestudentperformancethroughaseriesofhighlystructured,increasinglysophisticated,andchallengingtasksthatdeepenfacilitywithkeyconcepts(Edwards&Beattie,2015).Theselearningopportunitiesaresupportedbyinstructionalpracticesthatfacilitatestudentdiscussionandsupportcollaborativelearningaroundrichmathematicalproblems(Edwards&Beattie,2015;Edwardsetal.,2015).Third,Quantway1integratestwotypesofresearch-basedstudentsupportsdesignedtomeettheneedsofdiversestudentlearners–productivepersistence,andlanguageandliteracysupports.Onesetofsupportsisdesignedtopromotestudents’abilitytoproductivelypersistthroughrigorousmathcoursework.Thesocioemotionalintervention,whichwecallProductivePersistence,consistsofacollectionofstudentactivitiesandfacultyactionsthataddressthehigh-leveragenon-cognitivefactorsthatpromotestudenttenacityandeffectivelearningstrategies(Edwards&Beattie,2016).NICmembersworkedtogetherwithsocialpsychologiststoiterativelydevelopthispackageofproductivepersistenceroutines,interventions,andpracticesthatworktopromotegrowthmindset,reducemathanxiety,andincreasestudents’senseofbelonging.Asecondsetofinterventionsisdesignedtosupportstudentsinsuccessfullygrapplingwiththecomplexlanguageandliteracydemandsofmathematics,withitsdifferentformsofrepresentationandelaborategrammaticalforms.Quantway1lessonsembedlanguageandliteracytoolstosupportthecomprehensionandorganizationofinformationinquantitativesituations.Theselessonsarewrittentoavoidliteracybarriersthatdevelopmentalmathstudentscommonlyface(Gomez,Rodela,Lozano,&Mancevice,2013;Gomezetal.,2015).Fourth,becausetheQuantway1curriculumandpedagogysignificantlydifferfromtraditionalmethodsofteaching,Quantway1facultyareinvitedtoparticipateinacomprehensive


professionaldevelopmentprogram(Edwardsetal.,2015).ThisFacultySupportProgrampreparesfacultytoteachQuantway1andsupportsthemintheirfirstyearofteaching,andprovidesongoingopportunitiesforinstructionalimprovementandprofessionallearning.Throughonlineresources,facultymentorship,andongoingworkshops,thisprogrampreparesfacultytoeffectivelyimplementPathways’collaborativeinstructionalapproach,learningopportunities,andproductivepersistenceandlanguageandliteracysupports.Fifth,Quantway1facultyandadministratorsparticipateinanetworkimprovementcommunity(NIC)thatprovidesthemwithacollaborativelearningcommunitytosupporttheminteachingandimplementingQuantway1.TheNICsocialstructuresupportscommunitycollegesfacultyandadministratorsincollectivelygeneratinganddisseminatingpracticallearningaboutwhatworks,forwhom,andunderwhatconditionstoreliablydeliverefficacyatscale(Bryketal.,2015).BothQuantway1studentsandfacultyreportthatthecombina(onofthesedesignelementscreatesameaningfullydifferentmathexperiencefromtradi5onaldevelopmentalmathcourses.Inaddi'on,thesuccessratesofQuantway1studentsinthefirstfouryearsofimplementa0onaresignificantlyhigherthanins/tu/onalbaselinesateachofthepar/cipa/ngcolleges(Huangetal.,2016).However,evalua3ngtheeffec3venessoftheQuantway1programrequirescomparingQuantway1studentsuccesstoareasonablecounterfactualthatrepresentshowsimilarstudentswouldhaveperformediftheyhadnottakenQuantway1.Inthisstudy,weusedapropensityscorematchingtechnique(Rosenbaum&Rubin,1983)tocompareQuantway1studentswithsimilarstudentsintradi+onaldevelopmentalmathprogramsinthesameins$tu$on.Weconductedthepropensityscoreanalysiswithinahierarchicallinearmodelingframework(Raudenbush&Bryk,2002)toaccountforthenestedstructureofthedatawithstudentswithinins(tu(onsinthenetwork.Weconductedafollow-upsensi(vityanalysistoexaminewhethertheeffectscouldbeexplainedbyotherunmeasureddifferencesbetweenthetwogroupsofstudents.

Wealsolookedatvariationinperformanceacrossstudents,classroomsandinstitutions.Incontrasttotypicalevaluationsthatreportonlytheaverageimpactofanintervention,thisstudyassessedwhetherQuantway1iseffectiveacrosstherangeofclassroomsandinstitutionsintheNIC.Theseinvestigationsintovariationsupporttheprogram’sabilitytoscalewithefficacy(Bryketal.,2015)andcaninformwhereimprovementeffortsshouldbetargetedinordertofurtherincreasesuccessrates.WealsoexaminedpossibledifferentialeffectsofQuantway1acrosssexandrace/ethnicitysubgroupstodeterminethepotentialoftheprogramtopromoteanequityagendabyimprovingoutcomesacrossallrace/ethnicityandsexsubgroups.


METHODS

PARTICIPANTSQuantway1wasfirstimplementedduringthespringof2012.4Theinitialcohortofstudentsspanned8communitycollegesacrossthreestates(Georgia,NewYork,andOhio).Throughoutthe2014-15academicyear,Quantway1served5561studentsfromatotalof14colleges(seeAppendix)acrosseightstates(Georgia,Minnesota,NewJersey,NewYork,Ohio,Washington,WestVirginia,andWisconsin;Huangetal.,2016;Sowers&Yamada,2015).

DATAANDSTUDYDESIGNInstitutionalresearchersfromparticipatingcollegesprovidedbackgrounddataonstudentcharacteristics,courseenrollmentandperformance.Theanalyticsampleofthecurrentstudyconsistedof4,243Quantway1studentsfrom10collegeswhoenrolledinaQuantway1coursebetweenthespringof2012andthefallof2014,and83,887potentialcomparisongroupstudentsfromthecorrespondingsemesters.

First,weidentifiedagroupofcomparisontraditionaldevelopmentalmathstudentswithsimilarcharacteristicstoQuantway1students.Toobtainpropensityscores,wetookahierarchicallinearmodeling(HLM)approach(Hong&Raudenbush,2005,2006;Raudenbush&Bryk,2002;Yamada&Bryk,2016)andconstructedatwo-levelHLMmodelwithatotalof37student-levelcovariatesincludingstudentbackgroundcharacteristicsandpriorcoursetakingandsuccesspatternsduringthetwoyearspriortotheQuantway1term.Weselectedcovariatesbasedonpriorresearchfindingsandadvicefrominstitutionalresearchersintheparticipatingcolleges.Thelistincludesstandardstudentbackgrounddatasuchassexandrace/ethnicity.Ithasbeenshownthatthesecharacteristicstendtodifferentiatestudents’progressindevelopmentalmathsequences(Baileyetal.,2010).Wealsomatchedonstudents’priorcourse-takinghistoryandperformanceinthepasttwoyears.Previousresearchdemonstratedthatstudents’priorcoursetakinghistoryandsuccesspatternsareamorereliableindicatorofstudents’educationalandcareergoalsthantheirdeclaredprogramofstudy(Jenkins&Cho,2012).

Table1presentsallofthecovariatesusedinthepropensityscorematchingandtheirdescriptivestatisticsbeforeandafterpropensityscorematchingwasconducted.Wefoundasubstantialnumberofunknownrecordsforstudents’dateofbirthwhencomputingstudents’ageinyears.Tofactorthesecasesintothepropensitymodel,weconstructedadummyvariableandcodedmissingageas1,otherwise0.Also,weaccountedforsixcohortgroupsbyformulatingasetofdummyvariableswithSpring2014asareferencecategory.ThedescriptivedataontheleftpanelofTable1showsthatoverallQuantway1studentshavehigherproportionsoffemaleandHispanicstudentsthanthenon-Quantway1students.Quantway1studentshadmorecourserecordsinthetwoyearsbeforetakingaQuantway1course,

4 One college was on a quarter system until the fall of 2012 and implemented Quantway 1 for the first time in the winter of 2012.


suggestingthatthetermtheytookaQuantway1coursewaslesslikelytobetheirfirstsemesteroryear.Quantway1studentsalsostartedtheirdevelopmentalcourse(s)earlier,andattemptedmoredevelopmentalmathcoursesandcollege-levelcoursesthannon-Quantway1students.

Weconductedpropensityscorematchingseparatelyforeachcohortandcollegebyapplyinganearestneighbormatchingalgorithm(Rosenbaum&Rubin,1985).ThisalgorithmwasappropriateforourstudybecausewewantedtoretainasmanyQuantway1studentsaspossibleandhadalargepoolofnon-Quantway1studentsforcreatingmatches.WeattemptedtofinduptofivematchesperQuantway1student(5:1ratiomatching)tomaximizethebestmatchesfromthenon-Quantway1studentgroupwhilestillmaintainingprecision(Ming&Rosenbaum,2000).Wealsospecifiedacaliperdistanceofupto0.2toreducetheriskofbadnearestneighbormatchesbasedonrecommendationsintheliterature(Austin,2011;Rosenbaum&Rubin,1985).Forpropensityscorematching,weusedthepackageMatchIt(Ho,Imai,King,&Stuart,2011)inR(RCoreTeam,2015).WethenestimatedQuantway1’seffectivenessbycomparingsuccessratesofQuantway1studentswiththeirmatchedcomparisonsusingafour-levelHLMmodelwithabinaryoutcome.SuccesswasdefinedasapassinggradeoragradeofCorhigher5onaQuantway1courseforQuantway1studentsandadevelopmentalmathcourseonelevelbelowcollegelevel(oranothercoursedeemedequivalenttoaQuantway1coursebyfaculty)forthematchedcomparisonstudents.Forthelattergroup,wetrackedcourseoutcomesovertheentireacademicyear(i.e.,trackingcourseoutcomesoverthefallandspringsemestersforthefallcohortsandthespring,summer,andfallsemestersforthespringcohorts).Asdescribedearlier,Quantway1acceleratestraditionaldevelopmentalmathsequencesforstudentsplacedtwolevelsbelowcollegemathematicsinonesemester.Similarstudentsfollowingthetraditionaldevelopmentalmathroutetypicallycompletethedevelopmentalsequenceinoneandahalfyears(Baileyetal.,2010;Cullinane&Treisman,2010).Accordingly,ifcomparisonstudentsinthefallcohortshadfailedadevelopmentalmathcourseonelevelbelowcollegelevelinthefallsemesterbutpasseditthefollowingspringsemester,wecounteditassuccess.Therefore,analysiswasconservative,providingcomparisonstudentstwiceasmuchtimetoreachthesamesuccessbenchmarkasQuantway1students.InthisHLMframework,matchedclusters(level1)werenestedwithinQuantway1students(level2),whowereinturnnestedwithinQuantway1facultymemberclassrooms(level3)withintheircolleges(level4).BecausematchedcomparisonswereconstructedforeachQuantway1student,theirrespectivecomparisonstudentswerealsoassignedthecorrespondingQuantway1facultyID.Thisstrategyallowedustoformeachfacultymember’sclassroomasamini-experimentinwhichthemeanoutcomeoftheirQuantway1studentswascomparedwiththatofsimilarstudentswhopursuedtraditionalmathcourses,inorderto

5 A grade of C- or higher was used for a college that employed a +/– grading scheme.


estimatethevariabilityineffectamongfacultywithincolleges.WeusedHLM7(Raudenbush,Bryk,Cheong,Congdon,&duToit,2011)foralloftheHLManalyses.

RESULTSPROPENSITYSCOREMATCHINGToobtainpropensityscores,weconstructedatwo-levelBernoullimodelandestimateditsmodelparametersusingmaximumlikelihoodviapenalizedquasi-likelihoodestimation.ϕilistheprobabilityofstudentienrollinginQuantway1incollegel.Accordingly,ηilisthelog-oddsofthisincidentandformallyexpressedas:

Level-1Model(Student)Prob(QWil=1|βl)=ϕillog[ϕil/(1-ϕil)]=ηilηil=β0l+β1l(COV1il)+…+β37l(COV37il),Level-2Model(College)β0l=γ00+u0l,β1l=γ10,...Β37l=γ370,

whereQWisadummyvariableindicatingwhetheragivenstudentwasenrolledinQuantway1(codedas1)ornot(codedas0),COV1…COV37arethesetofpropensityscorecovariates.6Wematchedatotalof12,448comparisonstudentsto3,992Quantway1students.Table1comparesthedescriptivestatisticsoneachcovariatebeforeandaftermatchingtotheQuantway1group.Table2documentsthebalanceinpropensityscorecohortbycohortforeachcollege.TherewerenosignificantdifferencesinmeanpropensityscoresbetweentheQuantway1andmatchedcomparisonstudentsinanyofthecohortsforeachcollege(seetvalues).Theseresultsprovidestrongevidencethatcomparabilityofthegroupswasachievedonthemeasuredcovariates.Itmaybeworthwhileheretomentionthematchedratiosweaccomplished.AsdescribedearlierintheMethodsection,weattemptedtofinduptofivematchesperQuantway1student.Thematchedratiosinthefarrightcolumnsuggestthatingeneral,weidentified4to5

6 We initially included two covariates of college non-STEM courses (the number of courses attempted and the respective success rate). However, they involved collinearity with other covariates, and accordingly, the model did not converge. Thus, we excluded them from the propensity model.


matchesperQuantway1student.ForsomecohortsfromColleges3and8,however,weidentifiedfewermatchesandneededtoexcludesomeQuantway1studentstomaintainthecomparabilityofthegroups.Itappearsthatbothcollegeshavearelativelylargepopulationofstudentswhowereplacedintodevelopmentalmathcoursesandaccordingly,morestudentsatvaryinglevelsofdevelopmentalmath.Therefore,itmaybepossiblethatcertainkindsofstudents(e.g.,thosewhofaileddevelopmentalcoursesmultipletimes)wereadvisedtotakeQuantway1soastolimitthenumberofappropriatestudentsformatching.ESTIMATINGQUANTWAYEFFECTSToestimatedifferencesinsuccessrates,weconstructedafour-levelBernoullimodelandestimateditsmodelparametersusingmaximumlikelihoodviapenalizedquasi-likelihoodestimation.ϕijklrepresentstheprobabilitythatstudentimatchedwithQuantway1studentjassociatedwithfacultymemberk’sclassincollegelsuccessfullycompletedthedevelopmentalmathsequence.Correspondingly,ηijklisthecorrespondinglog-oddsofthisoutcomeandformallyexpressedas:

Level1Model(Student)Prob(SUCCijkl=1|πjkl)=ϕijkl,log[ϕijkl/(1-ϕijkl)]=ηijkl,ηijkl=π0jkl+π1jkl*(QWijkl),Level2Model(QWStudent)π0jkl=β00kl+β01kl(TERMjkl)+β02kl(W12jkl)+β03kl(S12jkl)+β04kl(F12jkl)+β05kl(S13jkl)+β06kl(F13jkl)+β07kl(F14jkl)+r0jkl,π1jkl=β10kl+β11kl(TERMjkl)+r1jkl,Level3Model(Faculty)β00kl=γ000l+u00kl,β01kl=γ010l,β02kl=γ020l,β03kl=γ030l,β04kl=γ040l,β05kl=γ050l,β06kl=γ060l,β07kl=γ070l,β10kl=γ100l+u10kl,β11kl=γ110l,


Level4Model(College)γ000l=δ0000+v000lγ010l=δ0100,γ020l=δ0200,γ030l=δ0300,γ040l=δ0400,γ050l=δ0500,γ060l=δ0600,γ070l=δ0700,γ100l=δ1000+v100l,γ110l=δ1100,

whereSUCCrepresentsdevelopmentalmathachievement(1forsuccessfullycompletedand0fornotsuccessfullycompleted),andQWisadummyvariableindicatingwhetherthestudentwasenrolledinQuantway1(codedas1)oroneofthematchedcomparisons(codedas0).AllthecovariatesatLevel2wereincludedasadditionaladjustmentvariablesfortheoutcome.Termisadummyvariableindicatingwhethertheoutcomeformatchedcomparisonstudentswasbasedononesemester(codedas1)ortheentireacademicyear(codedas0).W12toF14aredummyvariablesforthesixcohortgroupsdescribedearlierintheMethodsection.7TheresultspresentedinTable3indicatethatonaverage,Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccess,2.10(95%CI[1.35,3.27]8),insuccessfullycompletingthedevelopmentalmathcoursethanthematchedcomparisonstudents.Thecorrespondingestimatedprobabilitiesofsuccesswere56.78%fortheQuantway1groupand38.51%forthematchedcomparisongroup.VARIATIONINPERFORMANCETheestimatedcoefficientsbetweentheinterceptandtheslopeatbothcollegeandfacultylevelswerenegative(-.70and-.41,respectively),suggestingthatthelowertheoutcomeforthematchedcomparisongroup,thelargertheeffectofQuantway1.Thisinverserelationshipwasstrongeratthecollegelevel.Inaddition,wefoundvariationinQuantway1effectamongcollegesandfacultymembers(0.36and0.23forthecollegeandfacultyvariances).Figures3and4displaythevariationinQuantway1effectsizeatthecollegeandfacultylevels,respectively.Inbothcharts,weaddedthreelinesasreferences.ThecenterlinerepresentstheaverageeffectofQuantway1,andtheupperandlowerlinesrepresenttheupperandlowerboundsoftheaverageeffect(whicharedeviatedintwoSEsfromthecenterline).Figure3demonstratesthattherewerepositiveQuantway1effectsonstudentoutcomesinallbutCollege10(whichshowednoeffectof7 We also constructed the same four-level model with individual students’ propensity scores included in the level 1 model and those six cohort group variables added to the level 2 model for the slope. The results from this model revealed no significant coefficients of these additional covariates and closely mirrored those from the simpler model. For the ease of interpretation, we focus here on the results from the simpler model. 8HLM7generates95%confidenceintervalsofoddsra0os.


Quantway1).College8standsoutasapositivedeviantwithaQuantway1effectoutsidetheupperboundoftheaverageeffect.Figure4showsthevariationinQuantway1effectivenessacrosstheclassroomsintheNIC.ThevastmajorityofQuantway1facultyatCollege8drasticallyoutperformedtheaverageQuantway1faculty,suggestinginternalcoherenceatthisinstitution.Incontrast,awiderangeofvariationwasobservedamongfacultymembersatCollege3.Understandingthemechanismsthatenableconsistentlyhighperformanceandinvestigatingthecausesofvariationareareasoffuturestudy.

SUBGROUPANALYSISToexaminepossibledifferentialeffectsofQuantway1bysexandrace/ethnicitysubgroups,weconstructedafour-levelHLMsimilartothosedescribedabove.Inthissubgroupanalysis,however,weappliedeffectcodingtothegroupingvariablesinordertodirectlyrepresentbothmainandinteractioneffectsontheoutcome.ThereferencecategorieswerefemaleandWhite.Eachofthesewascodedas-1.Weexcludedcaseswiththeunknownsexstatus.Figure5presentsthemodel-basedresultstransformedbackintotheirnaturalmetricsofproportionofstudentssuccessfullycompletingadevelopmentalmathsequence.Thismetrictransformationwasmadefortheeaseofinterpretation.PositiveeffectsofQuantway1wereobservedforeachrace/ethnicitygroup.BlackandHispanicmalestudentsthattakeQuantway1exhibitedthelargestincreaseincompletionratesrelativetobaselinelevelsofperformance.SENSITIVITYANALYSISIngeneral,Quantway1effectswerestrongandprevalentforallsubgroups.Thevalidityoftheseeffectswasbasedonanassumptionofastronglyignorabletreatmentassignment.Inotherwords,allrelevantcovariateswereincludedinthepropensityscoreanalysis,sothatthebiasduetounmeasuredcovariatescouldbeignored.Thus,weexaminedthesensitivityoftheestimatedQuantway1effectstopossibleconfoundingbyunmeasuredvariables(Hong&Raudenbush,2005,2006;Lin,Psaty,&Kronmal,1998).Givensomeunmeasuredcovariates(U),theQuantway1effect(δ)canbere-estimatedbyadjustingforhypothesizedhiddenbias(γ(E[U1]-E[U0]))asδ*=δ-γ(E[U1]-E[U0]),whereγistheunmeasuredcovariates’associationwiththeoutcomeandE[U1]-E[U0]istheirassociationwithtreatmentassignment(i.e.,Quantwayornon-Quantwayenrollment).AdaptingtheapproachofHongandRaudenbush(2005,2006),weoperationallydefinedaproxyforγasacoefficientderivedfromafour-levelmodeldesignedtopredicttheoutcomewiththesamesetofcovariatesusedinthepropensityscoreanalysisandE[U1]-E[U0]astheobservedmeandifferencebetweentheQuantwayandnon-Quantwaygroupsonthecorrespondingcovariate.Wethenselectedthelargestpositivevalueoftheproductofthesetwovaluesasthelargestpossiblebias9andobtainedanadjustedQuantwayestimate(δ*).9 We used the sum of the product values for those requiring a set of dummy variables (e.g., cohort group,

race/ethnicity).


Accordingly,were-estimatedaQuantway1effectontheoutcomeandconstructeda95%confidenceintervalfortheadjustedestimate.Theadjustedestimatewas.69inlogits(95%CIs[.28,1.11]),andthecorrespondingconfidenceintervaldidnotcontain0oranynegativevalue,therebysupportingthestrongignorabilityassumption.OursensitivityanalysisconcludesthatitisveryunlikelythatourgeneralconclusionregardingQuantway1’spositiveeffectswasinfluencedbytheomissionofunmeasuredconfoundingfactors.DISCUSSIONThisstudyassessedQuantway1’seffectivenessforcommunitycollegestudentsacrosssixsemestersofimplementationusingarigorouscausalanalysis.Apropensityscorematchingtechnique(Rosenbaum&Rubin,1983)withinanHLMframework(Raudenbush&Bryk,2002)allowedustocontrolforpossibleselectionbiasbymatchingQuantway1studentswithcomparablestudentsenrolledintraditionaldevelopmentalmathsequencesacross37studentcharacteristics.Wealsoundertookasensitivityanalysistoexaminethepossibilitythattheestimatedeffectswereinfluencedbyunmeasuredconfoundingvariables.Throughouttheseanalyses,Quantway1studentsdemonstratedsignificantlyhigheroddsofsuccessthanmatchedcomparisonstudents.Ouranalysesalsoindicatedthattheseresultswerenotduetounmeasureddifferencesbetweenthetwogroups.WeconcludethatQuantway1substantiallyimprovesstudentsuccessratesinfulfillingdevelopmentalmathcourserequirements.WhiletypicalevaluationsmaystopatestimatingtheeffectsizeofQuantway1,thisstudyalsosoughttounderstandthevariationoftheQuantway1programacrossdifferentcolleges,faculty,andstudentsubgroups.Inordertoachieveefficacyatscale,Quantway1mustnotonlyproduceapositiveeffectonaverage,butmustalsobeeffectivefordiversestudentpopulationsacrossarangeofdifferentclassroomandinstitutionalcontexts.OuranalysisfoundthatQuantway1effectswerepositiveacrossallsexandracial/ethnicsubgroupsofstudents.Aspreviouslystated,studentsfromtraditionallyunderservedgroups,includingBlacks,Hispanics,andlow-incomestudents,aremorelikelytoparticipateindevelopmentalmathcourses(Chen2016).Becauseineffectivetraditionaldevelopmentalmathcourseshaveadisproportionateimpactonthesetraditionallyunderservedgroups,theseresultssuggestthatQuantway1canplayanimportantroleinincreasingtheoverallnumberoftraditionallyunderservedstudentscompletingtheirmathrequirements.Inaddition,Quantway1hasapositiveeffectinnearlyallclassroomsandcollegesinthenetwork,indicatingthattheprogramcanworkforvariedfacultyindifferentinstitutionalcontexts.Quantway1’sdesignincludeskeyleversthatmayexplainwhyitbettersupportstraditionallyunderservedcommunitycollegestudents.First,Quantway1isaquantitativeliteracycoursethatacceleratesstudentprogresstocollege-levelmathbyofferingdevelopmentalmathrequirementsinasingleterm.Second,Quantway1’sresearch-basedinstructionalsystemcontextualizesmathconceptsandisorganizedaroundthreelearningopportunitiestopromoterichmathematicallearningforabroaderrangeofstudents.AnecdotalaccountsfromstudentsindicatethatQuantway1’suniquemathematicalexperienceshelpthemseethemselvesas


mathematicallearnersanddoers.Third,Quantway1aimstosupportsocioemotionalskillsandprovidelanguageandliteracysupportstohelpstudentsgrapplewiththecomplexlanguageofmathematics.Finally,Quantway1’sFacultySupportProgramandtheNICstructuresupportfacultyinimplementingQuantway1’suniquepedagogicalpracticesacrossdifferentfacultyandinstitutionalcontexts.OurresultssuggestthatQuantway1’scomprehensiveandsystematicapproachtotacklingthetypicalbarriersthatdevelopmentalmathstudentsfaceiskeytoitssuccess.Furtherempiricalevidenceisneededtoconnectparticulardesignelementstothepositiveeffectsoftheprogram.Fornow,wecanconcludethattheQuantway1packageisaneffectivealternativetothetraditionaldevelopmentalmathsequenceandacceleratestheabilityofadiverserangeofstudentstocompletetheirdevelopmentalmathrequirementsinavarietyofcontexts.TheseresultsandQuantway1’sflexiblesingle-termstructuredemonstrateitssignificantpotentialtopositivelyimpactnumerousstudentsinavarietyofcommunitycollegecontexts.Itisworthnotingacoupleofthekeylimitationsofthecurrentstudythatwouldbefruitfultopicsforfutureinvestigations.Oneopportunityforfutureresearchistotrackcollege-levelmathachievementbetweenthetwomatchedgroupsoneyearafterQuantway1enrollmenttodetermineifQuantway1studentsperformcomparablyorbetterthanmatchedstudentgroupsinfuturecollege-levelmathcourses.SinceQuantway1isdesignedtonotonlygetstudentsthroughtheirdevelopmentalmathsequencesbuttopreparethemtomeettheircollegemathrequirements,thiswillbeaparticularlyimportantanalysisindeterminingQuantway1’seffectiveness.Aslongitudinaldatabecomeavailable,wealsoplantotracklonger-termoutcomes,suchastransferratesandacademicsuccessofQuantway1studentsin4yearinstitutions.ThesignificantvariationinoutcomesacrossfacultyandcollegesdemonstratedinFigures3and4presentsanotheropportunityforfurtherinvestigation.Thegoalofqualityimprovementistoreducethevariationbetweenclassroomsandcollegesachievingpositiveresultsacrossdiversecontexts.Investigatingpositiveandnegativedeviantsprovidesinsightintothekeysourcesofvariation.College8,forexample,significantlyoutperformstheothercollegesinthenetworkandmaintainshighperformanceacrossalltheclassroomsinthecollege.College10,incontrast,performssignificantlyworsethanothercolleges.Futureresearchshouldexplorewhetherthesecollegesdifferinhowtheyenactthekeydesignelementsdescribedabove,andstudythevariousadaptationsthatthesecollegesmadeinresponsetotheirlocalcontext.DiscoveringandsharingkeypracticesacrossNICcollegeswouldenhancethenetwork’sabilitytoreplicateQuantway1’spositiveoutcomesasitspreadstomorediversecontexts.Inconclusion,byredesigningthecontent,pedagogy,andstructureoftraditionaldevelopmentalmathcourses,Quantway1providesarichmathematicalexperienceforabroaderrangeofdevelopmentalstudents,includinghistoricallyunderservedgroups.TheseeffortshavecontributedtoQuantway1’spositiveimpactonequitableoutcomesbyimprovingcompletionratesforallsexandracial/ethnicsubgroupsandacrossdiversecontexts.Despitegreatadvancesinincreasingdevelopmentalmathcompletionratesonaverage,variationinoutcomes


acrossfacultyandcollegesintheNICindicatesthereisstillmuchworktobedoneinadvancingQuantway1’sefficacyreliablyatscale.ByleveragingtheNICstructure,wewillcontinuetoacceleratelearningthroughqualityimprovementtosolvethisdevelopmentalmathcrisis.


REFERENCESAustin,P.C.(2011).Optimalcaliperwidthsforpropensity-scorematchingwhenestimating

differencesinmeansanddifferencesinproportionsinobservationalstudies.PharmaceuticalStatistics,10,150-161.

Bailey,T.,Jeong,D.W.,&Cho,S.-W.(2010).Referral,enrollment,andcompletionin

developmentaleducationsequencesincommunitycolleges.EconomicsofEducationReview,29,255–270.

Blackwell,L.S.,Trzesniewski,K.H.,&Dweck,C.S.(2007).Implicittheoriesofintelligence

predictachievementacrossanadolescenttransition:Alongitudinalstudyandanintervention.ChildDevelopment,78,246–263.

Bryk,A.S.,Gomez,L.M.,Grunow,A.,&LeMahieu,P.G.(2015).Learningtoimprove:How

America'sschoolscangetbetteratgettingbetter.Cambridge,MA:HarvardEducationPress.

Bueschel,A.C.(2004).Themissinglink:Theroleofcommunitycollegesinthetransi?on

betweenhighschoolandcollege.InM.W.KirstandA.Venezia(Eds.),Fromhighschooltocollege:Improvingopportuni1esforsuccessinpostsecondaryeduca1on(pp.252–284).SanFrancisco:Jossey-Bass.

Carnevale,A.P.,&Desrochers,D.M.(2003).Standardsforwhat?:TheeconomicrootsofK-16reform.Princeton,NJ:EducationalTestingService.

Carnevale,A.P.,Smith,N.,&Strohl,J.(2013).Recovery:Projectionsofjobsandeducation

requirementsthrough2020.GeorgetownUniversityCenteronEducationandtheWorkforce.

Chen,X.(2016).RemedialcoursetakingatU.S.public2-and4-yearinstitutions:Scope,

experiences,andoutcomes(NCES2016-405).U.S.DepartmentofEducation,Washinton,DC:NationalCenterforEducationStatistics.

Cullinane,J.,&Treisman,P.U.(2010).Improvingdevelopmentalmathema1cseduca1onin

communitycolleges:AprospectusandearlyprogressreportontheStatwayini7a7ve.AnNCPRWorkingPaper.Na#onalCenterforPostsecondaryResearch.

Edwards,A.&Beattie,R.(2015).Promotingstudentlearningandproductivepersistenceindevelopmentalmathematics.NADEDigest,9,30-39.


Edwards,A.R.,Sandoval,C.,&McNamara,H.(2015).Designingforimprovementinprofessionaldevelopmentforcommunitycollegedevelopmentalmathematicsfaculty.JournalofTeacherEducation,66,466-481.doi:10.1177/0022487115602313.

Gillman,R.(2004).SIGMAAisFormed.Focus,24(1),5.Gomez,K.,Rodela,K.,Lozano,M.,&Mancevice,N.(2013).Designingembeddedlanguageand

literacysupportsfordevelopmentalmathematicsteachingandlearning.MathAMATYCEducator,5,43-56.

Gomez,K.,Gomez,L.M.,Rodela,K.C.,Horton,E.S.,Cunningham,J.,&Ambrocio,R.(2015).

Embeddinglanguagesupportindevelopmentalmathematicslessons:Exploringthevalueofdesignasprofessionaldevelopmentforcommunitycollegemathematicsinstructors.JournalofTeacherEducation,66,450-465.doi:10.1177/0022487115602127.

Handel,M.J.(2007,May31–June1).Anewsurveyofworkplaceskills,technology,

andmanagementpractices(STAMP):Backgroundanddescriptivestatistics.PaperpresentedattheWorkshoponResearchEvidenceRelatedtoFutureSkillDemandsoftheNationalResearchCounciloftheNationalAcademies,Washington,DC.

Haynes,T.L.,Perry,R.P.,Stupnisky,R.H.,&Daniels,L.M.(2009).Areviewof

attributionalretrainingtreatments:Fosteringengagementandpersistenceinvulnerablecollegestudents.InJ.C.Smart(Ed.),Highereducation:Handbookoftheoryandresearch(pp.227–272).NewYork,NY:Springer.

Hiebert,J.,&Grouws,D.(2007).Theeffectsofclassroommathematicsteachingonstudents’

learning.InF.K.LesterJr.(Ed.),Secondhandbookofresearchonmathematicsteachingandlearning(pp.371–404).Charlotte,NC:InformationAge.

Ho,D.E.,Imai,K.,King,G.,&Stuart,E.A.(2011).MatchIt:Nonparametricpreprocessingfor

parametriccausalinference.JournalofStatisticalSoftware,42,1-28.Hong,G.,&Raudenbush,S.W.(2005).Effectsofkindergartenretentionpolicyonchildren’s

cognitivegrowthinreadingandmathematics.EducationalEvaluationandPolicyAnalysis,27,205-224.

Hong,G.,&Raudenbush,S.W.(2006).Evaluatingkindergartenretentionpolicy:Acasestudyof

causalinferenceformultilevelobservationaldata.JournaloftheAmericanStatisticalAssociation,101,901-910.


Huang,M.,Hoang,H.,Yesilyurt,S.,&Thorn,C.(2016).CommunityCollegePathways:2014-2015ImpactReport.Stanford,CA:CarnegieFounda4onfortheAdvancementofTeaching.

Jenkins,D.,&Cho,S.W.(2012).Getwiththeprogram:Acceleratingcommunitycollegestudents’entryintoandcompletionofprogramsofstudy(CCRCWorkingPaperNo.32).NewYork,NY:ColumbiaUniversity,TeachersCollege,CommunityCollegeResearchCenter.

Johnstone,R.(2013).FiscalconsiderationsofStatway®andQuantway®:Weshouldbedoingthis

anyway,buthere’showitmayhelpthebottomline.NationalCenterforInquiry&Improvement(NCII).Retrievedfromhttp://www.inquiry2improvement.com/attachments/article/12/NCII-Carnegie_SW-QW_Fiscal_Considerations_110713-Rob-NCII.pdf.

Lin,D.Y.,Psaty,B.M.,&Kronmal,R.A.(1998).Assessingthesensitivityofregressionresultsto

unmeasuredconfoundersinobservationalstudies.Biometrics,54,948-963.Ming,K.,&Rosenbaum,P.(2000).Substantialgainsinbiasreductionfrommatchingwitha

variablenumberofcontrols.Biometrics,56,118-124.Mesa,V.(2011).Similaritiesanddifferencesinclassroominteractionbetweenremedialand

collegemathematicscoursesinacommunitycollege.JournalonExcellenceinCollegeTeaching,22(4),21-55.

RCoreTeam.(2015).R:Alanguageandenvironmentforstatisticalcomputing[Computer

software].Vienna,Austria:RFoundationforStatisticalComputing.Raudenbush,S.W.,&Bryk,A.S.(2002).Hierarchicallinearmodels:Applica2onsanddata

analysismethods(2nded.).ThousandOaks,CA:SagePublica<ons.Raudenbush,S.W.,Bryk,A.S.,Cheong,Y.F.,Congdon,R.T.,&duToit,M.(2011).HLM7:

Hierarchicallinearandnonlinearmodeling.Lincolnwood,IL:Scien/ficSo1wareInterna'onal.

Rosenbaum,P.R.,&Rubin,D.B.(1983).Thecentralroleofthepropensityscoreinobservationalstudiesforcausaleffects.Biometrika,70,41-55.

Rosenbaum,P.R.,&Rubin,D.B.(1985).Constructingacontrolgroupusingmultivariate

matchedsamplingmethodsthatincorporatethepropensityscore.TheAmericanStatistician,39,33-38.

Sowers,N.,&Yamada,H.(2015).Pathwaysimpactreport.Stanford,CA:CarnegieFoundation

fortheAdvancementofTeaching.


Yamada,H.,&Bryk,A.S.(2016).Assessingthefirsttwoyears’effectivenessofStatway®:Amultilevelmodelwithpropensityscorematching.CommunityCollegeReview,44,179-204.


APPENDIXListofPar+cipa+ngColleges

• AtlanticCapeCommunityCollege• BoroughofManhattanCommunityCollege• CuyahogaCommunityCollege• EastGeorgiaStateCollege• MadisonCollege• MarshallUniversity• OnondagaCommunityCollege• RidgewaterCollege• RocklandCommunityCollege• SinclairCommunityCollege• SouthGeorgiaStateCollege• UniversityofNorthGeorgia,Gainesville• UniversityofWashington,Bothell• WestchesterCommunityCollege


Table1.

Descrip(veSta(s(csofCovariatesintheTwo-LevelPropensityModel

Samplebeforematching Samplea(ermatching

Non-Quantway1 Quantway1

Non-Quantway1 Quantway1

% % % %

Sex

Female* 56 62

60 61

Male 44 37

39 38

Unknown 0 1

1 1

Race/Ethnicity

Asian 3 4

3 4

Black 31 30

32 30

Hispanic 18 26

21 26

White* 36 34

36 33

Mul$racial 1 1

1 1

Other 1 0

0 0

Unknown 9 5

6 5

Anycourserecordsinpasttwoyears

No* 45 38

46 41

Yes 55 62

54 59

Cohortgroup

Winter2012 6 2

3 2


Spring2012 16 11

16 11

Fall2012 14 13

20 13

Spring2013 16 20

14 20

Fall2013 16 21

20 21

Spring2014* 17 15

14 15

Fall2014 15 19

14 19

Agemissing 21 14

23 15

M SD M SD

M SD M SD

Age(inyears) 24.98 8.47 24.09 8.18 24.09 7.55 23.97 8.11

Semesterssincefirstdevelopmentalmathcourse 0.92 1.70 1.62 2.46

1.27 2.12 1.37 2.25

Courseload 3.69 1.20 3.95 1.14

3.95 1.19 3.93 1.14

Developmentalmath

Onelevelbelowcollegelevel

Numberofcoursesa-empted 0.09 0.31 0.21 0.55

0.15 0.41 0.17 0.47

Successrate 0.00 0.04 0.00 0.04

0.00 0.04 0.00 0.03

Twolevelsbelowcollegelevel


0.26 0.56 0.30 0.58

Successrate 0.11 0.31 0.19 0.38

0.13 0.33 0.17 0.36

Threeormorelevelsbelowcollegelevel


0.12 0.45 0.13 0.49

Successrate 0.10 0.29 0.08 0.27

0.07 0.25 0.07 0.25


DevelopmentalEnglish


0.08 0.35 0.08 0.32

Successrate 0.06 0.24 0.05 0.21

0.05 0.21 0.05 0.21

Developmentalreading


0.09 0.32 0.11 0.37

Successrate 0.05 0.21 0.06 0.24

0.05 0.22 0.06 0.23

Developmentalwri/ng

Numberofcoursesa*empted 0.10 0.32 0.11 0.34

0.09 0.32 0.10 0.34

Successrate 0.07 0.25 0.07 0.26

0.06 0.24 0.07 0.26

Collegemath


0.05 0.27 0.07 0.28

Successrate 0.01 0.07 0.01 0.08

0.00 0.06 0.01 0.07

Collegenon-math


3.29 5.17 3.76 5.42

Successrate 0.33 0.42 0.42 0.42

0.35 0.42 0.40 0.42

CollegeSTEM


0.44 1.48 0.45 1.25

Successrate 0.07 0.24 0.09 0.26 0.08 0.25 0.08 0.25


Note.Termswith"*"wereusedasreferencecategories(codedas0)whenformula:ngdummyvariables.Agewascomputedinyearsusingadateofbirthand9/1forthefallcohortsand3/1forthespringcohorts(1/1foronewintercohortgroup).Inthecurrentpropensitymodel,wecenteredagearoundage18.Semesterssincefirstdevelopmentalmathcoursetakesaninteger,suchas0,1,2,etc.,where0meansastudenttookadevelopmentalmathcourseforthefirst(meinthesametermastheQuantway1term,1onesemesterbefore,2twosemestersbefore,andsoon.CourseloadreferstothenumberofcoursesastudenttookduringtheQuantway1term.Successratewascomputedbydividingthenumberofcoursescompletedwithapassinapass/failgradingscheme,oraCorhigher(C-ifa+/-gradingschemeisused)bythenumberofcoursesa)empted.

Table2.

BalanceinLogitofthePropensityScorefornon-Quantway1andQuantway1Students

Samplebeforematching Samplea(ermatching

Non-Quantway1

Quantway1

Non-Quantway1

Quantway1

Matchedra#oCollege Cohort

n M SD

n M SD

n M SD

n M SD t

1 2012Spring 585 -2.78 0.48 43 -2.72 0.37 212 -2.74 0.32 43 -2.72 0.37 -0.42 4.93

1 2012Fall

470 -2.41 0.31

34 -2.20 0.54

149 -2.36 0.35

31 -2.31 0.42 -0.59 4.81

1 2013Fall

305 -1.88 0.19

59 -1.79 0.38

249 -1.89 0.15

54 -1.88 0.16 -0.12 4.61

1 2014Spring

273 -2.32 0.26

29 -2.01 0.52

112 -2.23 0.26

25 -2.16 0.34 -0.94 4.48

2 2013Fall 690 -2.34 0.96 69 -2.29 0.80 337 -2.31 0.75 69 -2.29 0.80 -0.16 4.88

2 2014Spring

270 -2.93 0.62

17 -3.02 0.27

85 -3.02 0.27

17 -3.02 0.27 -0.01 5.00

2 2014Fall 402 -2.86 0.41 47 -2.99 0.36 217 -2.95 0.26 46 -2.96 0.32 0.18 4.72

3 2012Spring

4138 -3.38 0.39

72 -3.37 0.27

360 -3.37 0.27

72 -3.37 0.27 -0.05 5.00


3 2012Fall

3234 -2.86 0.40

177 -2.74 0.42

875 -2.76 0.38

175 -2.76 0.38 -0.07 5.00

3 2013Spring

3745 -2.24 0.43

584 -1.93 0.75

1057 -2.11 0.53

544 -2.06 0.59 -1.66 1.94

3 2013Fall

2358 -2.11 0.51

408 -1.66 0.93

739 -1.89 0.64

378 -1.84 0.72 -1.28 1.96

3 2014Spring

3242 -2.58 0.54

290 -2.12 0.81

559 -2.20 0.72

287 -2.14 0.78 -1.08 1.95

3 2014Fall

4696 -2.46 0.36

402 -1.90 0.94

368 -2.08 0.69

368 -2.07 0.71 -0.16 1.00

4 2012Spring 594 -3.20 0.66 38 -2.78 1.27 175 -3.03 0.88 36 -2.97 0.97 -0.33 4.86

4 2012Fall

570 -2.88 0.47

45 -2.40 0.83

193 -2.67 0.48

42 -2.54 0.65 -1.22 4.60

4 2013Spring

599 -2.20 0.48

42 -1.91 0.73

188 -2.09 0.50

41 -1.96 0.68 -1.18 4.59

4 2013Fall

681 -2.33 0.46

69 -2.01 0.89

278 -2.28 0.48

63 -2.20 0.66 -0.84 4.41

4 2014Spring

617 -2.78 0.50

49 -2.43 0.81

213 -2.65 0.57

47 -2.51 0.72 -1.16 4.53

4 2014Fall 827 -2.66 0.38 87 -2.48 0.72 408 -2.63 0.41 85 -2.56 0.53 -1.17 4.80

5 2013Fall

712 -3.88 0.58

6 -3.60 0.76

30 -3.61 0.68

6 -3.60 0.76 -0.02 5.00

5 2014Spring

648 -4.58 0.58

19 -3.65 0.93

82 -3.88 0.76

19 -3.65 0.93 -0.99 4.32

6 2012Spring 1601 -4.72 0.45 21 -4.74 0.35 105 -4.74 0.34 21 -4.74 0.35 -0.05 5.00

6 2013Spring

1481 -3.75 0.55

35 -3.44 0.87

167 -3.57 0.63

34 -3.52 0.73 -0.34 4.91

6 2013Fall

1534 -3.82 0.49

49 -3.49 0.78

232 -3.60 0.65

48 -3.54 0.73 -0.53 4.83

6 2014Spring

1436 -4.20 0.45

32 -3.76 0.87

140 -3.88 0.55

29 -3.83 0.58 -0.41 4.83

6 2014Fall 1954 -4.16 0.35 17 -3.44 0.85 71 -3.70 0.63 16 -3.55 0.74 -0.72 4.44

7 2012Spring

3675 -4.79 0.91

63 -4.49 1.02

305 -4.53 0.99

62 -4.49 1.03 -0.08 4.92

7 2012Fall

2929 -4.68 0.58

65 -4.47 0.67

320 -4.52 0.58

65 -4.47 0.67 -0.35 4.92

7 2013Spring

3831 -4.05 0.49

42 -3.92 0.52

204 -3.97 0.45

42 -3.92 0.52 -0.08 4.86

7 2013Fall

3566 -4.21 0.44

68 -4.17 0.36

338 -4.18 0.34

68 -4.17 0.36 -0.02 4.97


7 2014Spring

4097 -4.65 0.45

29 -4.29 0.80

127 -4.54 0.47

27 -4.41 0.70 -1.11 4.70

8 2012Winter 4693 -5.45 1.30 70 -3.90 1.55 348 -3.93 1.52 70 -3.90 1.55 -0.16 4.97

8 2012Spring

1955 -4.39 0.91

108 -2.32 1.83

275 -2.91 1.22

97 -2.72 1.36 -1.23 2.84

8 2012Fall

3420 -3.94 0.88

124 -2.14 1.69

443 -2.51 1.30

120 -2.29 1.47 -1.48 3.69

8 2013Spring

3615 -3.36 0.60

129 -1.93 1.36

119 -2.13 1.17

119 -2.10 1.20 -0.19 1.00

8 2013Fall

3230 -3.33 0.62

119 -1.87 1.46

105 -2.23 1.19

105 -2.20 1.20 -0.22 1.00

8 2014Spring

3088 -3.79 0.52

118 -2.49 1.23

165 -3.16 0.74

88 -3.04 0.83 -1.10 1.88

8 2014Fall 3796 -3.61 0.49 158 -2.55 1.37 253 -3.07 0.97 138 -2.89 1.09 -1.60 1.83

9 2012Spring

683 -2.86 0.56

41 -2.88 0.70

195 -2.97 0.56

40 -2.93 0.62 -0.37 4.88

9 2012Fall

776 -2.75 0.34

55 -2.33 1.00

226 -2.71 0.37

50 -2.58 0.54 -1.58 4.52

9 2014Spring

543 -2.59 0.38

30 -2.63 0.38

142 -2.59 0.33

29 -2.62 0.38 0.38 4.90

9 2014Fall

574 -2.49 0.29

71 -2.36 0.45

303 -2.43 0.27

68 -2.39 0.41 -0.91 4.46

10 2012Spring 520 -2.36 0.52 72 -2.41 0.58 343 -2.44 0.52 72 -2.41 0.58 -0.36 4.76

10 2012Fall

376 -2.33 0.38

48 -2.29 0.41

229 -2.32 0.37

48 -2.29 0.41 -0.43 4.77

10 2013Fall

243 -1.94 0.15

39 -1.87 0.16

164 -1.91 0.14

39 -1.87 0.16 -1.33 4.21

10 2014Spring

291 -2.03 0.40

27 -2.13 0.32

134 -2.14 0.31

27 -2.13 0.32 -0.13 4.96

10 2014Fall 324 -2.12 0.31 27 -1.76 0.65 109 -2.05 0.25 22 -2.04 0.26 -0.22 4.95


Table3.

Model-BasedEs(ma(onofQuantway1EffectonDevelopmentalMathAchievement

Fixedeffect Coefficient SE t p Oddsra'o

Intercept -0.49 0.15 -3.20 .005 0.61

Term -0.71 0.14 -5.24 <.001 0.49

W12 0.08 0.14 0.55 .585 1.08

S12 0.04 0.07 0.63 .531 1.04

F12 -0.11 0.06 -1.79 .073 0.89

S13 -0.19 0.06 -2.88 .004 0.83

F13 0.11 0.06 1.83 .068 1.12

F14 0.01 0.07 0.21 .834 1.01

Quantway1 0.72 0.21 3.47 .003 2.05

Term 0.30 0.30 1.00 .317 1.35

Randomeffectatlevel4(college) Variance df χ2 p Correla'on

Intercept 0.22 9 311.84 <.001 -0.70

Quantway1 0.35 9 97.93 <.001

Randomeffectatlevel3(faculty) Variance df χ2 p Correla'on

Intercept 0.02 70 112.41 .001 -0.41

Quantway1 0.20 70 182.71 <.001


Figure1.Quantway1vs.Tradi*onalmathsequence

Quantway2orotherquan+ta+ve

reasoning

CollegeMathCredit

Quantway1

Elementary

Algebra

Intermediate

Algebra

College

Math

CollegeMathCredit

Semester1 Semester2 Semester3ormore

Tradi&onalMathSequence


Figure2.SixkeydriversofQuantway

AcceleratedPathwaythroughCollegeLevelMath

NetworkedImprovementCommunity

FacultyDevelopment

Socioemo'onalSupports

(Produc(vePersistence)

LearningPrinciplesforCurriculumandInstruc6on:

Produc'veStruggle,ExplicitConnec,ons,DeliberatePrac,ce

LanguageandLiteracySupports

Goal:

Increasethenumberofstudentsachievingcollegemathcreditwithinoneyearofcon*nuousenrollment


Figure3.Varia,oninQuantway1effectamongcolleges

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

10 6 1 5 3 2 4 9 7 8

QW

eff

ect (

in lo

gits

)

Quantwayeffectbycollege Upperbound Averageeffect Lowerbound


Figure4.Varia,oninQuantway1effectamongfacultymembers

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

10_0

2 10

_01

3_13

1_

05

1_04

6_

05

3_26

3_

11

3_07

6_

01

6_02

10

_03

3_27

3_

14

4_03

6_

04

6_03

3_

24

3_02

3_

09

3_16

3_

12

5_01

3_

20

2_01

4_

04

2_02

4_

06

1_01

9_

06

4_07

3_

18

4_05

3_

03

9_05

9_

02

5_02

3_

06

3_15

3_

08

7_02

9_

03

4_01

3_

01

3_22

1_

02

9_01

2_

03

3_10

3_

19

3_25

7_

05

3_17

7_

01

9_04

4_

02

3_28

3_

04

3_21

3_

23

1_03

7_

03

9_07

8_

04

3_05

7_

04

8_05

8_

03

8_02

8_

15

8_08

8_

12

8_09

8_

07

8_10

8_

16

8_06

8_

01

8_14

8_

11

8_13

Qua

ntw

ay e

ffec

t (in

logi

ts)

Quantway effect by faculty Upper bound Average effect Lower bound


Figure5.Model-basedsuccessratesbysexandrace/ethnicity

69%

48% 53%

58%

68%

46% 47%

61% 52%

30% 35%

40%

51%

29% 30%

43%

0%

20%

40%

60%

80%

100%

White Black Hispanic Other White Black Hispanic Other

Female Male

Succ

ess R

ate

(Pas

s or

C o

r hi

gher

)

Quantway Matched comparison


CarnegieFoundationfortheAdvancementofTeaching51VistaLaneStanford,California94305650-566-5100

CarnegieFounda-onfortheAdvancementofTeachingseekstovitalizemoreproduc2veresearchanddevelopmentineduca2on.Webringscholars,prac//oners,innovators,designers,anddeveloperstogethertosolveprac/calproblemsofschoolingthatdiminishourna'on’sabilitytoeducateallstudentswell.Wearecommi,edtodevelopingnetworksofideas,exper9se,andac9onaimedatimprovingteachingandlearningandstrengtheningtheins2tu2onsinwhichthisoccurs.Ourcorebeliefisthatmuchmorecanbeaccomplishedtogetherthaneventhebestofuscanaccomplishalone.

www.carnegiefoundation.org

Weinviteyoutoexploreourwebsite,whereyouwillfindresourcesrelevanttoourprogramsandpublicationsaswellascurrentinformationaboutourBoardofDirectors,funders,andstaff.

©2016CarnegieFounda,onfortheAdvancementofTeaching

Documents

Eﬀec%veness of PATHWAYSstorage.googleapis.com/cmp-wordpress-public.appspot.com/... · 2017-12-05 · MATH PATHWAYS TECHNICAL REPORT Assessing the Eﬀec%veness ... , and Lumina