GME Graduate Retention Rates: A Single Institution Study

Western Michigan University Western Michigan University

ScholarWorks at WMU ScholarWorks at WMU

Dissertations Graduate College

12-2015

GME Graduate Retention Rates: A Single Institution Study GME Graduate Retention Rates: A Single Institution Study

Tracy J. Frieswyk Western Michigan University, [email protected]

Follow this and additional works at: https://scholarworks.wmich.edu/dissertations

Recommended Citation Recommended Citation Frieswyk, Tracy J., "GME Graduate Retention Rates: A Single Institution Study" (2015). Dissertations. 1173. https://scholarworks.wmich.edu/dissertations/1173

This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected].

http://scholarworks.wmich.edu/


https://scholarworks.wmich.edu/

https://scholarworks.wmich.edu/dissertations

https://scholarworks.wmich.edu/grad

https://scholarworks.wmich.edu/dissertations?utm_source=scholarworks.wmich.edu%2Fdissertations%2F1173&utm_medium=PDF&utm_campaign=PDFCoverPages

https://scholarworks.wmich.edu/dissertations/1173?utm_source=scholarworks.wmich.edu%2Fdissertations%2F1173&utm_medium=PDF&utm_campaign=PDFCoverPages

mailto:[email protected]



GMEGRADUATERETENTIONRATES:ASINGLEINSTITUTIONSTUDY

by

TracyJ.Frieswyk

AdissertationsubmittedtotheGraduateCollegeinpartialfulfillmentoftherequirementsforthe

DegreeofDoctorofPhilosophyEducationalLeadership,ResearchandTechnology

WesternMichiganUniversityDecember2015

DoctoralCommittee:

JessacaSpybrook,Ph.D.,ChairChrisCoryn,Ph.D.AlanT.Davis,Ph.D.

GMEGRADUATERETENTIONRATES:ASINGLEINSTITUTIONSTUDY

TracyJ.Frieswyk,Ph.D.

WesternMichiganUniversity,2015

Graduatemedicaleducation(GME)referstotheadvancedinstructionprovided

toclinicianswhohavepreviouslyreceivedtheirMD(doctorofmedicine)orDO(doctor

ofosteopathicmedicine)degrees.GMEeducationtakesplaceintheclinicalsetting

(e.g.,hospitals,clinics),deliveringthetrainingnecessaryforphysicianstobecome

licensedtopracticemedicine,aswellastobecomeboard-certifiedintheirspecialty.

GMEtrainingprogramsthroughouttheUSareabsolutelyessential,astheyarethe

primarysourceinthiscountryforthephysicianworkforce.

Oneofthemajordecisionsinaphysician’slifeastheyapproachtheendofGME

trainingiswheretopracticemedicine.Federalandstategovernments,alongwithother

sourcesdevotesubstantialresourcestothetraininganddevelopmentofmedical

doctors.Inatimeofincreasedconcernsoveranimpendingshortageofphysiciansand

thecostsassociatedwithGMEtraining,therearekeyincentivestoidentifyingthose

GMEgraduateswhoaremostlikelytopracticemedicineinthestateinwhichthey

trained.Thus,GMEtrainingprogramsareinterestedinlearningmoreaboutthefactors

thatinfluencein-statepracticelocationdecisions,aswellashowtoidentifygraduates

thatarelikelytopracticein-state.

Thefocusofthisdissertationwastoutilizelogisticregressionwithcross-

validationtoexaminein-stateretentionusingindividualleveldemographicand

educationalpredictorsinordertocreateapilot-scoringtooltoidentifygraduatesfroma

Michigan-basedGMEtrainingprogramwhoarelikelytopracticemedicineinMichigan

post-training.ResultsshowedthataconnectiontothestateofMichigan(e.g.,being

borninMichigan,graduatingfromauniversityormedicalschoolinMichiganand

completingGMEtraininginMichigan),aswellasgraduatingfromaprimarycare

programandbeingmarried,werepredictiveofin-stateretention.Ascoreassociated

witheachvariablewasdeterminedandapilot-scoringtoolwascreatedtoidentifyGME

graduateslikelytopracticeinMichiganpost-training.Atoollikethiscouldbeusedin

targetedrecruitmenteffortstowardsgraduateslikelytopracticeinMichiganafter

training.Furtherstudiestodeterminethereliability,validityandapplicabilityofthis

scoringtoolarenecessary.

CopyrightbyTracyJ.Frieswyk

2015

ii

ACKNOWLEDGMENTS

First,Iwouldliketothankmybossandcommitteemember,AlanT.Davis,PhD,

aswellastheothermembersofmycommittee,JessacaK.Spybrook,PhDandChris

Coryn,PhD,forassisting,mentoringandguidingmethroughthisprocess.Ifitwerenot

forDr.DavisIamnotsureIwouldhavemadeitthroughtheEMRprogram/dissertation.

HisabilitytolistentomyfrustrationsandcomplaintsaboutfeelinglikeIwouldnever

understandanythingallthewhilepatientlymentoringmethroughschool,dissertation

andworkhasbeenaninvaluableexperiencetomydevelopmentasastudent,employee

andhumanbeing!

ThisgoesforDr.Spybrookaswell.Shehelpedkeepmesanethroughoutthe

dissertationprocess,whileIwasprobablydrivingherinsane.Herabilitytoturnthe

reviewsofmychaptersaroundonadimehelpedmegetthroughmydissertationmuch

morequicklythanIcouldhaveeverhopedfor.Also,sheisanoutstandingeducatorand

mentorandoverallamazingperson.Dr.Corynalsohasanoutstandingabilityto

educateandmentor,Ilearnedsomuchfromtheclasseshetaught.

TomyfriendSarah,yourock!Thankyouforallofthewritingsessionsandmeals

thatwesharedtogether.Twofriendsbondedforlifebygoingthroughdissertationat

thesametime.Youmadetheprocessmuchmorefunthangoingthroughitalone!A

specialthankyoutoJaclynGoodfellow,DirectorofGMEatGRMEP,andallthe

iii

Acknowledgements-Continued

programcoordinatorsandotherGRMEPstaffthatassistedwithmydatacollection

effortsandansweredmymanyquestions!Thanksforthesupportofmydaughter,

Ashley,andhusband,Henry,aswellastherestofmyfamily.Henry,thankyouforallof

yoursupport,encouragement,understandingandpatiencewhileImisseddinners,

familyfunctionsandotherimportanteventstowritepapers,studyandworkonmy

dissertation–Icouldn’thavedoneitwithoutyou!

TracyJ.Frieswyk

iv

TABLEOFCONTENTS

ACKNOWLEDGMENTS.............................................................................................. ii

LISTOFTABLES........................................................................................................ viii

LISTOFFIGURES...................................................................................................... ix

CHAPTER

I. INTRODUCTION............................................................................................ 1

StatementoftheProblem................................................................... 1

WhatisGME?...................................................................................... 2 2

GMETrainingSites.............................................................................. 3

GMEFunding....................................................................................... 4

GMEFundingfromMedicare..................................................... 4

GMEFundingfromMedicaid...................................................... 5

ImplicationsofCutbackstoGMEFunding.................................. 6

GMEGraduateRetentionandReturnonInvestment......................... 7

StudyPurpose...................................................................................... 9

StudyObjectives.................................................................................. 9

ContributiontoEvaluation,MeasurementandResearch................... 10

OrganizationofDissertation................................................................ 11

II. REVIEWOFLITERATURE............................................................................... 12

PhysicianRetention............................................................................. 12

v

TableofContents-Continued

CHAPTER

PhysicianRecruitment/RetentionintoPhysician/RuralAreas... 13

In-StatePhysicianRetentionData.............................................. 16

MethodologicalConsiderationsofthePublishedLiteratureon Retention............................................................................................. 21

AMAMasterfileStudiestoExamineRetention.......................... 21 RegressiontoExamineRetention............................................... 22

SurveyStudiestoExamineRetention......................................... 24

What’sMissingfromthePublishedLiterature?.................................. 25

ScoringSystemsDevelopedthroughMultivariateRegression............ 26

ChangingtheMethodologicalApproachtoExaminingGMEIn-State Retention............................................................................................. 28 ChapterSummary................................................................................ 28

III. RESEARCHDESIGN........................................................................................ 29

StudyProcedures................................................................................ 29

StudySampleDescription........................................................... 29

RationaleforInclusionofIndependentVariablesinthe RegressionModel....................................................................... 30 RationaleforExclusionofVariablesfromtheRegressionModel 31

DataCollection........................................................................... 32

DataPreparation........................................................................ 37

MissingData............................................................................... 39

vi


CHAPTER

LogisticRegressionAssumptions................................................ 41

AnalyticProcedures............................................................................. 50

LogisticRegression..................................................................... 50

Over-Fitting................................................................................. 52

Five-FoldCross-Validation................................................. 53

Bootstrap........................................................................... 54

Pilot-ScoringTool........................................................................ 55

ChapterSummary................................................................................ 59

IV. RESULTS....................................................................................................... 60

SummaryData..................................................................................... 60

LogisticRegression.............................................................................. 61

Five-FoldCross-Validation................................................................... 66

Bootstrap............................................................................................. 68

Pilot-ScoringTool................................................................................ 68

ChapterSummary................................................................................ 75

V. DISCUSSION.................................................................................................. 76

StudyPurpose..................................................................................... 76

ResearchObjectiveOne............................................................. 77 77

TheMichiganConnection.................................................. 77

vii


CHAPTER

FinalGMETrainingLocation.............................................. 78

PrimaryCareandIn-StateRetention................................. 79

MarriageandIn-StateRetention....................................... 80

VariablesNotintheFinalRegressionModel..................... 81

ModelPerformance........................................................... 83

ResearchObjectiveTwo............................................................. 83 83

ContributiontoEvaluation,MeasurementandResearch................... 84

StudyLimitations................................................................................. 85

FutureResearch.................................................................................. 87

Conclusion.......................................................................................... 89

REFERENCES............................................................................................................. 90APPENDICES

A. Pilot-ScoringTool......................................................................................... 100

B. HumanSubjectsInstitutionalReviewBoardLetter...................................... 102

viii

LISTOFTABLES

3.1 Datasources................................................................................................. 34

3.2 Codingforregressionvariables.................................................................... 38

3.3 Percentagecompletedataforeachstudyvariable...................................... 39

3.4 Multicollinearityassessment........................................................................ 44

3.5 Multicollinearityreassessment.................................................................... 45

3.6 Modelstatisticscomparisonwithandwithoutoutliers............................... 47

3.7 Regressionmodelstatisticswithoutliers..................................................... 48

3.8 Regressionmodelstatisticswithoutoutliers............................................... 49

3.9 Methodologyforassigningscorestoregressionvariables.......................... 56

3.10 Cut-pointsderivedfromanROCanalysis..................................................... 58

4.1 Summarydataforthesample...................................................................... 61

4.2 Resultsofthebestsubsetslogisticregressionanalysis................................ 63

4.3 Summarydataforsampleincludedinthefinalmodel................................ 64

4.4 Finalregressionmodelstatistics.................................................................. 65

4.5 Five-foldcross-validationRMSEcomparison............................................... 67

4.6 Scorecalculationofpredictorvariablesinthefinalregressionmodel........ 71

4.7 Sensitivityandspecificityforscorecut-points............................................. 74

ix

LISTOFFIGURES

3.1 DBETAplot.................................................................................................... 46

3.2 DBETAplotafterremovingoutliers.............................................................. 50

4.1 ROCforfinalmodel...................................................................................... 66

4.2 ROCscorecut-points.................................................................................... 73

1

CHAPTERI

INTRODUCTION

StatementoftheProblem

Oneofthemajordecisionsinaphysician’slifeastheyapproachtheendof

graduatemedicaleducation(GME)trainingiswhattodonext.Whileasmallminorityof

doctorsgointoindustryorpursuefulltimeresearch,thevastmajorityofphysicianswill

practicemedicine.Thenthequestionbecomes,wheretopractice?Thefederal

governmentandGMEsponsoringinstitutionsdevotesubstantialresourcestothe

traininganddevelopmentofmedicaldoctors.Inatimeofincreasedconcernsoveran

impendingshortageofphysiciansandthecostsassociatedwithresidencytraining,there

arekeyincentivestoidentifyingthoseresidentswhoaremostlikelytopractice

medicineinthestateinwhichtheytrained(i.e.,in-stateretention).Thus,GMEtraining

programsareinterestedinlearningmoreaboutthefactorsthatinfluencein-state

practicelocationdecisions,aswellashowtoidentifygraduatesthatarelikelyto

practicein-state.

Thisdissertationutilizedtechniquesfromthefieldofresearch,logisticregression

withcross-validation,toexaminethedemographicandeducationalcharacteristicsthat

mayinfluencethedecisiontopracticemedicineinMichigan.Afterthepredictorsfrom

theregressionwereidentified,apilot-scoringtooltoaidintheevaluationofwhether

graduateswerelikelytopracticewithinthestateoftheirGMEtrainingwasdeveloped.

Astherearecurrentlynotoolslikethisavailable,aninstrumentlikethisisavaluable

2

contributiontothefieldofevaluation.ItalsocontributestothefieldofGMEasitwould

allowsponsoringinstitutionstoidentifywhetherupcominggraduatesarelikelyto

practicemedicineinthesamestateinwhichtheytrained.Thistoolcouldbeusedin

targetedrecruitmenteffortsoftheseidentifiedindividuals.

WhatisGME?

Theeducationaljourneytobecomingaphysicianspansmanyyears.Itbegins

withanundergraduatedegreefollowedbyfouryearsofmedicalschoolor

undergraduatemedicaleducation(UGME)andthenanotherthreeto11yearsofGME.1

Thistypeofeducationinvolvescaringforpatients,attendingeducationalconferences

anddidacticsessions,learningnewskillsandtechniques,participatinginscholarly

activities(e.g.,research)andworkingwithotherhealthprofessionslearners.GME

deliversthetrainingnecessaryforphysicianstobecomelicensedtopracticemedicine,

aswellastobecomeboard-certifiedintheirspecialty.GMEtrainingprograms

throughouttheUSareabsolutelyessential,astheyaretheprimarysourceinthis

countryforthephysicianworkforce.2

ThelengthofGMEtrainingisbaseduponthespecialtythephysicianispursuing.1

Forexample,aninternalmedicineresidencyprogramlaststhreeyears,whileageneral

surgeryprogramrequiresafiveyearcommitment.Sub-specialtytraining(e.g.,vascular

surgery,pediatrichematology/oncology),intheformoffellowshipprograms,canadd

anotheronetothreeyearsofadditionalGMEtraining.Asafellow,thephysician’srole

isbasicallythatofanattendingphysician(i.e.,aboardcertifiedphysicianwhoisin

3

practiceandseeingpatients)intheareaofwhateverspecialtyinwhichtheycompleted

residency,whilelearningamorespecificsub-setofknowledgewithinthatspecialty.

GMETrainingSites

ThebulkofGMEtrainingtakesplaceinteachinghospitalsandambulatory

officesunderthesupervisionofpracticingfacultyphysicians.Teachinghospitalsmake

upapproximately6%ofallofthehospitals(n=5,686)withintheUS.2-4Michiganalone

has44GMEsponsoringinstitutions(52teachinghospitals),ranking6thinthenation,

supportingjustunder5,000medicalresidentsperyear.5,6,7

Teachinghospitalsplayapivotalroleinsupportingphysiciantrainingby

providingclinicalexperiencestolearners,aswellasopportunitiesforscholarlyactivity

intheformofresearch,patientsafetyandqualityimprovementprojects.1,8Inaddition

toprovidingalargeamountofcomplexandacutepatientcare,onefifthormoreofall

ofthecarewhichtakesplaceinUShospitalsoccursatteachinghospitals.1,2

Thesetrainingsitesprovidemanyadvancedserviceswiththelatesttechnology

thatnon-teachinghospitalscannotprovide.2,4Thisincludesburnunits,pediatricand

neonatalintensivecareunits,transplantservicesandcardiacsurgery.2GMEtraining

sitesnotonlyprovidemanybenefitsintheshort-termintheformofofferingmore

extensiveandadvancedservices,theyarealsothesourceofAmerica’sphysiciansfor

thefuture.

However,providingtheseresourcesdoesnotcomewithoutaprice.Residents

andfellowsreceivesalariesandbenefitsduringtheirtrainingperiod.Factoringinthe

indirectcoststotheteachinghospitals,whichareassociatedwithresidencytraining

4

(e.g.,moretestsordered,longerpatientlengthofstay),theaveragecostofGME

traininghasbeenreportedtoaveragebetween$152,000to$239,000perresidentor

fellowperyear.1,9

GMEFunding

GMEisfundedbymultiplesources,includinggovernmentsources,aswellas

privatesources.Thetwomaingovernmentsourcesincludefederalfundingintheform

ofMedicareandMedicaid.1,4,10,11OthersourcesincludeVeteransAffairs,Health

ResourcesandServicesAdministrationandtheDepartmentofDefense.10,11State

governmentsarealsoabletoprovideadditionalsourcesoffundingtoGME.Private

fundingcancomefromtheteachinghospitalsthemselves,philanthropy,private

insurers,universitiesandfacultyphysicianpracticegroups.1,11,12

GMEFundingfromMedicare

ThemainsourceoffundingcomesfromMedicarepaymentsfromthefederal

government,whichaccountforover$9billionofthefundsprovidedforGMEper

year.4,10,11TherearetwotypesofMedicarepaymentsmadetoteachinghospitals,

directGME(DGME)andindirectGME(IGME).1,11DGMEpaymentscoverresident

salariesandteachingfacultytime.Sinceteachinghospitalsarepaidatthesame

dischargerateasnon-teachinghospitals,andtheseratesareusuallylessthanthecost

oftheserviceprovided,congressproposedIGMEpaymentstocovertheincreased

expensesrelatedtotheoperationalcostsoftheGMEprograms(e.g.,moretests

ordered,longerpatientstays,advancedservicesoffered,complexpatientpopulation).1

5

PressuresonGMEfundinghavebeenmountingoverthelast20years.In1997,

theBalancedBudgetActwasintroducedandpassedbyCongress.1,8,11Itincludeda

provisiontoreducetheamountoffederalsupportforGMEfundingbycappingthe

numberofresidencyslotsthatcanbesupportedbyMedicare.Inmorerecentyears

therehavebeenseveralbillsintroducedintheUSCongresstoreduceGMEpayments.

Forexample,aproposalofa50%reductioninGMEbytheJointSelectioncommitteeon

DeficitReductionwasaddedtotheBudgetControlActof2011,however,thislanguage

waslateromitted.11

GMEFundingfromMedicaid

MedicaidisthesecondlargestsourceofGMEfunding,providingover$3.78

billionperyear.10,11StategovernmentscanuseMedicaiddollarstosupportGME

programs,althoughtherearenofederalregulationsonhowastatechoosestodisburse

fundingoriftheychoosetodisbursefundstoGMEatall.6,11Moststateshave

historicallyprovidedGMEfundingthroughMedicaid.TherangeofstateMedicaid

supportisvariable.In2012,Alaskaspent$375,000andNewYorkspent$1.8billion,

whileeightstatesusednoMedicaidfundstosupportGME.6Thislackofregulation

leadstowidevariabilityinhowMedicaidfundsareused.

Inmorerecentyears,somestateshavereducedand/orlimitedthenumberof

MedicaiddollarsallottedforGME,whileothershaveeliminatedthistypeoffinancial

supportalltogether.Forexample,MichiganreducedMedicaidfundingtoGMEby3.6%

from2009to2012.6,13Michiganspent$163millioninMedicaiddollarsonGMEfunding

during2012,averaging3.1milliondollarsperteachinghospital.6Morerecently,a$57

6

millioncuttoMedicaidfundingforGMEwasproposedforthe2016budget.14However,

thecurrentfundingstructurewasmaintainedforthe2015-16budgetyear.15

ImplicationsofCutbackstoGMEFunding

MajorcutstoGMEfundingmayforcesometeachinginstitutionstoreduceorcut

GMEprogramsand/oreliminateservicesthatareunavailableelsewhereinthe

community.1,4,16Thesetypesofreductions,inanerawhenAmericansarelivinglonger

andarerequiringmorecomplexcare,mayadverselyaffectthequalityandavailabilityof

medicalcareintheUS.Theagegroupwiththegreatesthealthcareneedsarethose

>65,andtheCensusBureauprojectsa36%increaseinthisgroupby2020duetothe

agingbabyboomers.4,16,17

Tofurtheraddtothecomplexityofthesituation,theAffordableCareAct(ACA)

hasintroducedapprehensionoverreducedprofitmarginsforhospitals.18Inaddition,

demandforhealthcareisalsoprojectedtoincreasewiththeimplementationofthe

ACA.4Thosewhodidnothavehealthinsurancebeforewillbeabletotakeadvantageof

servicesthatwerenotreadilyavailabletothemduetolackofcoverage.Finally,theACA

hasfueledconcernsovertheprojectedphysicianshortage,estimatedtobe91,500

nationwideby2020,withashortfallof6,000projectedforthestateofMichigan.2,17,19

Thestrainonavailablehealthrelatedservicesprovidedbyphysiciansisfurther

complicatedbythefederalcap,whichlimitsthenumberofGMEtrainingpositions

fundedbyMedicare.ThisbarriertoGMEtraininghascreatedabottleneckrelatedto

thenumberofphysiciansenteringtheUSworkforceatanygiventime.Anaging

physicianworkforce,ofwhich1/3areprojectedtoretireby2020,willfurtherwidenthe

7

gapbetweenhealthcareneedsandthenumberofphysiciansavailable.4,16,17Inan

efforttoclosethegap,medicalschoolshavebeenpoppingupallovertheUS,including

threeinMichiganwithinthepastthreeyears.8However,theincreaseinmedicalschool

graduateswilldonothingtooffsettheburdenifthereisnoincreaseingovernment

fundingtosupportadditionalGMEtrainingslots,aswellasmaintainingthecurrent

fundingstructureofGME.

GMEGraduateRetentionandReturnonInvestment

ThereisaneedformoretransparencyandaccountabilityfromtheGME

sponsoringinstitutionsthatreceivethefundsfortheirtrainingprograms.11Theissue

withthecurrentmodelisthatGMEfundsreceivedthroughMedicare,Medicaidand

othersourcesgodirectlyintotherevenuestreamoftheteachinghospitals.11This

practicemakesitdifficulttogetanaccuratepictureofhowthefundsforGMEare

actuallybeingspentbythesetrainingsites.Thispracticehasresultedinalackof

accountabilitytoGMEfundingsourcesonthepartofGMEinstitutions.Ifthecurrent

fundingstructuresaretobemaintained,GMEsponsoringinstitutionsneedtofindways

todemonstratetofundingsourcesthevalueoftheirinvestment(i.e.,returnon

investment(ROI)).

SeveralrecentreportshavefocusedonhowtheGMEsystemiscontrolledand

funded,andhavealsoexaminedalternativefundingsourcesandwaystodemonstrate

ROIforinvestmentinGME.10,12,20,21Manyofthesereportssuggestthattrackinglocation

ofpracticepost-trainingcouldbeusedtodemonstrateROI.Locationofpracticeafter

graduationcouldbeintheformofratesofgraduateswhopracticeinhealthprofessions

8

shortageareas(HPSA)and/orin-stateretentionrates(i.e.,practicinginthesamestate

asGMEtraining),bothofwhichcoulddemonstrateROItoGMEfunders.Infact,

legislationinMassachusettsdirectedaspecialcommissionongraduatemedical

educationtolookattheeffectofGMEonfulfillingphysicianneedsinspecificspecialties,

aswellastodeterminedifferentwaystofundGME.20Thefinalreportrecommended

thatperformancebenchmarkstiedtoGMEfundingcouldbeintheformofin-state

retentionandtrainingphysiciansinspecialtiesthatarelinkedwithphysicianshortage

projections,justtonameafew.22

InastudytoinvestigatethepotentialexpansionofGMEinNorthwestIndiana,

TrippUmbachreportedthatphysiciansthatpracticeinthestateofGMEtraining

generateapproximately$1.5millionperyearineconomicbenefits.21IfGMEsponsoring

institutionscoulddemonstrateahighin-stateretention,thiscouldfurtherjustify

fundingforGME,especiallyintheformofMedicaiddollars,whichcomefromthestate.

Forexample,ifMichiganprovidesMedicaidfundstowardsGMEandlessthanhalfofthe

GMEtraineesenduppracticinginthestate,legislatorsmaynotrecognizethisasagood

ROI.

SinceMichiganisoneof22statesthathavelinkedthegoalofincreasingthe

physicianworkforcetoMedicaidGMEpayments6,theconceptofin-stateretention

couldbeusedasamarkerforjustificationoffunds.TheROIinthiscasewouldbe

closingthephysicianshortagegapleadingtoimprovedaccesstohealthcare,aswellas

theeconomicbenefittothestateandcommunitywherethephysicianisemployed.In

thisscenario,therewouldbeaneedforMichigan-basedGMEprogramstoexploreways

9

toidentifyresidents/fellowsearlyintrainingwhoarelikelytoenduppracticing

medicineinMichigan.Targetedrecruitmenteffortsaimedattheseindividualscould

potentiallyincreasein-stateretentionrates.

StudyPurpose

Inaneraoflimitedfinancialresources,thereisagrowingneedforGME

sponsoringinstitutionstodemonstratetofundingsourcesthevalueoftheirinvestment.

Onestrategyfordongthisistoincreasetheirin-stateretentionrates.Thiswouldserve

adualpurposeofjustifyingtheexpenseofthephysicianeducation,aswellasaddress

concernsoverphysicianshortageissuesinthestate.Thepurposeofthisstudyisto

examineindividuallevelcharacteristicsofgraduatesfromaMichigan-basedGME

sponsoringinstitutiontodevelopatooltoidentifygraduatesthatarelikelytopractice

medicineinthestateofMichigan.

StudyObjectives

Morespecificallythefirstobjectiveofthestudyis:

1. Touselogisticregressionwithcross-validationtoexaminetheindividual

characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-

basedGMEsponsoringinstitutionpracticemedicineinMichigan.

Further,usingtheidentifiedpredictorsfromthepreviousanalysistocreatea

mechanismtohelppinpointgraduatesthatarelikelytopracticeinMichiganpost-

trainingwouldbeusefultoMichigan-basedGMEsponsoringinstitutionsand/or

physicianrecruitersinMichigan.AtoollikethiscouldalsoproveusefultoGME

10

sponsoringinstitutionsinotherstates.Thesecondobjectiveofthestudywilladdress

thisconcept.

2.Createascoringtoolbasedonthelogisticregressionwithcross-validationto

categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganornot

likelytopracticeinMichigan.

ContributiontoEvaluation,MeasurementandResearch

EmpiricalevidencefromasingleMichigan-basedGMEsponsoringinstitutionwas

usedtoexaminevariablesrelatedtowhetherornotagraduatepracticedmedicinein

Michigan.Five-foldcross-validationandbootstraptechniqueswerecombinedwith

logisticregression,techniquesfromthefieldofresearch,tobuildandtestthemodel

priortocreationofthepilot-scoringtool.Variablesthatwereidentifiedaspredictorsof

practicelocationwereweightedandusedtocreateapilot-scoringtoolthatcouldbe

usedtoevaluatewhetherornotGMEgraduatesarelikelytopracticemedicinein

Michigan.

Thisdatadrivenapproachtodevelopingapilot-scoringtoolisacontributionto

evaluation.ThetoolhasthepotentialtoadvancetheevaluativecapacityofMichigan-

basedGMEinstitutionstoassessthelikelihoodaGMEgraduatewouldpracticein

Michigan.Thistoolcouldbeusedtoproducealistingofidentifiedgraduateslikelyto

practiceinMichiganforhospitalsandotherphysicianrecruitersinMichigantousefor

targetedrecruitmentoftheseindividuals.

11

OrganizationofDissertation

Therearefivechaptersincludedinthisdissertation.Thematerialinchapterone

introducedthereadertotheconceptofGMEanditsimportance.HowGMEisfunded

andtheissuessurroundingfundingarealsocovered.Theconceptofin-stateretention

isdescribedaswell.Lastly,thestatementoftheproblem,studypurposeandobjectives

aredescribed.

DataavailableintheliteratureongraduateretentionwithintheGMEtraining

statearediscussedinchaptertwo.Themethodologiesofthisstudyaredetailedin

chapterthree,includingthestudysample,datacollection,assumptionsoflogistic

regressionandotheranalyticalmethods.Theresultsofthestudyarereportedin

chapterfour.Chapterfiveincludesadiscussionofthestudyfindings,implicationsofthe

study,recommendationsforfutureresearchandconclusions.

12

CHAPTERII

REVIEWOFLITERATURE

ThischapterincludesareviewoftheavailableliteratureonGMEretention.Also

describedarelimitationsofthemethodologiesusedtoobtain,analyzeandreportthe

datarelatingtoretention.Studiesthathavebeenusedtodesignscoringsystemsare

alsodiscussed.Lastly,adifferentmethodologicalapproachtoexaminingin-state

retentionisdescribed.

PhysicianRetention

Physicianretentionisanimportantpartofthediscussionrelatedtofundingfor

GMEtraining.IthasbeenstatedthatGMEtrainingprogramsthatproducephysicians

thatreturnorstaytopracticewithinthestateofGMEtrainingiseconomicallybeneficial

andincreasestheaccessofthepublictohealthcare.2,16,21Retentionofphysiciansinthe

stateinwhichtheycompletedGMEtraining,aswellasinphysicianshortage/ruralareas,

havebeenproposedasperformancemeasurestodemonstrateROI.22

Predictorsofretentionofphysiciansinphysicianshortage/ruralareasandin-

stateretentionhavebeenstudied.First,theliteraturerelatedtopredictorsofretention

inphysicianshortage/ruralareasisreviewed.Next,studiesfocusingonexamining

factorsrelatedtoretainingphysicianswithinthestateofGMEtrainingarediscussed,as

wellasadditionalpublishedreportsthatincludedataonin-stateretentionratesof

physicianswhopracticemedicineinthestateinwhichtheyreceivedtheirGMEtraining.

13

PhysicianRecruitment/RetentionintoPhysicianShortage/RuralAreas

Withtheimpendingphysicianshortageissues,accesstohealthcare,especiallyin

ruralsettings,isanimportantcomponentoftheretentionconversation.Manystudies

havefocusedonexaminingfactorsrelatedtoretainingphysicianstopracticein

physicianshortage/ruralareas.23-29Duetothelackofresourcesandhealthdisparities

(e.g.,older,sicker,poorer)associatedwithruralsettings,recruitmentandretentionof

primarycarephysiciansinruralareashasbeenatopicofinvestigationoverthepast100

years.23

Morerecently,studiestoinvestigatefactorsrelatedtochoosingprimarycare,as

wellaspracticinginruralandunderservedcommunities,havebeenconducted.23-29

Thesereportsincludetheuseofhistoricaldata,aswellasdatafromsurveys,toexamine

thefactorsthatinfluencethechoicetopracticemedicineinruralareas.23-26Other

studieshaveincludedvariousmethodstoobtaindata(e.g.,semi-structuredinterviews,

documentreview,observations)inordertoinvestigatespousalandcommunityroles,as

wellasotherfactorsrelatedtorecruitmentandretentionintoaruralpracticesetting.27-

29Thesestudiesarediscussedinthefollowingparagraphs.

Rabinowitzetal.conductedaretrospectivestudyofJeffersonMedicalCollege

(JMC)graduatesfrom1978to1993,includingasubsetofgraduatesthatparticipatedin

aPhysicianShortageAreaProgram.23Onepurposeofthestudywastodetermine

predictorsofruralprimarycarephysicianretention.Thisstudyutilizeddatapreviously

collectedforalongitudinalstudytrackingJMCgraduates,aswellasfromtheAmerican

MedicalAssociation(AMA)PhysicianMasterfile.

14

Therewere19studyvariablesinvestigatedbytheauthorsfromthefollowing

areas:demographic,pre-medical,careerplansinmedicalschool,medicalschool

programs/curriculaandeconomicissues.Univariateanalysesforeachofthe19

variableswereperformedtoexaminetheirrelationtoruralpractice.Significantfindings

fromtheunivariateanalyseswereenteredintoamultivariatemodeltoassesstheir

predictiveabilityrelatedtopracticinginaruralsetting.Significantpredictorsfromthe

multivariateregressionanalysisincludedplanstopracticeinfamilymedicineduringthe

freshmanyearofmedicalschool,participationinaphysicianshortageprogram,

NationalHealthServiceCorps(NHSC)scholarshiprecipient,ruralpreceptorship

experienceandmalegender.Theauthorsalsofoundthatindividualsthathadarural

background,combinedwithaplantopracticeinfamilymedicine(developedduringthe

freshmanyearofmedicalschool),weremorethantwiceaslikelytopracticeinarural

settingthanindividualswithonlyoneofthesevariables.

Anotherstudy,conductedbytheRobertGrahamCenterandfundedbythe

JosiahMacyJr.Foundation,examinedfactorsthatinfluencemedicalstudentand

residentchoicesaboutmedicalspecialties(primarycarevsnon-primarycare)and

locationofpractice(ruralandunderservedpopulations).26Specifically,therolesthat

studentdebt,scholarshipandloanprograms,typeofschool,curriculum,institutional

culture,andpotentialincomehaveonmedicalstudents’specialtychoices,aswellas

decisionstocareforunderservedpopulations.

Datawerecompiledfromsurveyresultsfrommedicalstudentscompletedat

graduationandhistoricaldatarelatedtoreceiptofTitleVIIfundsduringtraining,the

15

AMAPhysicianMasterfileandparticipationintheNHSC,aswellasfromdatarelatedto

currentpracticespecialtyandlocation,andserviceinRuralandFederallyQualified

HealthCenters.Datafromatotalof322,131individualswereavailableforanalysis.

Stepwiselogisticregressionanalyseswereusedtoexamineavarietyof

independentvariablesandtheirrelationtothedependentvariablesofpracticingina

ruralsettingandpracticinginaphysicianshortageorunderservedarea.Giventhelarge

samplesize,itwasclearthatmost,ifnotall,oftheindependentvariableswouldbe

statisticallysignificant,sotheinvestigatorsputtheaddedrestrictionthatthemost

importantvariableswouldhaveanoddsratio(OR)>2or<0.5.Forexample,intheir

analysisoflikelihoodtopracticeinaruralarea,20ofthe21independentvariableswere

statisticallysignificant(p<0.05),whileonlyfourofthevariableswereconsideredtobe

importantpredictors.

Chanetal.alsoexaminedvariablesrelatedtothedecisiontopracticerural

medicine.Asurveywasconductedofruralfamilyphysicianswhograduatedfrom

Canadianmedicalschoolsbetween1991-2000.24Atotalof382/651physicians(58.7%)

completedthequestionnaire.Comparisonsweremadebetweenphysicianswithan

urbanupbringingandaruralupbringingwithregardtoratingimportantfactorsthat

influencedthedecisiontopracticeruralmedicine,usingthechi-squaretest.Statistically

significantdifferenceswereseenwithregardtoeducationaltraining(ruralexposure

duringmedicalschool/residency),ruralexposuregrowingupandotherfactorsincluding

proximitytofamily,spouse/partnerinfluence,desiretopracticewhereneedthe

greatest.

16

Asurveytoassessaspectsoftheenvironmentthatcouldbeusedtopredict

retentiontouseintherecruitmentofphysicianstotheHawaiianIslandswasconducted

byPellegrin.25Hawaiisuffersfromphysicianshortageissues,thereforetheauthorfeltit

wasimportanttostudyfactorsrelatedtorecruitmentandretentionofphysicianstothe

island.Therewere127participantsinthesurvey.Predictorsofretentionincluded

accesstogoodK-12schoolsystems,financialsustainability,communitysupportand

professionalopportunities.

Otherstudieshavetakenamorequalitativeapproachtotheissueofrecruitment

andretentionofphysiciansintoruralmedicine.27-29OnestudybyHancocketal.used

datacollectedfrom22semi-structuredinterviewstocreateamodelconsistingof“four

mainpathwaystosuccessfulandfulfillingruralpractice:familiarity,community,sense

ofplace,andself-actualization”(pg.1372).27These22interviewswereconductedwith

mostlywhitemales,withanaverageageof55years.Mayoetal.conducted13

interviewsofspousesofruralphysiciansto“gainabetterunderstandingofspousal

concernsandexperienceregardingruralliving”(pg.272).28Cameronetal.usedacase

studydesigntostudyfourruralcommunitiesto“exploretheprofessional,personaland

communitydomainsofphysicianretention”(pg.47).29

In-StatePhysicianRetentionData

Overtheyears,in-stateretentionhasbeenexaminedinmanydifferentways.

Theseincludestudiesthatusedlogisticregressiontoexaminevariablesrelatedtoin-

stateretention,aswellasstudiesthatreportin-stateretentionofspecificprogram(e.g.,

familymedicine)graduatesorsummarydatarelatedtopercentagesofgraduateswho

17

practiceinthestatein-whichtheyunderwentGMEtraining.9,30-35Thesestudiesare

discussedinmoredetailinthefollowingparagraphs.

Astudyin1986byBurfieldetal.usingdatafromtheAMAPhysicianMasterfile

examinedtherelationshipbetweenmedicaltrainingandpracticelocation.30The

authorsincludedpersonal(e.g.,age,timesincegraduation)andprofessional(e.g.,

practicespecialty)characteristicsofphysicians,aswellasmedicalschool(e.g.,USvs

non-US,reputation)andstate(e.g.,population,percapitaincome)characteristicsfor

activephysiciansin1982intheirstudy.Summarydatawerereported.When

specificallylookingatin-stateretentionofGMEgraduates,theresultsshowedthata

higherpercentageofgeneral/familypractitioners,aswellasfemalephysicianspracticed

inthestateinwhichtheycompletedGMEtraining.Overall,51.1%ofphysicianswere

practicinginthestateinwhichtheyreceivedGMEtrainingand29.7%completedboth

undergraduatemedicaleducation(UGME)andGMEinthestate.

In1995,Seiferetal.publishedastudythatalsousedtheAMAPhysician

MasterfiletoassesstherelationshipbetweenGMEandpracticelocation,specifically

locationinthestateinwhichGMEtrainingoccurred.31Theinvestigatorstookarandom

samplefromthe1993editionoftheAMAPhysicianMasterfileandcoupleditwiththe

AmericanOsteopathicAssociation(AOA)PhysicianMasterfilefortheanalysis.Summary

datawerereported.Alogisticregressionwasalsoperformedtoexaminethe

relationshipbetweenphysician/statecharacteristicsandpracticelocationinthestateof

GMEtraining.Theauthorsreportedthatin1993,51%ofphysicianswerepracticingin

thestateinwhichtheytrained.Datawerealsoreportedforeachstate.Michigan

18

showedanin-stateretentionrateof48%.Thelogisticregressionanalysisrevealedthat

gender(female),UGMEinthesamestateasGMEtraining,graduatingfromaUSmedical

school(inverserelationship),specialty(generalist),professionalactivity(teaching,

research,administration),boardcertification(inverserelationship),afederalemployee

(inverserelationship),numberofresidentphysiciansinthestate(inverserelationship),

numberofnon-residentphysiciansinthestateandpercentageofthepopulationbeing

urbanwerepredictiveofin-stateretention,whileage,professionaldegree(MDvsDO)

andstateincomelevelwerenot.

AnotherstudyusingdatafromtheAMAPhysicianMasterfileanalyzedthe

effectsofbirthlocation,medicaleducationandcompletionofGMEtrainingonpractice

locationforfamilymedicineresidents.32Thestudysampleincludedphysiciansthat

completedGMEbetween1997and2003thatwereborninVirginia,attendedmedical

schoolinVirginiaorcompletedGMEtraininginVirginia.Sevendifferentvariable

combinationswereanalyzedtoassessthelikelihoodofpracticinginVirginia.Atotalof

806physiciansmettheinclusioncriteria.Theresultsforeachvariableonitsownwere

reported.Only6%ofphysicianswhowereborninVirginiabutreceivedUGMEandGME

traininginanotherstatewerepracticinginVirginia.Therewere17%ofphysicians

practicinginVirginiawhohadattendedmedicalschoolthere,butwerenotborninthe

statenordidtheyundergoGMEinthestate.DataforphysicianswhoreceivedGME

traininginthestate,butwerenotborninVirginiaanddidnotattendmedicalschoolin

thestatewerereportedtobe49%.Combinationsofvariablesrevealedthat74%of

physicianspracticinginVirginiawereborninthestateandcompletedGMEinthestate,

19

while82%wenttomedicalschoolinthestateandcompletedGMEtraininginthestate.

Finally,81%ofthephysicianswhohadthecombinationofborninthestate,attended

medicalschoolinthestateandcompletedGMEtraininginthestatewereidentifiedas

practicingmedicineinVirginia.

Faganetal.alsoutilizedtheAMAPhysicianMasterfiletoassesstherelationship

betweenthelocationofpracticeandwherethephysicianreceivedtheirfamilymedicine

training.33Theinvestigatorsexaminedpracticelocationwithin5,25,50,75and100

milesofGMEtraining,aswellasthepercentageoffamilymedicinegraduatespracticing

inthestateinwhichtheytrained.Atotalof64,972physicianswereincludedinthe

study.Theresultsshowedthat,nationally,57%ofthefamilymedicinegraduates

practicedinthesamestateastheirGMEtrainingsiteand55%within100milesofthe

trainingsite.Individualrateswerereportedforeachstateaswell.

Anothersourceofin-stateretentionratesthatusesdatafromtheAMAPhysician

MasterfilecomesfromtheCenterforWorkforceStudies,whichpublishesabiennial

reportonGMEintheUS,physiciansupply,andmedicalschoolenrollment.5Includedin

thisreportaredataonratesofGMEgraduateswhopracticedwithinthestateinwhich

theyreceivedtheirGMEtrainingforactivephysicians(e.g.,administration,direct

patientcare,medicalresearch,medicalteaching).

Themainsourcesofdataforthe2013reportarelistedastheAMAMasterfile

(December2012file),populationestimatesfromtheUSCensusBureau,Associationof

AmericanMedicalColleges(AAMC)StudentRecordSystem,AOAandNationalGME

Census(conductedbytheAAMCandAMA).

20

Theworkbookreportsdataforeachstate,whicharethenre-reportedinvarious

reportsandbriefsbyindividualhealthhealthcareworkforcecommittees.Forexample,

theOfficeforHealthcareWorkforceAnalysisandPlanning,basedinSouthCarolina,

publishedabriefin2013onretentionofphysicianswhotrainedinthestate.34Thedata

inthisbriefwerebasedonthe2011StatePhysicianWorkforceDataBookpublishedby

theCenterforWorkforceStudies.Itwasreportedthat,oftheactivephysiciansin2010,

2,389attendedmedicalschoolandresidencytraininginSouthCarolina.Ofthose

physicians,1,829(76.6%)werepracticingmedicineinSouthCarolina.

In-stateretentiondatawerealsopublishedfortheGeorgiaStatewideArea

HealthEducationCenter(AHEC)Networkinrelationtoincreasingthenumberof

primarycareGMEslotsinGeorgia.Datareportedindicatedthatabout74%of

physicianswhograduatedfromaGeorgia-basedhighschoolandGMEtrainingprogram

practiceinGeorgia.Further,over80%ofphysicianswhograduatedfromaGeorgia-

basedhighschool,medicalschoolandGMEprogramremainedinthestatetopractice.

AnotherreportbytheCenterforWorkforceStudiesfocusedonGMEgraduates

inNewYork.35Forthepast15yearsduringtheSpringpriortograduation,anannual

exitsurveyhasbeengiventoresidentsandfellowscompletingtheirtrainingintheState

ofNewYork.Thesurveyresultsreportinformationthatincludespost-trainingplans,

specialtyneedsandjobprospectsinthestateofNewYork.

ThesurveyisdistributedtoGMEsponsoringinstitutionslocatedinNewYork.Itis

assumedthattheGMEinstitutionwillthensendthesurveytograduatingresidentsand

fellows.Theoverallresponserateforthemostrecentlypublishedyear(2014)was56%

21

(2,951/5,275).Acrossthepast15years,theoverallresponseratehasbeen61%.Data

collectedinthesurveyspanfourcategories:demographic/backgroundcharacteristics,

post-graduationplans,futureemploymentplans,jobsearch/jobmarketexperiences.

Resultsfromthemostrecentsurveyshowedthat,ofthosethatresponded,just

overhalfofthegraduateswereplanningtoenterthepatientcaresettingupon

graduation,andofthose,83%hadconfirmedpracticeplans.Ofthegraduateswhohad

confirmedpracticeplans,justunderhalf(45%)wereplanningonpracticinginNewYork.

Ahighpercentage(80%)oftherespondentswhowerestayinginNewYorkhadstrong

tiestothestate,attendingbothhighschoolandmedicalschoolinNewYork.

RespondentswhowereleavingNewYorktopracticegavereasonssuchas:tobecloser

tofamily(27%),betterjobs(14%)andsalary(9%),nointentiontostayaftertraining

(6%),climate/weather(2%)andtaxes(1%).

MethodologicalConsiderationsofthePublishedLiteratureonRetention

AMAPhysicianMasterfileStudiestoExamineRetention

Whilethedatafromthepublishedliteratureprovideusefulinformationon

retention,themajorityofthesestudiesareprimarilybasedupondatafromthesame

source,theAMAPhysicianMasterfile.23,26,30-35Thisdatasourcehasbeenmaintained

since1906andisthesourceofdataforover1.4millionphysicians,residentsand

medicalstudentsintheUS.TheAMAhasadivision(DivisionofSurveysandData

Resources)thatmanagesthedatafile.36,37

22

DatafortheAMAPhysicianMasterfilearesourcedfrommedicalschools,GME

programs,statelicensingagencies,disciplinaryactionreportsfrommedicaland

osteopathicboards,NationalBoardofMedicalExaminersreports,Educational

CommissionforForeignMedicalGraduates,AmericanBoardofMedicalSpecialties,and

FederalDrugEnforcementAdministrationregistration.ArequesttoAMAmembersto

updatetheirprofilesismadeeverythreeyears.Othernon-memberprofilesare

updatedusingtheothersourceslisted.

Oneconcernwiththedatafromthesourceistheaccuracyatanygivenpointin

time.OnestudyshowedthattheAMAPhysicianMasterfiledataoverestimatedthe

numberofphysicianspracticinginsmallruralcommunities.38Thethree-yearlagtime

betweenupdates,aswellastherelianceofphysicianswhobelongtotheassociationto

updatetheirinformationmayleadtoinaccuraciesinthedata.Forexample,one

graduateincludedinthedataforthisdissertationwasstillshowingaGrandRapids,

Michiganlocation(leftoverfromresidency)usingthisresource,whenitisknownthat

thisindividualhadnotpracticedinMichigansincegraduation.

RegressiontoExamineRetention

Asmallnumberofstudiesusedregressiontoassesspredictorsofpractice

locationwithinphysicianshortage/ruralareas,aswellaswithinthestateofGME

training.23,26,31Thesestudiesprovideusefulinsightastofactorsthatinfluencepractice

location.However,therearelimitationstosomeofthemethodologyusedtoexamine

retention.Therearealsodifferencesbetweenthefactorsusedtoexamineretentionas

23

definedforthisdissertationandthoseexaminedinotherstudies.Theselimitationsand

differencesarediscussedinthefollowingparagraphs.

ThestudyofRabinowitzetal.usedmultivariateanalysistodetermineoptimal

predictivefactorsforruralsupplyandretentionofphysicians.23Oneconcernisthe

methodbywhichthevariableswereselectedforinclusioninthemodel.The

investigatorssettheirsignificancelevelcut-ofat0.05fortheunivariateanalysesas

beingthecriterionforinclusioninthemultivariateanalysis.However,thislowofa

significancelevelcanleadtoincorrectlyrejectingvariablesthatshouldbeincludedin

theregressionmodel.Sunetal.showedthatvariableselectionusingthismethodcan

leadtoincorrectlyrejectingvariablestoincludeinthemodel,whichmayoccurdueto

confoundingwithothervariables.39Harrellhasalsoexpressedconcernswiththis

methodologyforvariableselection.40

Seiferetal.utilizedlogisticregressiontoexaminefactorsrelatedtoin-state

retentionofGMEgraduates,whichwasthesameoutcomevariableusedforthecurrent

study.31However,theyincludedfactorsrelatedtophysician/professionalcharacteristics

andstate-levelcharacteristicsintheanalyses,manyofwhichweredifferentthanthe

factorsexaminedinthisdissertation.Morecurrently,theRobertGrahamCenterused

regressiontoexaminefactorsthatinfluencemedicalstudentandresidentchoices

relatedtopracticinginruralandunderservedareas,aswellaspracticespecialty

choices.26However,thismorerecentstudydiffersfromthecurrentdissertationasin-

stateretentionratesofGMEgraduateswasnotthefocus.

24

SurveyStudiestoExamineRetention

Multiplefactorsrelatedtoerror(e.g.,coverage,sampling,non-response,

measurement)areassociatedwithsurveystudies.41Errorrelatedtocoveragecanbe

introducediftheentiretargetpopulationwasnotincludedbasedonthemethodof

surveydissemination.Forexample,under-coveragecouldbeanissueiftheentire

targetpopulationdoesnothaveanopportunitytocompletethesurvey.Conversely,

over-coveragecanbeproblematicifduplicatesand/orthoseotherthanthetargeted

grouphaveanopportunitytocompletethesurvey.Non-responsebiascanbea

limitationtotheinterpretationofsurveyresultsifresponseratesarelow.

Measurementerrorcanalsobeintroducedthroughtheinstrumentitselfthroughpoorly

wordedquestionsandpoordesign/layout.

InthecaseoftheexitsurveyofNewYorkGMEgraduates,responsebiasmaybe

anissue.35Inthiscase,under-coverageisanissuesincethereisanassumptionmade

thatGMEsitesaresendingthesurveytotheirgraduates.Forthesurveystudy

conductedbyPellegrin,itisunclearwhetherunder-coverageorover-coveragemaybea

concern.25Forthisstudy,keyhealthcareleadersandotherphysicianswereaskedto

distributealinktothesurveyviaemailtocliniciansintheirassociatedinstitutions.It

wouldbeunclearfromthismethodofsurveydistributionwhetherornoteveryonein

thetargetpopulationreceivedthatemailornot.Thesearelimitationsofsurveystudies

thatrelyonothersfordistribution,however,sometimesthisistheonlyoptionof

disseminationavailabletoinvestigators.

25

Non-responsebiasmaybeanissueformanyofthesestudiessincetheir

responserateswerelessthan100%andrandomsamplingwasnotemployed.24,25,35

Pellegrinreportedthattheresponserateisunknownforherstudyduetothemethodof

distributionofthesurvey.25Chanetal.reportedaresponserateof59%.24However,

thereportedratemaybeproblematicsincetheinitialmailingincluded784physicians

and133returnedquestionnairesweredeemedtobeineligibleforvariousreasons.Itis

unknownhowmanyothersurveysweresenttoineligiblecandidates(over-coverage),so

thetrueresponseratecanneverbeknown.

OnelastlimitationoftheNewYorkGMEsurveyisthatthein-stateretention

dataareonlyreportedforthosewithconfirmedpracticeplansatthetimeofsurvey

dissemination.35Thereisnoconfirmationthatthosephysicianseverendedup

practicinginthestate.Further,thosewithoutconfirmedpracticeplanswhocompleted

thesurveymayhaveendeduppracticinginNewYork.Anothermissingcomponentof

thein-stateretentiondataarethepercentageofnon-responderswhostayedinNew

Yorktopracticemedicine.

What’sMissingfromthePublishedLiterature?

WhatseemstobelackingintheliteratureisthecontributionofspecificGME

trainingsitestoin-stateretentionrates.Further,amorecurrentlookatpredictors,

specificallydemographicandeducationalcharacteristicsofthephysician,ofin-state

retentionisneeded.Lastly,amethodtoidentifygraduateslikelytopracticewithinthe

stateofGMEtraining,suchasascoringtool,wouldbebeneficialtoGMEsponsoring

institutions,aswellashospitalandphysicianrecruiters.

26

ScoringSystemsDevelopedthroughMultivariateRegression

Manystudieshavebeendesignedtocreatescoringsystemstopredictspecific

outcomes.42-45Theabilitytopredictanoutcomeisusefulinmanyways.Inmedicine,

scoresdevelopedfrommultivariateanalysestopredictoneyearsurvivalinICUpatients

onamechanicalventorriskofheartdehydrationinchildrenwithdiarrheaaredesigned

toassistphysicianswithtreatmentdecisions.Ineducation,scorescanbeusedtoassist

withmakingdecisionsabouteducationalinterventionsdesignedtoretainstudents.

Methodsusedtodevelopthesescoresincludetheuseofmultivariatepredictormodels

coupledwithcross-validationtechniques.

Houghetal.developedasystemtopredictmortalityforadultICUpatientson

mechanicalventilatorsforprolongedperiodsoftime.42Themodelwasdevelopedusing

retrospectivedatacapturedonday14forpatientsreceivingmechanicalventilation,

from40USmedicalinstitutions.Astepwiselogisticregressionanalysiswasemployedto

developthemodel.Adevelopmentcohort(n=491)andvalidationcohort(n=245)were

usedtocreateandtestthemodel.Theinvestigatorsusedtheareaunderthecurve

(AUC)developedfromareceiveoperatorcharacteristic(ROC)analysisandHosmerand

Lemeshow’sgoodness-of-fitstatisticformodelevaluation.Priortodevelopingthe

scoresforthemodel,continuousvariableswereturnedintocategoricalvariablesand

anotherlogisticregressionwasperformedonthedevelopmentcohorttoobtaintheβ-

coefficientstouseinscoredevelopment.Scoreswererangedfromzerotofouror

greater.Kaplan-Meieranalyseswereperformedtoplotsurvivalforeachscorecategory

forboththedevelopmentandvalidationcohortstoassessperformance.

27

Zodpeyetal.createdarisk-scoringsystemtoassesschildrenwithdehydration

relatedtodiarrhea.43Amultipleregressionanalysiswasconductedusingdatafrom774

patients.Atotalof17hypothesizedriskfactorsweretestedinthemodel.Factorsin

thefinalmodelwereweightedandeachweightwasroundedtothenearestwhole

numberaftermultiplyingitby10.Scoresforeachofthepatientsinthesamplewere

thendetermined.Theinvestigatorsthencalculatedthesensitivity,specificity,and

predictiveaccuracy.Thebestcut-offscorewasdeterminedusinganROCcurveandthe

maximumCohen’skappavalueasdeterminedforeachofthetotalscores.

Imperialeetal.createdariskindexusingpointsdevelopedfromtheβ-

coefficientsderivedfromalogisticregressionanalysistodetectadvancedneoplasia.44

Therewere3,025individualsinthedevelopmentdatasetand1,475inthevalidation

dataset.Therewerefivefactorsinvestigatedinthemodelandincludedage,gender,

bodyfat,historyofsmokingcigarettesandfamilialhistoryofcoloncancerasthe

independentvariables,whileadvancedneoplasiawastheoutcomevariable.The

investigatorsusedanapproachdescribedbySullivanetal.whendevelopingtheirpoint

system.45Theβ-coefficientswereusedtoderivethepointsforeachvariable,which

werefurthercategorizedintoriskgroups(verylow,low,intermediate,high)oncethe

pointsweredetermined.Comparisonsofmodelparametersforthedevelopmentand

validationsetsweremadeusingtheriskandlikelihoodratios.

Thesestudiesallusedregressionanalysistocreatescoringtoolsforusein

predictionofriskofsomehealthrelatedissue.Scoringtoolswerecreatedusing

differentmethodologiesandanalyses.However,theultimategoalwasthesame,the

28

creationofatooltobeusedintheassessmentofpatients.Theanalytictechniques

fromthesestudieswereusedtomapoutanapproachtocreatingthepilot-scoringtool

forthisdissertation.

ChangingtheMethodologicalApproachtoExaminingGMEIn-StateRetention

Fromamethodologicalperspective,therearenopreviousreportsintheGME

literaturethathaveutilizedlogisticregressionwithcross-validationtoanalyzein-state

retentioninordertocreateaninstrumenttoidentifygraduateslikelytopracticewithin

thestateofGMEtraining.Thisdissertationutilizedtheseanalyticmethodstoproducea

novelpilot-scoringtooltoidentifygraduatesfromaMichigan-basedGMEinstitution

whoarelikelytopracticeinMichiganpost-training.Atoollikethiscouldbebeneficial

toMichigan-basedGMEinstitutions,aswellashospitalandstatephysicianrecruiters

withinthestate.However,furtherresearchwillberequiredtodeterminetheuniversal

applicabilityofthispilot-scoringtoolandareoutsidethescopeofthisdissertation.

ChapterSummary

Anoverviewoftheliteraturerelatedtoretentionofphysicianswithinphysician

shortage/ruralareas,aswellasin-stateretentionofGMEgraduateswasprovided.

Limitationsrelatedtothemethodologiesusedtoexamineretentionrelatedtothese

areas,aswellaswhat’slackingfromtheGMEretentionliterature,werealsodiscussed.

Lastly,studiesrelatedtothedevelopmentofscoringsystemsusingmultivariate

regressionandcross-validationtechniqueswereexplored.

29

CHAPTERIII

RESEARCHDESIGN

Thepurposeofthisstudywastoexamineindividuallevelcharacteristicsof

graduatesfromasingleGMEsponsoringinstitutiontodevelopatooltoidentify

graduatesthatarelikelytopracticemedicineinthesamestate.Thestateinthisstudy

isMichigan.Thischapterdescribesthestudyproceduresandanalytictechniquesused

toaddressthefollowingtwostudyobjectives:





categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganor

notlikelytopracticeinMichigan.

StudyProcedures

StudySampleDescription

Allgraduates(n=1161)fromthe18GMEtrainingprogramsofferedbyGrand

RapidsMedicalEducationPartners(GRMEP)from2000through2014wereincludedin

theinitialreview.ResidentsandfellowswhograduatedfromoneoftheGRMEPtraining

programsandwerecurrentlystillintraining(e.g.,additionaltraining,fellowship)atthe

timeofdatacollectionwereexcludedfromthereview.Transitionalyearand

preliminarysurgeryresidentswholeftGRMEPaftertheirone-yearoftraininginorderto

30

enrollinanotherresidencytrainingprogramwerealsoexcludedfromthestudy.The

rationaleforthisisthatthesegraduateshadnotcompletedtheirGMEresidency

training.Datafromthegraduatesfrom2000through2014wereusedtobuilda

predictivemodelandcreateapilot-scoringtool.

RationaleforInclusionofIndependentVariablesintheRegressionModel

MultiplestudiesandreportshavestatedthataconnectiontothestateofGME

trainingisrelatedtoin-stateretention.30-35Thisincludesbeingborninorattendinghigh

schoolormedicalschoolinthestateinwhichGMEtrainingwascompleted.Basedon

thisinformation,thefollowingvariableswereselectedtoreflectatietothestateof

Michigan(e.g.,borninMichigan,obtainingabachelor’sdegreeinMichigan,attending

medicalschoolinMichigan).AnadditionalpotentialtietothestateofMichiganisthe

lengthoftimespentinGMEtraining(timeprogram).

WhereaphysiciancompletedGMEtraininghasalsobeenreportedtobe

predictiveofGMElocation.MultiplereportsintheliteraturestatethatmanyGME

graduatespracticewithinthestatewheretheycompletedtheirGMEtraining.8,32,33

CompletionofGMEtraininginthiscasemeansthatgraduatesdidnotleavethestatein

ordertoundergofurtherGMEtraining.ForthisdissertationcompletionofGMEin

Michiganisdefinedastheresident/fellowwentintopracticeaftergraduatingfroma

GRMEPGMEprogram.

Marriage(evermarried)couldalsobeconsideredapotentialpredictorfor

whetherornotagraduatepracticesinMichigan.Forexample,ifaresidentorfellow

marriessomeonefromMichiganduringtheirtraining,theymaybemorelikelyto

31

practiceinMichiganpost-GMEthaniftheirspousewasfromsomewhereelse.

Conversely,ifaresidentorfellowisalreadymarriedpriortostartingtheirtraining,they

mayhavetiesestablishedsomewhereelseandplantogobackaftertraining.Spousal

influencewasshowntobeanimportantfactorinpracticelocationdecisionsinmultiple

studies.24,28,46

Typeofprogram(primarycarevs.non-primarycare)wasalsoincludedfor

testinginthemodel.Graduatingfromaprimarycareprogramwasreportedtobe

associatedwiththepracticingwithinthestateofGMEcompletion.30,31,33,35Female

genderwasalsoshowntobeanimportantfactorrelatedtoin-stateretentionafterGME

training.30,31Theinvestigatorsofthesestudiesreportedthatfemalesweremorelikelyto

practicewithinthestateofGMEtrainingthanmales.Therefore,genderwasincludedas

astudyvariable.

Lastly,temporaryvisaholdershaverestrictionsastohowlongtheycanbeinthe

country,aswellasonwheretheycanpracticeafterGMEgraduation.Ifthese

individualswanttoremainintheUStopractice,theyarerestrictedtopracticingin

physicianshortageareas.35Therefore,visastatuswasincludedinthemodeltoassess

whetherthisstatushadanyinfluence(positivelyornegatively)onpracticingwithinthe

stateofMichigan.

RationaleforExclusionofVariablesfromtheRegressionModel

Ageatgraduationwasnotaccountedforinthelogisticregressionmodel.Since

theoutcomevariableiswhetherornotthegraduatehaseverpracticedinMichiganat

anypointintimepost-training,ageatthetimeofgraduationdoesnothavemuch

32

meaningsincethisdecisioncouldhappenwellbeyondgraduation.Also,morethanhalf

oftheresidentsthatgraduatefromtheGMEprogramsgoontodoafellowshipafter

residency(lasting1-3years)priortodeterminingtheirpracticelocation.

Timesincegraduationwasalsonotaccountedforinthelogisticregression

model.Timesincegraduationisareasonableconsideration,sinceonecouldpropose

thatanindividualwhograduatedintheyear2000wouldhave10morepossibleyearsto

considerMichiganasaplacetopracticemedicinethansomeonewhograduatedin

2010.However,inclusionofthisvariableinascoringsystemoffuturefellowshipand

residencygraduatesisproblematic,astimesincegraduationwouldhavenomeaningfor

someonewhoisbeingevaluatedpriortograduation.

Asensitivityanalysiswasconductedtoassesstherelationshipoftimesince

graduationtotheoutcomevariable,everpracticedinMichigan.Avariablewascreated

thatincludedthreetimegroupings.Graduatesweregroupedintothosethatgraduated

from2000-2004,2005-2009and2010-2014.Thisvariablewasincludedinalogistic

regressionmodel,alongwiththeotherpredictorvariablesdescribedabove,toseeif

therewasasignificantrelationshipwitheverpracticinginMichigan.Thisvariablewas

notasignificant(p>0.1)predictorinthemodel,therefore,anassumptioncanbemade

thatthetimethataresident/fellowhastoreturntopracticeinMichiganisnot

confoundingthedissertationresults.

DataCollection

Table3.1showseachsource,thetypeofdocumentation/informationthatcould

beobtainedfromthesource,aswellasthestudyvariablesrelatedtothesource.The

33

sourcesincludedtheNewInnovationsdatabase(Uniontown,OH),aresidencydata

managementsystemusedbyGMEsponsoringinstitutionsacrossthenation,GRMEP

GMErecords,Google,theDepartmentofLicensingandRegulatoryAffairs(LARA)

website,whichgivesaccesstolicensingdataofoveramillionindividualsandbusinesses

inMichiganandGRMEPprogramdirectorsandcoordinators.

34

Table3.1

Datasources

Source

Informationavailable

Datapoint

NewInnovations

Residencyapplications,personalstatements,CV,educationalhistory,spouseinformation,visastatus,programstart/enddates,birthplace

Gender,UGMEMichigan,undergraduateMichigan,timeinprogram,primarycareprogram,birthplaceMichigan,visastatus,evermarried

GMEdepartmentrecords

Paperapplications,schooltranscripts,diplomas,documentsrelatedtovisastatus,lastnamechangeforms,jobverificationspost-training,CVs,locationaftergraduationrecords

Gender,UGMEMichigan,undergraduateMichigan,timeinprogram,primarycareprogram,birthplaceMichigan,visastatus,completedGMEtraininginMichigan

Googlesearchesusinggraduatesname

Physicianbioswithcurrentemployer,LinkedInprofiles

Gender,UGMEMichigan,undergraduateMichigan,birthplaceMichigan,evermarried,everpracticedinMichigan,completedGMEtraininginMichigan

LARAlicensingwebsite

Michiganmedicallicensehistory

EverpracticedinMichigan

GRMEPstaff:GMEDirector,ProgramDirector/Coordinator

Connectionwithformerresident/fellowthroughemail,LinkedIn,Facebook,and/orresident/fellowisGRMEPfaculty

UGMEMichigan,undergraduateMichigan,birthplaceMichigan,visastatus,evermarried,everpracticedinMichigan

Theauthorusedthedifferentdatasourcesindatacollectionefforts.The

DirectorofGMEwasavailabletoassisttheauthorwithdatacollection.Insome

35

instancestheauthorandtheDirectorofGMEreviewedapplicationstogether.Inother

cases,theauthorconsultedtheDirectorofGME,orotherpertinentcolleagues(e.g.,

programcoordinators,programdirectors),whenconflictingevidencewasdiscovered

and/orifavariablecouldnotbefound.Forexample,ifundergraduatemedical

educationcouldnotbelocatedinanyofthedatasources,theauthorreachedouttothe

programcoordinatortoseeifthatinformationcouldbeobtainedthroughtheiroffice

records.Ifdatacouldnotbelocatedand/orifconflictsinthedatacouldnotbe

resolved,thosedatapointswereleftblank.

Theauthorstartedwithalistofgraduatesfrom6/1/2000–6/30/2014.Study

numberswereassignedtoeachgraduateandacorrelationtoolwascreatedinorderto

de-identifythedata.ThenumberandnamewerethencopiedandpastedintoanExcel

document(correlationtool)andpasswordprotected.Thenameswerethendeleted

fromtheExcelfilewiththestudyvariableslisted,leavingonlythestudynumbertobe

usedastheidentifier.Theauthorusedthecorrelationtooltoidentifyindividualswhen

needed.Thisway,theindividualnamewasnotstoredwiththestudyvariablesinorder

toprotecttheidentityoftheindividualsincludedinthestudy.

TheNewInnovationsdatabasehousesinformationrelatedtocurrentresidents

andfellows,aswellasthosethathavegraduatedfromGRMEP.Electronicdocuments

housedinthedatabaseincluderesidencyapplications,curriculumvitaes(CVs),and

personalstatements.Thisdatabasewasusedtoobtaininformationrelatedtogender,

undergraduatemedicaleducation,undergraduateeducation,programstartdateand

graduationdate,residency/fellowshipprogram,placeofbirth,visastatus,maritalstatus

36

andlocationaftergraduation.SectionsontheresidencyapplicationandCVsrelatedto

hobbiesandinterests,aswellaspersonalstatements,werereviewedtosearchfor

informationrelatedtothestudyvariables(e.g.,wife,placeofbirth,educational

information).OthersourcesfromGMEDepartmentpaperrecordsincludedresidency

applications,CVs,personalstatements,schooltranscripts,copiesofdiplomas,

documentsrelatedtovisastatus,lastnamechangeforms,locationofgraduation

documentationandjobverificationspost-trainingthatcouldbeusedtogather

information.

Googlesearchesusingthegraduate’sfullname,ifavailable,orjustthefirstand

lastname,alongwiththetypeofmedicaldegree(MD/DO),werealsoperformedto

locatemissingdatapoints.Inmostcases,Googlesearcheswouldleadtothegraduate’s

currentemployerwhereinformationaboutthegraduatecouldbeobtained.For

example,SpectrumHealthliststhephysician’seducationalbackground(e.g.,UGME

institution,residency/fellowshiplocation),aswellastheirage.Iftherewasaphysician

biographyavailable,theinformationwasreviewedtoobtainorverifyinformation

relatedtothestudy(e.g.,birthplace,educationalbackground).Forexample,ifthe

authorwasreviewingaphysicianbioonlineandeducationalhistorywaslisted,thedata

collectedforthisindividualwerecheckedagainstthewebsiteforconsistency.Again,if

therewerediscrepancies,colleagueswereconsultedtodiscuss.Anyinstanceswhere

therewasdoubtabouttheaccuracyofthedata,thevariablewasleftblank.

TheDepartmentofLicensingandRegulatoryAffairs(LARA)websitewasusedto

verifytheoutcomevariable,everpracticedinMichigan.TheGMEDepartment

37

maintainsrecordsoflocationaftergraduationforallgraduates.Thesedatawere

verifiedbyusingLARAforallgraduatestoconfirmwhetheralicensetopracticein

Michiganwasobtainedafterthegraduationdate.

Lastly,eachresidency/fellowshipprogramhasaprogramcoordinatoranda

programdirector.Theprogramcoordinatorskeeptheresidencyprogramsrunning

smoothlyandareveryinvolvedintheday-to-dayactivitiesoftheirresidents.Also,

manyofthecoordinators/directorskeepintouchwiththeirformerresidents/fellows

throughemail,socialmedia(e.g.,LinkedIn,Facebook)orifthegraduateworksforthe

residency/fellowshipprogramasapartoftheteachingfaculty.Theprogram

coordinatorsanddirectorswereconsultedduringdatacollectiontoobtaindatathatthe

authorcouldnotlocateorhadquestionsabout.

DataPreparation

Priortodataanalysis,categoricalstudyvariableswerecoded(Table3.2).Timein

programwascalculatedusingprogramstartdateandgraduationdate.

Residency/fellowshipprogramswerecodedasprimarycareandnon-primarycare.

PrimarycareincludesFamilyMedicine,Pediatrics,InternalMedicineandInternal

Medicine-Pediatrics.Allotherprogramsweredesignatedasnon-primarycareand

includeallfellowshipprograms,aswellasRadiology,GeneralSurgery,Orthopaedic

Surgery,PlasticSurgery,SurgicalCriticalCare,VascularSurgery,andObstetricsand

Gynecology.Placeofbirth,locationofundergraduatedegree,locationofmedical

educationandcompletiononGMEtrainingwerecodedasMichigan/notMichigan.

38

Genderwascodedasmale/femaleandallothervariables(marriedpriorto/during

residency,everpracticedinMichigan,visastatus)werecodedasyes/no.

Table3.2Codingforregressionvariables

Variables

Coding

Y=PracticedinMichigan(outcomevariable)

0=No1=Yes

X1=Gender 0=Male1=Female

X2=Residencyprogramtype 0=Non-primaryCare1=PrimaryCare

X3=Placeofbirth 0=NotMichigan1=Michigan

X4=Timeinprogram Years

X5=Evermarried 0=No1=Yes

X6=LocationofBachelor’sdegree 0=NotMichigan1=Michigan

X7=Locationofmedicaleducation 0=NotMichigan1=Michigan

X8=Visa 0=No1=Yes

X9=LocationcompletedGMEtraining 0=NotMichigan1=Michigan

39

MissingData

Missingdataforeachvariablewereassessedpriortotheanalyses.Thepercent

ofcompletedataforeachvariableareshowninTable3.3.Themajorityofthevariables

wereatleast90%complete.

Table3.3

Percentagecompletedataforeachstudyvariable

Variable

%Complete

n=988

EverMichigan

988/988(100%)

PrimaryCare 988/988(100%)

UGMEMichigan 988/988(100%)

Gender 988/988(100%)

CompletedGMEinMI 988/988(100%)

BirthStateMI 945/988(95.6%)

EverMarried 925/988(93.6%)

VisaStatus 970/988(98.2%)

UndergradMI 988/988(100%)

TimeinProgram 988/988(100%)

Whendescribingmissingdata,itisimportanttodeterminewhethertheabsent

valueshavearelationshiptoanyoftheothervariablesofinterest.Allison,usingthe

conceptsdescribedbyRubin,detailedthreetypesofmissingdata:missingcompletely

40

atrandom(MCAR;thereisnorelationshipbetweenthemissingdataandanyother

variable),missingatrandom(MAR;missingdataareduetoarelationshipwithoneof

theothervariables)andnotmissingatrandom(NMAR;missingdataareduetothe

inherentnatureofthevariableitself).47,48

First,ananalysiswasperformedtodetermineifmissingdataforthedissertation

wereMCAR.Thiswasdoneusinglogisticregressiontotestwhethertherewereany

relationshipsbetweenthemissingnessofanyonevariableandtheotherpredictor

variablesinthemodel.Forthisdissertation,theoutcomevariableforthisanalysiswas

missing/notmissingdatafortheevermarriedvariable(thevariablewiththemost

missingdata)andthepredictorvariablesweregender,primarycare,UGMEMichigan,

undergradMichigan,birthstateMichigan,visanumericandcompletedGMEin

Michigan.Ifanyvariablesweresignificantlyrelatedtothemissingnessoftheever

marriedvariable,thedatawouldnotbedeemedtobeMCAR.Therewerethree

statisticallysignificantvariables(p<0.05)intheregressionmodel,gender,visastatus

andtimeinprogram.TheseresultsindicatedthatthemissingdatawerenotMCAR.

Next,thedatawerenotconsideredNMARsincethecauseofmissingnesswas

notrelatedtotheinherentnatureofthevariablesthemselves.Inthiscase,the

variablesweremissingduetolackofrecordstoprovidetheinformation.Thereforethe

missingdataforthisdissertationwereconsideredtobeMAR.

BasedonthedecisionthatthemissingdatawereMAR,thenextdecisionwas

whetherornotsomeformofimputationshouldbeusedtoallowfortheanalysisofall

ofthesubjects.Allisonprovidessomeinsightintothisissue,inhisdiscussionof

41

imputationofcategoricalvariables.49Hedevelopedadataset,thenrandomly

eliminated50%ofthedataforacategoricalvariable.Hecomparedacompletecase

analysis(describedasonlyusingthehalfofthedatasetthathadcompletedata)against

fourdifferentimputationtechniques,atfourdifferentproportionsforthecategorical

variablewithmissingdata(0.5,0.2,0.1,and0.01).Theresultsshowedthat,forMAR

data,allfivemethodshadequivalentestimatesoftheβ-coefficients,aswellas

equivalentstandarddeviations.Hisconclusionwasthat,forMARdata,therewasno

particularbenefittousinganimputationtechnique.Baseduponhisfindings,an

imputationtechniquehasnotbeenusedforthisdissertation,instead,acompletecase

analysiswasperformed.

LogisticRegressionAssumptions

Logisticregressionanalysiswasusedtoaddressthefirstobjectiveofthe

dissertation:Touselogisticregressionwithcross-validationtoexaminetheindividual

characteristicsrelatedtowhetherornotgraduateswhotrainedinaMichigan-based

GMEsponsoringinstitutionpracticemedicineinMichigan.Priortoperformingthe

analysis,datawerecheckedtoseeiftheassumptionsforlogisticregressionweremet

usingStata/IC13.0forMac(StataCorp,CollegeStation,TX).Theassumptionsincluded

(1)independenceoferrors,(2)allcategoriesweremutuallyexclusiveandexhaustive,(3)

alinearrelationshipbetweenthecontinuouspredictorvariablesandthelogit

transformationofthedependentvariable(practicedinMichiganY/N)existed,(4)no

multicollinearityofthepredictorvariables,and(5)nosignificantoutliersorinfluential.

42

First,theassumptionofindependencewasassumed.Thedatasetdidnot

includemultipleobservationsforanyoneperson,eachrowrepresentedaunique

individual.Also,allcategoriesforeachstudyvariableweremutuallyexclusiveand

exhaustive.

Next,linearitybetweenthecontinuouspredictorvariableoftimeinprogramand

thelogittransformationofthedependentvariable(everpracticedinMichiganY/N)was

checkedusingtheBox-Tidwellprocedure.50Thisprocedureconsistsofcreatingan

interactiontermbetweenthenaturallogofthecontinuousvariableandtheoriginal

continuousvariabletobetestedinthemodelwiththeothervariables.Theinteraction

termfortimeinprogramwascreatedusingthenaturallogofthetimeinprogram

variableandtheoriginaltimeinprogramvariable.Next,themodelwasrunincluding

theinteractionterm.Asignificant(p<0.05)interactiontermwouldindicateanon-linear

relationshipandwouldneedfurtherevaluation.Basedontheassessmentofthep-

valuefortheinteractionterm,theassumptionoflinearitywasmetfortimeinprogram

(p=0.997).

Next,multicollinearityamongthepredictorvariableswasassessedusingthe

conditionindexandtheregressioncoefficientvariance-decompositionmatrix.The

conditionindexisderivedfromtheeigenvalues,andrepresentscollinearityrelatedto

thevariablecombinations.Theconditionindexisthesquarerootofthequotientofthe

largesteigenvaluedividedbythesmallesteigenvalue.Valuesindicativeof

multicollinearitysuggestedintheliteraturerangedfrom10–30,itwasdecidedtotake

43

aconservativeapproachforthisdissertationandgowithaconditionindex>10as

suspectforcollinearity.51,52

Table3.4showstheconditionindicesandtheirproportionofvariancerelatedto

eachregressioncoefficient.Theconditionindexwithavalueof16.92wasconsideredto

bemoderatetostrongcollinearity.51Basedonthisfinding,thecollinearitywasfurther

investigatedbyreviewingthevariancedecompositionproportions.CallaghanandChen

notethatahighconditionindex(>10),alongwithtwoormoreregressioncoefficient

variances>0.5,isindicativeofproblematiccollinearity.51AsshowninTable3.3,both

theconstantandthetimeinprogramvariablecontributegreaterthanhalfoftheir

variabilitytotheeigenvalueassociatedwiththeconditionindexof16.92.Thisindicated

thatatransformationtothetimeinprogramvariablewaswarranted.

SneeandMarquardtsuggestedtheuseofcenteringinordertoaddresstheissue

ofcollinearity.53Forthisstudy,thevariabletimeinprogramwascenteredbycreatinga

newvariable(timeinprogram–3.5).Thevalueof3.5wasusedsinceitwasthemean

timeinprogramwas3.5years.Aftercenteringthevariable,theoriginaltimein

programvariablewasremovedfromthemodelandreplacedwiththecenteredtime

variable.Areassessmentoftheconditionindexshowedithaddroppedfrom16.92to

8.48(Table3.5),indicatingweakcollinearity.51Baseduponthisfinding,thecentered

timevariablewasusedinfurtheranalyses.

44

Table3.4Multicollinearityassessment

Conditionindex

Constant

Primarycare

UGMEMI

Gender

Birthstate

Ever

married

Visa

Undergrad

MI

Time

program

GME

completionMI

1.00

0.00

0.01

0.00

0.01

0.00

0.01

0.00

0.00

0.00

0.00

1.98 0.00 0.01 0.04 0.01 0.04 0.00 0.15 0.03 0.00 0.00

2.89 0.00 0.07 0.02 0.03 0.03 0.05 0.43 0.02 0.01 0.00

3.32 0.00 0.04 0.00 0.70 0.00 0.07 0.04 0.00 0.00 0.00

3.92 0.00 0.70 0.01 0.15 0.00 0.00 0.35 0.01 0.00 0.00

4.87 0.00 0.00 0.48 0.00 0.77 0.00 0.00 0.02 0.00 0.00

5.32 0.01 0.02 0.03 0.09 0.01 0.77 0.02 0.03 0.03 0.11

5.66 0.00 0.04 0.23 0.01 0.09 0.00 0.00 0.49 0.03 0.29

5.97 0.01 0.03 0.19 0.00 0.06 0.06 0.00 0.40 0.08 0.28

16.92 0.98 0.09 0.00 0.00 0.00 0.04 0.01 0.00 0.84 0.30

45

Table3.5Multicollinearityreassessment

Conditionindex

Constant

Primarycare

UGMEMI

Gender

Birthstate

Ever

married

Visa

Undergrad

MI

Time

program

GME

completionMI

1.00

0.00

0.01

0.01

0.01

0.01

0.01

0.00

0.01

0.00

0.00

1.82 0.00 0.02 0.03 0.00 0.03 0.00 0.14 0.02 0.06 0.00

2.28 0.00 0.01 0.01 0.02 0.01 0.02 0.00 0.01 0.57 0.00

2.87 0.00 0.01 0.02 0.14 0.02 0.02 0.59 0.02 0.09 0.01

3.19 0.01 0.02 0.00 0.61 0.00 0.16 0.01 0.00 0.02 0.01

3.82 0.00 0.87 0.01 0.10 0.00 0.01 0.22 0.00 0.09 0.00

4.55 0.00 0.00 0.48 0.00 0.76 0.00 0.00 0.02 0.00 0.00

5.13 0.03 0.04 0.03 0.10 0.00 0.56 0.02 0.02 0.02 0.37

5.37 0.00 0.00 0.41 0.01 0.16 0.01 0.00 0.89 0.01 0.02

8.48 0.95 0.03 0.01 0.01 0.00 0.21 0.01 0.02 0.14 0.58

46

Thedatawerealsocheckedforsignificantoutliersandinfluentialpointsbyusing

Pregibon'sdeltabeta(DBETA)influencemeasures.54ThistechniqueusestheDBETAsto

determineinfluentialobservations,definedasdatapointswhichhaveastronginfluence

onthemodelperformance.54DBETASareafunctionofthestandardizeddifferencein

betaswithdeletionofindividual.TheDBETASwereplottedtocheckforoutliers(DBETA

>0.25).Basedonthiscriterion,therewere21observationsidentifiedasoutliersinthe

dataset(Figure3.1).

Figure3.1

DBETAplot

The21outlierswereremovedtoassesstheirinfluenceonmodelperformance.

Thedifferencebetweentheperformanceofthemodelswasminimal(Table3.6).Table

47

3.7showsthemodelparameterswithoutliersandTable3.8showsthemodel

parameterswiththeoutliersremoved.

Table3.6

Modelstatisticscomparisonwithandwithoutoutliers

Model

#observations

Model χ2

Prob>chi2

Modelwithoutliers

879

281.65

<0.001

Modelwithoutoutliers 858 284.67 <0.001

48

Table3.7

Regressionmodelstatisticswithoutliers

Variable

β-coefficients

SE

pvalue

95%CI

PrimaryCare

0.778

0.182

<0.001

0.4221.134

UGMEMI 0.762 0.262 0.004 0.2491.275

Gender -0.020 0.166 0.904 -0.3460.306

BirthStateMI 1.215 0.262 <0.001 0.7011.729

EverMarried 0.583 0.180 0.001 0.2310.936

VisaStatus -0.142 0.241 0.557 -0.6140.331

UndergradMI 1.021 0.264 <0.001 0.5041.538

TimeinProgram(centered)

0.137 0.090 0.126 -0.0390.314

CompletedGMEinMI 0.593 0.224 0.008 0.1541.033

49

Table3.8Regressionmodelstatisticswithoutoutliers

Variable

β-coefficients

SE

pvalue

95%CI

PrimaryCare

0.797

0.183

<0.001

0.4381.156

UGMEMI 0.679 0.266 0.011 0.1571.201

Gender -0.003 0.170 0.988 -0.3350.330

BirthStateMI 1.275 0.266 <0.001 0.7541.795

EverMarried 0.554 0.183 0.002 0.1960.913

VisaStatus -0.161 0.258 0.534 -0.6670.345

UndergradMI 1.082 0.267 <0.001 0.5591.604

TimeinProgram(centered)

0.137 0.090 0.129 -0.0400.314

GMECompletioninMI 0.605 0.226 0.007 0.1621.048

TheDBETAswerealsoexaminedwithouttheoutlierstoassesswhetherthere

maybeissueswithotherobservationsaftertheirremoval.Noadditionaloutliers(>0.25)

wererevealedareviewoftheDBETASplot(Figure3.2).Basedontheperformanceof

themodelswithandwithoutoutliers,itwasdeterminedtoleavetheoutliersinthe

dataset.

50

Figure3.2

DBETAplotafterremovingoutliers

AnalyticProcedures

Thisnextsectiondescribestheanalysesperformedtoaddressobjectiveoneof

thedissertation:touselogisticregressionwithcross-validationtoexaminethe

individualcharacteristicsrelatedtowhetherornotgraduateswhotrainedina

Michigan-basedGMEsponsoringinstitutionpracticemedicineinMichigan.

LogisticRegression

Thefirststepwastoperformalogisticregressionusingthebestsubsetslogistic

regressionapproach.55,56Thisanalysisusesalldifferentvariablecombinationsofthe

51

modeltodeterminethebestmodelsubset.Thecriteriausedinmodelevaluation

includedtheadjustedR2,Mallow’sC,theAkaikeInformationCriterion(AIC)andthe

BayesianInformationCriterion(BIC).ThesecriteriaaresuggestedbyHosmeretal.,as

wellasLindsey&Sheather,touseintheselectionoftheoptimalmodelwhen

comparingmultiplemodels.55,56TheadjustedR2isusedtodeterminethemodelthat

explainsthemostamountofvariabilitywithintheoutcomevariable.Optimalvaluesfor

Mallow’sChavebeendescribedaseitherequaltothenumberofpredictorsplusone,or

asthesmallestvalue.56,57TheAICandBICarebothmeasuresofmodelfitthatare

penalizedforthenumberofparametersinthemodel.56TheconcernwiththeAICis

over-fittingandtheconcernforBICisunder-fitting.58ThemodelwiththelowestAIC

andBICismostdesirable.56

TheadjustedR2,Mallow’sC,AICandBICwereusedformodelcomparisonand

selectionforthisdissertation.Inanidealsituation,allofthesecriteriawouldalignon

thesamemodel.Incaseswherethisdoesnotoccur,judgmentsmustbemadeusing

thesecriteria.Forexample,Hosmeretal.notethatthedifferencesinthesecriteriacan

besonegligible(e.g.,adifference<0.1)thatthevariablesincludedineachmodelmay

needtoalsobeconsidered.55LindseyandSheatheralsoreportedthisissue.56Inone

example,theyshowedhowthevariouscriteria(adjustedR2,Mallow’sC,AICandBIC)

wouldindicatethateithera5,6or8factormodelwouldbeappropriate.They

suggestedthata6factormodelwouldbeagoodcompromise,asthevaluesforthe

variouscriteriawereonlymarginallydifferentbetweenthethreepossiblemodels.

52

Aftermodelselectionoccurred,alogisticregressionwasperformedusingthe

reducedmodel.ThismodelwasassessedusingtheHosmer&Lemeshowgoodnessoffit

statisticandtheAUC.Thegoodnessoffitstatisticisusedtodeterminehowwellthe

modelfitsthedata.Asignificantp-value(<0.05)wouldwarrantfurtherinvestigationof

themodel,whileap-value>0.05representedgoodmodel-datafit.

TheAUC,derivedfromanROCcurveanalysis,wasusedtoevaluatehowwellthe

modelwasdiscriminatingbetweenthosethateverpracticedinMichiganandthosethat

neverpracticedinMichigan.ModelperformancebasedontheAUCwasdetermined

usingthecriteriaestablishedbyHosmeretal.55Basedonthesecriteria,anAUCof0.5is

nobetterthanflippingacoin,whileanAUCof1.00representsperfectdiscrimination.

AnAUCof>0.7wasconsideredacceptableandwasthecriterionusedforthisstudy.55

Next,themodelunderwenttwoformsofcross-validationtocheckforover-fitting.

Over-Fitting

Oneoftheconcernsforanypredictivemodelderivedfromaspecificsampleis

whetherornotitwillstillhavevaluewhenappliedtootherdatasets.59,60Theprimary

concernisover-fitting.Thisoccurswhenthemodelissotightlyfittedtothedata,thatit

maynotperformthesamewhenusedwithotherdatasets.Thetechniqueofcross-

validationisavaluabletooltotestthereproducibilityofadataset,withagoaloftesting

theaccuracyandvalidityofthemodel.59Thereareavarietyofwaystodothis,

includinghold-outmethods,leave-one-outcross-validation,k-foldcross-validationand

bootstraptechniques.

53

Forexample,theholdoutmethodinvolvessplittingthesampleintotwogroups,

atrainingsetandatestset.Themodelparameterswouldbedefinedusingthetraining

set,andthenevaluatedbyestimationoftheerrorrateusingthetestset.Therearetwo

majorconcernswiththismethodology.Thefirstisthatthesamplemaybetoosmallto

allowforsplittingintotwosub-sets.Thesecondisthatusingasingletestsetand

trainingsetmayleadtoanunreproducibleresult,duetothespuriousdecisionasto

whichdataareinthetestsetandwhichareinthetrainingset.59

Onesolutiontothisproblemistoinvolvemultiplesamplesoftheentiredataset,

suchasinak-foldvalidationorbootstrappingtechnique.Fortheformer,thedata

wouldbedividedintokrandomsamples.Thelatterusesrandomsamplingwith

replacement.Forthisstudy,themodelunderwentbothafive-foldandbootstrap

internalcross-validationprocedurestotestforover-fitting.

Five-FoldCross-Validation

Thefirstmethodusedforthisdissertationwasfive-foldcross-validation.Inthis

step,thedatasetwasrandomlysplitintofivegroupsofsimilarsize.Fourofthegroups

werethetestset,whiletheremaininggroupwasusedasthevalidationset.Thetestset

wasusedtorunthemodelandthevalidationsetwasusedtodeterminehowwellthe

independentvariablesfromthevalidationdatasetwerepredictingthedependent

variable.

Thisprocesswasrepeatedfivetimes,sothateachofthefivegroupsofdata

wereusedasthevalidationsetonce,whilealloftherestofthedatawereusedasthe

testsets.Thecriterionfortheanalysiswastherootmeansquareerror(RMSE),as

54

derivedfromtestingeachofthefivevalidationsets.ThefiveRMSEvaluesobtained

werereviewedtodetermineiftheywereconsistentwiththeRMSEobtainedfromthe

finallogisticregressionmodel.Consistencyforthisevaluationwasdefinedaseachof

thefiveRMSEsfromthecross-validationshouldbenogreaterthan15%differentfrom

theoriginalRMSE.59

Bootstrap

Themodelalsounderwentbootstrappingtofurthercheckforover-fitting.The

bootstrappingprocedureusesrandomsamplingwithreplacement.Forexample,as

therewere800peoplewithcompletedata,thebootstrappingprocedureusedsampling

withreplacementtochoose800individualsfortheprocedure.Thisrandomsampleof

800couldcontainperson23fivetimes,person799notselectedatall,person200

selectedtwice,etc.,foratotalof800randomindividuals.Thebootstrappingprocedure

forthisstudywasreplicated200times.

TheevaluationcriterionforthisanalysiswastheHarrell’sC,whichisequivalent

totheAUC33.Forthesakeofconsistency,theHarrell’sCwillbereferredtoastheAUC

throughouttherestofthedissertation.TheAUCderivedfortheoriginalmodelwas

comparedtotheAUCfromthebootstrapcross-validationprocedure.Thebootstrap

methodproducedacorrectivefactor,whichwasusedtocreatetheover-fitting

correctedestimateoftheAUC.40TheoriginalandcorrectedvaluesfortheAUCwere

comparedforconsistency.Forthisstep,consistencywasdefinedthattheaverage

correctedAUC(derivedfromthe200replications)shouldbenomorethan15%

differentfromtheoriginalAUC.59

55

Pilot-ScoringTool

Thissectiondescribesthemethodsusedtoaddressthesecondobjectiveofthe

dissertation:Createascoringtoolbasedonthelogisticregressionwithcross-validation

tocategorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganornot

likelytopracticeinMichigan.

Thefinalmodelselectedduringtheanalysesrelatedtoobjectiveonewasused

tocreateascoringsystemtoidentifygraduateswhoaremostlikelytopracticein

Michigan.ThemethodofSullivanetal.wasusedforthisstep.45Theβ-coefficients,

obtainedfromtheindependentvariablesusedinthefinalmodel,wereusedtocreate

thescores.Theβ-coefficientswerecompared,withthelowestvaluerepresentingthe

referentvalue,fromwhichtheremainingscoresweredetermined.Eachβ-coefficient

wasdividedbythelowestβ-coefficientinordertodeterminethescoreforthatvariable.

Productsfromthiscalculationwereroundedtothenearestwholenumber.

Table3.9showsanexampleofSullivan’smethodologyinthecontextofthis

dissertation.45Inthisexample,scoresweredeterminedforprimarycare,

undergraduatemedicaleducationinMichigan(UGMEMI)andundergraduateeducation

inMichigan(UndergradMI).First,eachvariable’sβ-coefficientwasmultipliedbythe

valuesassociatedwitheachcategory(e.g.,0=noand1-yes)forthevariable(Table3.9,

column2).Next,theproductsweredividedbythelowestβ-coefficient,whichwas

associatedwithprimarycare(Table3.9,column3).Finally,theproductsofthese

calculationsareroundedtothenearestwholenumbertocreatethefinalscore

associatedwitheachvariable(Table3.9,column4).

56

Table3.9

Methodologyforassigningscorestoregressionvariables

Variable

β-coefficient

Scorecomputation

Roundedscore

Primarycare

Yes=1

No=0

0.69

1*0.69=0.69

0*0.69=0

0.69/0.69=1

0/0.69=0

1

0

UGMEMI

Yes=1

No=0

1.26

1*1.26=1.26

0*1.26=0

1.26/0.69=1.82

0/0.69=0

2

0

UndergradMI

Yes=1

No=0

1.37

1*1.37=1.37

0*1.37=0

1.37/0.69=1.99

0/0.69=0

2

0

Thenextstepincludedassigningscorestoallofthegraduateswithcomplete

dataforthevariablesincludedinthefinalmodel.Eachcodedvariableinthemodelwas

multipliedbythescoreassociatedwithit.Thescoresforeachvariablewerethenadded

togethertodeterminethefinalscoreforindividuals.Whencomplete,eachpersonhad

ascoreassociatedwithwhetherornotthegraduateeverpracticedmedicineinthe

stateofMichigan.Next,theROCanalysiswasperformedtoobtaintheAUCtoseehow

wellthemodelwasdiscriminatingbetweengraduatesthatpracticedinMichiganand

thosethatdidnot.FortheROCanalysis,thex-axisrepresentsthefalsepositiverate(1-

specificity)andthey-axisrepresentsthetruepositiverate(sensitivity).Onceagain,the

57

AUCcriteriaofHosmeretal.wereusedtoevaluatetheAUC.55AnAUC>0.7was

consideredtobeacceptable.

Next,ascorecut-pointwasdeterminedinordertomaximizethepredictive

modelofthescoringsystem,whichinthiscasemeanstomaximizetheabilityto

determinewhetherornotagraduatewilleverpracticeinMichigan.Perkinsand

SchistermansuggestedtheuseoftheYoudenIndex,alsoknownasYouden’sJ.61This

hasbeenshowntobeagoodestimateofthepreferredscorecut-point,andisequalto

themaximumvalueofsensitivityplusspecificityminus1,asderivedfromallpossible

cut-pointsusedtocreatetheROCcurve.Thispreferredscorecut-pointrepresentsthe

scorebestsuitedtodiscriminatebetweenthetwogroupsbeingevaluated.Forthis

step,thesensitivity(truepositiverate)andthespecificity(truenegativerate)foreach

cut-pointassociatedwithascoreweredeterminedfromtheROCanalysis.

Table3.10showsanexampleofthecut-pointsderivedfromaROCanalysis,with

theirassociatedsensitivityandspecificity.Athirdcolumnshowsthevaluesforthe

sensitivityplusspecificityminus1.Thecut-pointthatmaximizesthesensitivity(i.e.,the

abilitytoidentifythosewhopracticeinMichigan)whilemaximizingthespecificity(i.e.,

theabilitytoidentifythosewhodonotpracticeinMichigan)canbeidentifiedthrough

examinationinthistable.Forexample,ifacut-pointof>=0isselected,thiscanbe

interpretedaseveryonewouldbeidentifiedaspracticinginMichigan.Wewouldhavea

100%truepositiverate,however,ourtruenegativeratewouldbe0%.Conversely,ifwe

chose>=5asthecut-point,thisistheequivalentasstatingthatanyonewithascoreof

fiveorabovewouldbeidentifiedasstayinginMichigan.Inthiscase,ourtruepositive

58

ratewouldbeverylow,whileourtruenegativeratewouldbereallyhigh.Therefore,in

thisexample,theYouden’sJis0.44andisassociatedwithacut-pointofascoregreater

thanorequalto2.Meaning,thosewithascoreof2ormorewouldbeidentifiedas

practicinginMichiganandthosewithascoreof0or1wouldbeidentifiedasnot

practicinginMichigan.Thetruepositiverateis60%,whilethetruenegativerateis

84%.

Table3.10

Cut-pointsderivedfromanROCanalysis

Score

Sensitivity

Specificity

(Sensitivity+Specificity)-1

>=0

1.0

0.0

0.0

>=1 0.84 0.52 0.36

>=2 0.60 0.84 0.44

>=3 0.54 0.89 0.43

>=4 0.44 0.93 0.37

>=5 0.24 0.97 0.21

Thefinalstepintheprocedureincludedcreatingascoringtool.Thetoolcreated

includedquestionsbaseduponeachstudyvariablethatwasincludedinthefinalmodel,

withpointsassignedtotheresponsetoeachquestion.Forexample,ifbirthstatein

Michiganweretobeincludedinthefinalmodel,thequestioninthetoolcouldbe“Was

theresident/fellowborninMichigan?”Theresponseoptionswouldbeyesornowith

59

pointsassignedtoeachresponse(yes=1pointandno=0points).Thiswouldbedone

foreachvariableincludedinthemodel.

ChapterSummary

Describedinthischapterweretheobjectivesofthestudyandthemethodology

usedtocarryoutthestudy.Thestudyprocedures,includingadescriptionofthe

sample,datapointsandtheirrationaleforinclusion,datacollection,andmethodsfor

checkingtheassumptionsoflogisticregression,weredetailed.Lastly,theanalytical

proceduresusedtodeterminethevariablesforthepilot-scoringtoolweredescribed.

60

CHAPTERIV

RESULTS

Summarystatisticsaredescribedforthedatainthischapter.Theresultsofthe

logisticregressionandcross-validationanalysesarealsodetailed.Next,the

developmentandresultsoftheperformanceofthescoringsystemforthedataare

discussed.Lastly,thescoringtoolderivedfromthefinalregressionmodelispresented.

SummaryData

Datafortheanalysisincluded988graduates.Summarydataforthesampleare

showninTable4.1.JustoverhalfofthegraduatespracticedinMichiganatsomepoint

aftergraduation.Thesampleconsistedofmostlymales.Closetoathirdofthesample

attendedUGMEinMichigan,attendedanundergraduateinstitutioninMichiganorwere

borninMichigan.Justunderhalfofthegraduateswerefromaprimarycareprogram.

Approximately80%completedtheirGMEtraininginMichigan.

61

Table4.1Summarydataforthesample

Variable

EverMichigan

504/988(51.1%)

Primarycare 475/988(48.1%)

UGMEMichigan 302/988(30.6%)

Gender(%Male) 573/988(58%)

CompletedGMEinMI 793/988(80.3%)

BirthstateMI 266/945(28.0%)

Evermarried 661/925(71.5%)

Visastatus 126/970(13%)

UndergradMI 336/988(34.0%)

Timeinprogram(mean+SD) 3.5+1.0yrs

LogisticRegression

Thefirstanalysiswasperformedtoaddressobjectiveone,whichwastouse

logisticregressionwithcross-validationtoexaminetheindividualcharacteristicsrelated

towhetherornotgraduateswhotrainedinaMichigan-basedGMEsponsoring

institutionpracticemedicineinMichigan,usedabestsubsetslogisticregression

approach.TheoutcomevariablewaseverpracticedinMichiganandthepredictor

variablesincludedbirthstateinMichigan,undergradinMichigan,primarycare,ever

62

married,UGMEinMichigan,completedGMEinMichigan,timeinprogram,genderand

visastatus.

Thebestsubsetsanalysisproducedthe11bestvariablecombinationsforthe

model(Table4.2).ThistableincludestheadjustedR2,Mallow’sC,theAICandtheBIC,

whichwereusedtodeterminethebestmodelforfurtheranalyses.Thecriteria

describedinChapterIIIforthisevaluationdidnotconvergeonthesamemodel.The

modelwiththelowestBICwasafivevariablemodel,whichincludedbirthstatein

Michigan,primarycare,undergradMichigan,UGMEMichiganandevermarried,though

theMallow’sCwashigherthanthenumberofvariablesinthemodel.Theseven

variablemodel(birthstateinMichigan,primarycare,undergradMichigan,UGME

Michigan,completedGMEinMichigan,evermarried,programyears)hadthehighest

adjustedR2,aMallow’sClessthanthenumberofparametersinthemodelandlowest

AIC.

TheproblemofselectingthemodelwiththelowestBICistheconcernforunder-

fitting,whileproblemofselectingthesevenvariablemodelwiththelowestAICand

maximumadjustedR2istheconcernforover-fitting.Withthisinmind,thesixvariable

model,whichincludedbirthstateinMichigan,primarycare,undergradMichigan,UGME

Michigan,completedGMEinMichiganandevermarriedwasselected.Thismodelhada

Mallow’sCsmallerthanthenumberofparametersinthemodelcoupledwithan

adjustedR2,AICandBICsimilartothatofthefiveandsevenparametermodels.

63

Table4.2Resultsofthebestsubsetslogisticregressionanalysis

Block

Modelparameters

Adj.R2

Mallow’sC

AIC

BIC

1

BSMI

0.2051

85.229

1076.302

1085.860

2 BSMI,UGMI 0.2368 47.905 1041.547 1055.883

3 BSMI,PC,UGMI 0.2527 29.756 1024.101 1043.217

4 BSMI,PC,UGMI,Married 0.2625 18.814 1013.380 1037.274

5 BSMI,PC,UGMI,Married,UGMEMI 0.2704 10.336 1004.944 1033.617

6 BSMI,PC,UGMI,Married,UGMEMI,GMEMI(selected) 0.2740 6.991 1001.573 1035.025

7 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY 0.2754 6.358 1000.915 1039.145

8 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY,Visa 0.2749 8.000 1002.553 1045.562

9 BSMI,PC,UGMI,Married,UGMEMI,GMEMI,PY,Visa,Gender 0.2740 10.000 1004.553 1052.341

BSMI=BirthStateMI;UGMI=UndergradMI;PC=PrimaryCare;GMEMI=CompletedGMEinMI;PY=ProgramYears

64

Thefinalmodelselectedincluded889observations,seeTable4.3forsummary

dataforthefinalmodelsample.

Table4.3

Summarydataforsampleincludedinthefinalmodel

Variable

%

EverMichigan

458/889(51.5%)

Primarycare 425/889(47.8%)

UGMEMichigan 268/889(30.2%)

BirthstateMI 253/889(28.5%)

Gender(%male) 507/889(57.0%)

Evermarried 631/889(71.0%)

UndergradMI 293/889(33.0%)

CompletedGMEtraininginMI 715/889(80.4%)

Table4.4showstheβ-coefficients,standarderrors,p-valuesand95%confidence

intervalsforthevariablesincludedinthemodel.Othermodelparametersthatwere

assessedincludedtheHosmer&Lemeshowgoodnessoffitstatistic(χ2=3.21;p=0.201)

andtheNagelkerkepseudoR2=0.361,percentagecorrectlyclassifiedbythemodel

(73.57%)andthesensitivity(truepositiverate:60.48%)andspecificity(truenegative

rate:87.47%).Thenon-significantgoodnessoffitstatisticsuggeststhatthereisgood

modeldatafit.ThepseudoR2impliesthemodelexplains36.1%ofthevariabilitywithin

65

theoutcomevariable.Themodelcorrectlyclassified73.6%oftheobservationsinthe

datasetaseitherpracticinginMichiganornotpracticinginMichigan.

Table4.4

Finalregressionmodelstatistics

Variable

β-coefficient

SE

pvalue

95%CI

BirthstateMI

1.214

0.261

<0.001

0.7031.725

Primarycare 0.694 0.162 <0.001 0.3761.012

UndergradMI 1.038 0.262 <0.001 0.5231.552

Evermarried 0.629 0.178 <0.001 0.2800.978

UGMEMI 0.784 0.260 0.003 0.2741.295

CompletedGMEinMI 0.474 0.208 0.022 0.0670.881

TheresultsoftheROCanalysisareshowninFigure4.1.Thex-axisrepresents

thefalsepositiverate,whilethey-axisshowsthetruepositiverate.Thediagonalline

representsthevaluerelatedtonodiscrimination,similartoflippingacoin.Thelinewith

thecircularsymbolsrepresentsthedataforthisstudy.

TheAUC,derivedfromtheROCanalysis,was0.802.Thisfinalmodelmetthe

criteria(AUC>0.7)establishedbyHosmeretal.deemingitanacceptablemodelfor

discriminatingbetweenthosethateverpracticedinMichiganandthosethatdidnot.55

66

Figure4.1

ROCforfinalmodel

Five-FoldCross-Validation

Thenextanalysisaddressedthesecondpartofobjectiveone,whichwastouse

logisticregressionwithcross-validationtoexaminetheindividualcharacteristicsrelated

towhetherornotgraduateswhotrainedinaMichigan-basedGMEsponsoring

institutionpracticemedicineinMichigan.Afive-foldcross-validationprocedurewas

performedusingtheselectedmodeltoassessforover-fittingthemodeltothedata.In

thisstep,thedatasetwasrandomlysplitintofivegroupsofsimilarsize.Fourofthe

groupswerethetestset,whiletheremaininggroupwasusedasthevalidationset.The

67

testsetwasusedtorunthemodelandthevalidationsetwasusedtodeterminehow

welltheindependentvariablesfromthevalidationdatasetpredictedthedependent

variable.

Thisprocesswasrepeatedfivetimes,sothateachofthefivegroupsofdata

wereusedasthevalidationsetonce,whilealloftherestofthedatawereusedinthe

varioustestsets.ThecriterionfortheanalysiswastheRMSE,asderivedfromtesting

eachofthefivevalidationsets.ThenthefiveRMSEvaluesobtainedwerecheckedfor

consistencyagainsttheRMSEfromthefinalmodel.Consistencyforthisevaluationwas

definedaseachofthefiveRMSEshouldbelessthan15%differentfromtheoriginal

RMSE.59TheresultsoftheRMSEcomparisonareshowninTable4.5.Basedonthe

evaluationcriterion(<15%different)forthisanalysis,itdoesnotappearthatthereare

issueswithover-fittingthemodeltothedata.

Table4.5

Five-foldcross-validationRMSEcomparison

RMSE

Originalmodel

0.423

1 0.427

2 0.447

3 0.410

4 0.427

5 0.418

68

Bootstrap

Asecondformofcross-validationtoaddressobjectiveonewasalsoperformed.

Thisstepincludedperformingthebootstraptechniquetofurthercheckthefinalmodel

forover-fitting.Thebootstrapprocedureusedrandomsamplingwithreplacement.This

analysiswasrunon889graduates,therefore200randomsamplesof889graduates

wereobtained.Thisrandomsampleof889couldcontaingraduate757fivetimes,

graduate12notselectedatall,graduated579selectedtwice,etc.,foratotalof889

subjects.Thebootstrappingprocedureforthisstudywasreplicated200times.

TheevaluationcriterionforthisanalysiswastheAUC.ThevaluefortheAUC,

wascomparedtotheAUCfromthebootstrapcross-validationprocedurefor

consistency,definedaslessthan15%differentfromoneanother.TheoriginalAUCand

correctedAUCare0.802and0.803,respectively.AsdescribedbyHarrell,thecorrected

AUCrepresentsavaluewhichhasbeenadjustedforbiasallowingthecalculationofan

over-fittingcorrectedestimate.40Basedontheevaluationcriterion(<15%different)set

forthisanalysis,over-fittingisnotanissue.59Boththefive-foldcross-validationmethod

andthebootstrapmethodsuggestedthatover-fittingwasnotaconcern.

Pilot-ScoringTool

Thisnextsectiondescribestheresultsofthesecondobjectiveofthe

dissertation,whichwastousetheresultsfromthelogisticregressionwithcross-

validationanalysistocreateascoringmechanismtocategorizeGMEprogramgraduates

intogroups:likelytopracticeinMichiganornotlikelytopracticeinMichigan.Theβ-

69

coefficientsassociatedwiththepredictorsfromthefinalmodelwereusedtocreatea

scoringsystemtoidentifygraduateswhoaremostlikelytopracticeinMichigan.The

methodofSullivanetal.wasusedforthisstep.45

Table4.6showshowthescoreswerecalculatedforeachvariableincludedinthe

finalmodel.Thiswasdonebyfirstestablishingthereferencecategoryforeachvariable.

ThefirstcolumnofTable4.6showsthereferencecategoriesforeachvariable.Inthis

case,allofthevariablesinthemodelareyes/novariables,withyescodedasoneandno

codedaszero.

ThesecondcolumninTable4.6showstheβ-coefficientassociatedwitheach

variableandtheresultoftheβ-coefficientmultipliedbyeachcodedvariable.For

example,theβ-coefficientforbirthstateMichiganis1.214.Thiswasmultipliedby1

correspondingwithayesresponseand0correspondingwithanoresponse.Next,the

lowestβ-coefficientofallthevariableswasusedtodeterminethevaluecorresponding

toonepointonthescoringscale.Inthiscase,thecompletedGMEinMichiganvariable

hadthelowestβ-coefficient(0.474).

ThethirdcolumnofTable4.6showsthecomputationforthescore.This

involveddividingeachoftheproductsfromthecolumntwocalculationsbythelowest

β-coefficient(completedGMEinMichigan=0.474).ForbirthstateMichigan,this

involveddividing1.214by0.474fortheyescategoryand0by0.474forthenocategory.

ThelastcolumninTable4.6showstheresultsofthecalculationsincolumnthree

roundedtothenearestwholenumber.Thescoringsystemrangedfrom0-10points.A

70

yesresponsetobeingborninMichiganwasassociatedwiththreepoints,ayesresponse

tograduatingfromaprimarycareprogramwasassociatedwithonepoint,ayes

responsetoobtaininganundergraduatedegreeinMichigancorrespondedwithtwo

points,ayesresponsetoeverbeingmarriedcorrespondedwithonepoint,ayes

responsetocompletingUGMEinMichigancorrespondedwithtwopointsandonepoint

wouldbegiventoanyonewhohadayesresponsetocompletingGMEinMichigan.

71

Table4.6Scorecalculationofpredictorvariablesinthefinalregressionmodel

Variable

β-coefficient

Scorecomputation

Roundedscore

BirthstateMI

Yes=1

No=0

1.214

1*1.214=1.214

0*1.214=0

1.214/0.474=2.561

0/0.474=0

3

0

Primarycare

Yes=1

No=0

0.694

1*0.694=0.694

0*0.694=0

0.694/0.474=1.464

0/0.474=0

1

0

UndergradMI

Yes=1

No=0

1.038

1*1.038=1.038

0*1.038=0

1.038/0.474=2.189

0/0.474=0

2

0

Evermarried

Yes=1

No=0

0.629

1*0.629=0.629

0*0.629=0

0.629/0.474=1.327

0/0.474=0

1

0

UGMEMI

Yes=1

No=0

0.752

1*0.784=0.784

0*0.784=0

0.784/0.474=1.654

0/0.474=0

2

0

CompletedGMEinMI

Yes=1

No=0

0.474

1*0.474=0.474

0*0.474=0

0.474/0.474=1.000

0/0.474=0

1

0

72

Thenextstepincludedassigningscoresassociatedwithwhetherornotthe

graduateeverpracticedmedicineinthestateofMichigantoallofthegraduatesinthe

datasetwithcompletedata.Eachcodedvariableinthemodelwasmultipliedbythe

scoreassociatedwithit.Thescoresforeachvariablewerethenaddedtogetherto

determinethefinalscoreforindividuals.Whencomplete,eachpersonhadascore

associatedwithwhetherornotthegraduateeverpracticedmedicineinthestateof

Michigan.

AnROCanalysisusingthescoresderivedfromthemodelwasthenperformedto

obtaintheoptimalcut-pointforthescoringtool(Figure4.2).Thesensitivity(true

positiverate)andthespecificity(truenegativerate)foreachofthescorecut-points(0-

10)areshowninTable4.7.Thecut-pointthatmaximizedtheabilitytodetermine

whetherornotagraduateeverpracticedinMichiganwasdeterminedusingYouden’s

J.61ThepercentageassociatedwithYouden’sJisequaltothemaximumvalueof

sensitivityandspecificityminus1,asderivedfromallpossiblecut-pointsusedtocreate

theROCcurve.

Table4.7showsthattheYouden’sJforthisanalysisis0.486,whichisassociated

withacut-pointof4.Thiscut-pointhasasensitivityof61.6%(truepositiverate),a

specificityof87.0%(truenegativerate)andcorrectclassificationrateof73.9%.The

interpretationforthiscut-pointmeansthatindividualswithascore>4aremorelikelyto

practiceinMichigan,thanthosewhoreceiveascore<4.

73

Figure4.2

ROCscorecut-points

74

Table4.7

Sensitivityandspecificityforscorecut-points

Score

Sensitivity

Specificity

Correctlyclassified

(Sensitivity+Specificity)-1

>=0

1.000

0.000

51.5%

0.000

>=1 0.993 0.053 53.8% 0.046

>=2 0.924 0.281 61.2% 0.205

>=3 0.775 0.689 73.3% 0.464

>=4 0.616 0.870 73.9% 0.486

>=5 0.587 0.889 73.3% 0.476

>=6 0.533 0.919 72.0% 0.452

>=7 0.443 0.933 68.1% 0.376

>=8 0.382 0.954 65.9% 0.336

>=9 0.286 0.975 62.0% 0.261

>=10 0.127 0.986 54.3% 0.113

>10 0.000 1.000 48.5% 0.000

TheAUCforthisanalysiswas0.7975andexceededthecriterionofacceptable

discrimination>0.7usedforthisdissertation.55Meaningthemodelwasdiscriminating

quitewellbetweengraduateswhoeverpracticedinMichiganandthosethatdidnot.

SeeAppendixAforthepilot-scoringtool.Thefirstcolumncontainsthe

questionscreatedforeachvariableincludedinthefinalmodelthetoolwasderived

75

from,aswellasthepointsassociatedwithayesornoresponse.Thenextcolumnisfor

recordingthepointsforeachresponsetothequestions.Thelastrowofthetoolis

wherethetotalscore,calculatedfromthepointsrecorded,willberecorded.Score

interpretationisalsoshown.Anelectronicformthatauto-calculatesscoresmayalsobe

anoptionforthistool.

ChapterSummary

Describedinthischapterweretheresultsoftheanalyticalproceduresusedto

createandcross-validatethemodelfromwhichscoringsystemwasderived.Also

detailedweretheresultsoftheanalysesusedtoassessthepilot-scoringtoolforthe

2000-2014graduates.Adescriptionofthepilot-scoringtoolderivedfromthelogistic

regressionwithcross-validationanalyseswasalsodiscussed.

76

ChapterV

DISCUSSION

Thepurposeofthedissertationandstudyobjectivesarepresentedinthis

chapter.Next,adiscussionoftheresultsfortoeachobjectiveisdetailed.Lastly,

limitationsofthework,suggestionsforfutureresearchandstudyconclusionsare

presented.

StudyPurpose

DuetotheoverwhelmingfinancialsupportofGMEtrainingbyfederal,stateand

traininghospitalsalongwithincreasedconcernsoverbudgetsandphysicianshortage

issues,GMEsponsoringinstitutionsareunderincreasedpressuretodemonstrateROIto

fundingsources.OnesuggestedmethodfordemonstratingROIistotrackin-state

retentionratesofgraduates.21,22Amechanismforidentifyingresidents/fellowsduring

trainingwhoarelikelytopracticeinthestateinwhichtheytraincouldallowfor

targetedrecruitmentofthesephysicians,whichcouldpossiblyresultinhigherin-state

retentionofGMEgraduates.

Thepurposeofthisstudywastoexamineindividuallevelcharacteristicsof

graduatesfromaMichigan-basedGMEsponsoringinstitution,inordertodevelopatool

toidentifygraduatesthatarelikelytopracticemedicineinthestateinMichigan.The

followingobjectiveswereusedtoguidethedissertation.

77





categorizeGMEprogramgraduatesintogroups:likelytopracticeinMichiganor

notlikelytopracticeinMichigan.

ResearchObjectiveOne

TheMichiganConnection

Theregressionanalysisusedtoaddressthefirstobjectiveproducedamodelthat

includedsixpredictorvariables:borninMichigan,medicalschoolinMichigan,

undergraduateeducationinMichigan,GMEeducationinaprimarycareresidency,

completionofGMEtraininginMichiganandwhetherornotthephysicianhadmarried

priortoorduringresidency.Basedontheresultsofthisdissertation,anindividualwho

wasborninMichigananddidbothundergraduateandmedicaleducationinMichigan

whocompletedresidency/fellowshipinaMichigan-basedGMEprogramishighlylikely

topracticeinMichiganpost-training.ThethreevariablesrelatedtoMichigansupport

thetheorythatGMEgraduateswithsometietothestatemaybemorelikelytopractice

inthestateinwhichtheytrainedthanthosewithnotietothestate.9,30-35,46

Seiferetal.showedthatphysicianswhoattendedmedicalschoolinthesame

stateasGMEtrainingwerealmostfourtimesaslikelytoberetainedinthestateto

practicethanthosewhodidnot.31TheGeorgiaStatewideAreaHealthEducationCenter

78

statedthatretentionwasjustover80%forphysicianswhograduatedfromGeorgia

basedhighschools,medicalschoolsandresidencyprograms.9Retentionwasjustunder

75%forthosethatgraduatedfromGeorgiabasedhighschoolsandresidencies.

BowmanfoundsimilarfindingsforphysiciansinVirginia.32Manyotherreportshave

shownhighin-stateretentionratesforGMEgraduateswithatietotheirrespective

states.30,34,35,46Furthermore,forrecruitmentpurposes,theIowamedicalsociety

developedadatabaseofphysiciansinresidencythathaveatietothestateofIowa.62

Theseincludethosethatwereborninthestate,graduatedfromanIowa-basedmedical

schoolorarecompletingresidencytraininginIowa.Thethoughtwasthatfocusingon

physicianswithanIowaconnectionwouldgiveagreaterpossibilityoflong-term

retentionofthephysicianwithinthestate.Giventhesefindingsfrompreviousreports,

itisnotsurprisingthatthethreevariableswiththelargest β-coefficients(andthe

greatestweightsinthepilot-scoringtool)wereborninMichigan,UGMEinMichiganand

undergraduateinMichigan.

FinalGMETrainingLocation

ThefindingsofthisstudysuggestthatcompletingGMEtraininginMichigan,

meaningthegraduatedidnotleavethestateforfurthertraining(e.g.,elsewhere),was

predictiveofpracticinginMichigan.Thisisconsistentwithmuchoftheliterature,which

suggeststhatlocationattheendofGMEtraininghasbeenassociatedwithahigh

percentageofgraduatespracticingwithinthestateinwhichtheytrained.8,32,33In

Bowman’sstudy,ahigherpercentageoffamilymedicinephysiciansthatonlycompleted

GMEtraininginVirginiapracticedinVirginiapost-trainingthanphysiciansthatonly

79

attendedmedicalschoolinVirginiaandthosethatwereborninVirginia,butdidnot

receivetheirUGMEorGMEtrainingthere.32Faganetal.reportedthat57%offamily

physicianspracticedwithinthesamestateastheirresidencytraining.33Physicians

graduatingfromaGMEprogramthatarereadytogointopracticeasopposedto

enteringanotherGMEprogram(e.g.,fellowship)maybemorelikelytopracticeinthe

stateinwhichtheytrained.

PrimaryCareandIn-StateRetention

Primarycarewasalsoincludedinthefinalmodel.Theresultsshowedthatthose

whograduatedfromaprimarycareprogramweremorelikelytopracticeinMichigan

thanthosethatdidnot.ThisresultisconsistentwiththestudybySeiferetal.who

reportedthatgeneralpractitionerswere1.4timesmorelikelytopracticemedicine

withinthestateofGMEtrainingthanspecialists,whichwasasignificantpredictorin

theirregressionmodel.31Armstrongetal.alsoreportedthattheneedforprimarycare

physiciansinNewYorkwasgreaterthanthethatofgraduateswithmorespecialized

training,whichimpliesthatthereweremorejobopportunitiesavailablewithinthe

state.35Theysupportedthiswithadditionaldatawhichshowedthatprimarycare

physiciansreceivedanaverageof4.3jobofferscomparedto3forspecialists.

Thisfindingforthecurrentstudymaybedrivenbytheneedforprimarycare

graduatesinthestateofMichiganandnotasmuchofaneedfortheothergraduates

fromdifferentprograms.In2014,thestateofMichiganhad293locations,withinits83

counties,designatedasprimarycarehealthprofessionsshortageareas(HPSA).63,64At

thattime,only63.6%oftheprimarycarehealthneedswerebeingmetinthose

80

designatedareasanditwasreportedthatitwouldtakejustover200physiciansto

alleviatetheneedsandtoremovetheshortagedesignations.63In2015,locationwith

theHPSAdesignationforprimarycaregrewfrom293locationsto308locations.65

Withthisgrowingneed,jobopportunitiesinMichiganmaybemoreabundant

forgraduatesfromprimarycareprograms.Themajorityoftheprimarycaregraduates

makepracticedecisionsimmediatelyfollowinggraduation,whereasthoseinprograms

suchasgeneralsurgery,radiologyandorthopedicsaremorelikelytogoontodomore

specializedtraininginafellowshipprogram.Therefore,primarycaregraduatesmayget

recruitedmoreheavilyasgraduationapproaches,makingthempotentiallymorelikely

topracticeinMichiganaftergraduation.Further,graduateswhogoontodoa

fellowshipinanotherstatemaybelesslikelytoreturniftheyhavenoconnectiontothe

stateinwhichtheydidtheirtrainingornopotentialjobprospectswhentheyleave.

MarriageandIn-StateRetention

Whetherornotthegraduatewasevermarriedwasasignificantpredictorinthe

finalregressionmodel.Inotherwords,graduateswhowereevermarriedweremore

likelytopracticeinMichiganthantheirsinglecounterparts.Manystudieshaveshown

thatspousesofphysicianshaveaninfluenceonpracticelocationdecisions.24,28,46

Somepossibleexplanationastowhymarriedgraduatesaremorelikelytopracticein

Michiganareasfollows.Itmaybethattheresident/fellowmettheirspouseduringtheir

trainingandthespousehasties(e.g.,family,friends,career)toMichigan.Alternatively,

theresident/fellowandtheirspousehavebuiltalifeduringtrainingandhave

establishedrelationshipsthatmayinfluencethepracticelocationdecisiontopracticein

81

thestateofGMEtraining.Perhapstherewereunderlyingfinancialincentivesinthe

formofmedicalschoolloanrepaymentprogramstopracticeinoneofMichigan’srural

areasthatthegraduateandtheirspousethoughtwasworthittostay.Explorationof

theunderlyingreasonthisvariablewassignificantinthemodelisoutsidethescopeof

thisstudy.

VariablesNotintheFinalRegressionModel

Variablesnotincludedinthemodelweretimeinprogram,genderandvisa

status.Itwasinterestingthattimeinprogramdidnotinfluencethedecisiontopractice

inMichigan.Thetheoreticalrationaleforincludingthisvariablewasthatresidentswho

trainedintheareaforlongerperiodsoftimemightbeinclinedtostayduetohaving

moretimetoestablishstrongertiestothecommunity.However,theresidentsinthose

longerprogramsusuallyheadofftodoafellowshipsomewhereelseandmayendup

practicingwheretheycompletetheirGMEtraining.Asnotedearlier,thereisagood

evidencetoindicatethatphysicianstendtopracticemedicineinthestateinwhichthey

finishGMEtraining.8,32,33Ofthe211GRMEPgraduateswhowentontofellowship

trainingaftergraduation,178(84.4%)leftthestatetodoso.Only35%ofthose

graduatesreturnedtopracticeinMichiganafterfellowship.

Additionally,thisfindingcouldbetiedtotheneedsofcertainspecialtiesin

Michigan.Theprimarycarespecialtiesare3-4yeartrainingprograms,whereasthenon-

primarycarespecialtiesare4-6yeartrainingprograms.Iftherearenojobswithinthe

graduate’sspecialty,thenthechoicetostayorcomebackmaynotbeanyoptionfor

thesegraduates,makingthelengthoftimespentintheprogramimmaterial.Asthe

82

needforprimarycarephysiciansisgrowingacrossthenation,theopportunitiesare

greaterforthisgroupofgraduates.17

ThedecisiontopracticeinMichiganwasnotinfluencedbygenderdifferences.

Burfieldetal.showedthat60%offemalegraduatespracticeinthestateinwhichthey

underwentGMEtraining,asopposedtoonly50%ofmalegraduates.30Similarly,Seifer

etal.foundthatfemaleGMEgraduateswere1.2timesmorelikelytopracticemedicine

inthestateofGMEtrainingthanmales,whichwasastatisticallysignificantresulton

multivariateanalysis.31Thesefindingswereabsentinthisdissertation.Thiscouldbe

duetothefactthatfarmorefemaleshaveenteredintothephysicianworkforceover

thelast20years.

VisastatuswasalsonotasignificantpredictorofpracticingmedicineinMichigan

ornot.Residents/fellowswithtemporaryvisasareonlyallowedtobeinthecountryfor

acertainperiodoftimebeforetheymustleaveagain.Therearestrictrequirementsfor

thoseontemporaryvisasregardingpracticingintheUSaftergraduation.Ifavisaholder

wantstoremainintheUS,theymustpracticeinastateorfederallydesignated

physicianshortagelocation.35MichiganhasmanyruralareasthatholdanHPSA

designation.63Thesetypesofopportunities,availablebothinsideandoutsideof

Michigan,mayhaveinfluencedthefactthathavingatemporaryvisadoesnothavea

positiveornegativerelationshipwithpracticinginMichigan.Althoughnotsignificantin

thefinalmodel,35.7%ofthevisaholdersinthedissertationdatasetendedup

practicinginMichiganatsomepointpost-GMEtraining.

83

ModelPerformance

TheROCanalysisshowedthemodelwasabletodiscriminatebetweenthosethat

practicedinMichiganandthosethatdidnotjustbelowthelevelofexcellent

discrimination(0.8),basedonthecriteriaestablishedbyHosmeretal.55Thetechnique

ofcross-validationwasusedtotestthereproducibilityofadataset,withagoalof

testingtheaccuracyandvalidityofthemodel.Over-fittingthemodeltothedatais

oftenachallengeinregressionanalysesthusthefinalmodelwasalsocheckedforover-

fittingusingtwodifferentmethods.

Thefirstwasfive-foldcross-validation.Thecriterionforexcessover-fittingwasa

changeinRMSE>15%,betweentheoriginalRMSEandanyofthefivevaluesderived

fromthecross-validationtechnique.59Themaximumdifferencefoundinthe

dissertationdatawas5.7%,whichwasmuchlessthanthecriterionvalue.

Thesecondprocedure,bootstrapping,usedsamplingwithreplacementandis

anothermethodforcheckingthemodeldatafit.Forthisanalysis,thecriterionfor

excessover-fittingwasachangeinAUC>15%.59Thedifferencefoundinthe

dissertationdatawas0.1%,whichwasfarlessthanthecriterionvalue.Theanalysesfor

thefive-foldcross-validationandthebootstraptechniqueshowedthatover-fittingwas

notanissueforthismodel.

ResearchObjectiveTwo

Thesecondobjectiverelatedtocreatingapilot-scoringtoolfromthefinalmodel

derivedandtestedusingthelogisticregressionwithcross-validationresults.Thesix

84

predictorsfromthefinalmodelwereusedtocreatepointsrelatedtoeachpredictorand

acut-pointforthetotalscorewasdeterminedtousewhendecidingwhetheragraduate

waslikelytopracticeinMichiganornot.ThemethodsofSullivanetal.wereusedto

developthepilot-scoringtool.45

ThepointsderivedforeachvariablewerebirthstateMichigan(3points),UGME

inMichigan(2point),UndergradMichigan(2points),primarycare(1point),ever

married(1point)andcompletedGMEtraininginMichigan(1point).Therangesof

scoresforthistoolwere0pointsto10points.Acut-pointoffourwasestablishedusing

anROCanalysis,meaningthatgraduateswithascoreof>4werelikelytopracticein

Michiganandgraduateswithascoreof0-3werenotlikelytopracticeinMichigan.A

questionforeachvariablewasalsodevelopedforthefinalpilot-scoringtoolforeaseof

use.Thispilot-scoringtoolcanbeusedtoassessresidents/fellowsatanytimeduring

theirtrainingforidentificationforpotentialrecruitmentbyhospitals,localoffices/clinics

and/orMichigan-basedphysicianrecruiters.

ContributiontoEvaluation,MeasurementandResearch

EmpiricalevidencefromasingleMichigan-basedGMEsponsoringinstitutionwas

usedtoexaminevariablesrelatedtowhetherornotagraduatepracticedmedicinein

Michigan.Throughtheuseoftechniquesfromthefieldofresearch,includinglogistic

regressionwithcross-validation,anovelpilot-scoringtoolwasdevelopedtoassistinthe

evaluationofwhetherornotGMEgraduatesarelikelytopracticemedicineinMichigan.

Thetoolisacontributiontothefieldofevaluation.Itprovidesadatadrivenapproach

toevaluatingthelikelihoodaGMEgraduatewouldpracticeinMichigan.Thistoolcould

85

beusedtoproducealistofidentifiedgraduateslikelytopracticeinMichiganfor

hospitalsandotherphysicianrecruitersinMichigantousefortargetedrecruitmentof

theseindividuals.

StudyLimitations

Onelimitationofthestudywasmissingdata.Allofthevariablesinthedataset

wereatleast90%complete.Sincethedataforthisdissertationwereconsideredtobe

MAR,thereportofAllisonwouldindicatethatthedatacouldbeanalyzedusingthe

completecases.49Additionally,asdatasetforthisdissertationwaslarge,missingdata

maynothaveasmuchofaninfluenceonoutcomesinthisstudyasitwouldifthe

samplesizeweremuchsmaller.60However,missingdatacanaffecttheoutcomeof

analysesandthereforecautionintheinterpretationofresultsmaybewarranted.

Anotherlimitationincludesusingdatafromasingleinstitution,whichcanlimit

thegeneralizabilityofthefindings.Thepilot-scoringtoolwascreatedusingdatafrom

oneinstitution,therefore,thistoolmaynotperformthesameorincludethesame

variablesifdatafrommultipleinstitutionsinMichiganand/orotherstateswereusedto

createthetool.

Theextendedstudytimeframe,14years,introducesissuesrelatedtofactors

thatcouldinfluencepracticelocationdecisions.Forexample,Michiganwentthrougha

recessionnotthatlongago,whichcouldhavepotentiallyinfluencedthedecisionto

practiceinMichigan.Also,theemploymentopportunitiesareunlikelytobethesame

fromoneyeartothenext.Thefindingsofthisdissertationcouldalsobeinfluencedby

thoseresidentswhograduated,butwerestillinfellowshiptrainingatthetimeofdata

86

collectionandthereforewerenotincludedinthestudy.Lastly,themorerecent

graduates(e.g.,2012,2013,2014)haven'thadasmuchtimetoreturntopracticein

Michiganassomeoftheearlyyears(e.g.,2000,2001,2002).However,asensitivity

analysis,usingamultivariatelogisticregression,wasconductedtodeterminethe

relationshipoftimesincegraduationandtheoutcomevariable,everpracticedin

Michigan,priortotheplannedanalysesforthisdissertation.Therewasnosignificant

effectseenfortimesincegraduation.

Manyfactorsremainunaccountedforinthemodelandsubsequentlythescoring

tool.Themodelinthisstudyaccountedfor36%ofthevariabilitywithintheoutcome

variable.However,thedataforthestudywerederivedfromdatathatwereavailable

throughhistoricalrecords,whichleavesmanyvariablesthatmayfurtherexplain

practicelocationdecisionsoutofthemodel.Thesecouldincludefactorssuchasthe

weather,proximitytofamily,employmentopportunitiesatthetimeofgraduation,

salaryandbenefitinfluences,military/loanrepaymentprogramstatus,spouses

career/education.35Additionally,datafromsurveysand/orinterviewsofformer

residentscouldhavebeenusedtoidentifyotherinfluentialfactorsrelatedtopractice

location.However,bothofthesemethodswouldprovetime-consumingandwould

mostlikelyfurtherreducethesamplesize.

Lastly,thequalityofphysiciansthatchoseMichiganasapracticelocationand

howlongtheypracticedinthestateareunknowns.Ahighin-stateretentionof

mediocrephysicianswouldnotbeadesirableoutcome.Also,ahighin-stateretention

ofphysicianswhostaylessthanayearisanundesirableoutcome.Theadditionofa

87

meanstoidentifythequalityofthegraduatesalongwiththescoresofwhoislikelyto

practiceinMichiganwouldfurtherimproverecruitmenteffortsandpotentiallytheROI

tofundersofGMEtraining.Trackingthelengthoftimethataphysicianpracticesinthe

stateofGMEtrainingisanotheradditiondescriptorthatcouldbeaddedtothe

performancemarkerofin-stateretention.

FutureResearch

Furtherresearchcouldincludepilot-testingthescoringtoolonfuturegraduates

fromGRMEP.Targetedrecruitmentofindividualsidentifiedaslikelytostayand

practiceinMichigancouldbeundertaken.In-stateretentionratescouldbetracked

overtimetodetermineifthetargetedrecruitmentwasworththeeffort.Forexample,

graduatescouldbescoredearlyonintheirlastyearoftraining.Alistofrandomly

selectedindividualslikelytopracticeinMichigancouldbegiventohospitalphysician

recruiters,aswellasotherMichigan-basedphysicianrecruitersinordertotargetthose

individualstofillpositionslocally,regionallyorstatewide.In-stateretentionofthe

individualslikelytopracticeinMichiganonthelistandthoselikelytopracticein

Michigannotonthelistcouldbetrackedovertimetoseeifin-stateretentionratesare

differentbetweenthesetwogroups.Ifnot,maybetargetedrecruitmenteffortsarenot

necessaryinthisgroupandshouldpotentiallybefocusedatthosenotlikelytostay.

SimilarstudiesusingthisdesigncouldincludeotherinstitutionsinMichiganand/or

otherinstitutionsintheMidwest.

Anotherpotentialtwistonthisdesignwouldbetoincluderandomsamplesof

thoselikelytopracticeinMichiganandthosenotlikelytopracticeinMichiganonthe

88

listfortargetedrecruitment.Comparisonsofin-stateretentionratesbetweenthe

groupscouldbeusedtoprovidesupportingevidence,ornot,ofthescoringtoolsability

todiscriminatebetweenthoselikelytopracticeinMichiganornot.

Thepilot-scoringtoolcouldalsobeusedtoscreenresidencyapplicants(i.e.,

medicalschoolgraduates)fordeterminingthosethatarelikelytopracticeinMichigan

aftergraduation.Applicantsidentifiedashavingascoreoffourorhigherpriorto

startingresidencymaybemorelikelytopracticeinMichiganpost-training.This

informationcouldbeusedintheresidencycandidateselectionprocess.

Forexample,theFamilyMedicineProgramDirectorcouldscoreeachresidency

candidateafterreviewinghisorherresidencyapplicationandinterviewingthe

candidate.Whendeterminingtherankofcandidatesfortheprogramthatratesimilarly

ontheprogram’sselectioncriteria(e.g.,educationalperformance,personalities,

volunteerexperience)thescorecouldbeusedtorankcandidatesmorelikelytopractice

inMichiganhigherthanthosenotaslikely.In-stateretentionratescouldthenbe

trackedovertimetoseeifthereisanincreasingtrendovertime.Thiscouldbedonefor

fiveyearsandthein-stateretentionratesofthetwograduatingclassespriorto

implementationofthispracticecouldbecomparedtothetwograduatingclassesafter

implementation.

Onelastsuggestioncouldbetoassessthereliabilityofthepredictabilityofthe

scoringtool.Forexample,scorescouldbegeneratedforresidents/fellowsinthenext

twograduatingclasses(2016and2017).Thepracticelocationofthesephysicianswould

thenbetrackedthrough2021.ThedataforthevariableeverpracticedinMichigan

89

couldthenbecomparedusingROCandAUCanalysestotheoriginaldataforthis

dissertation.

Conclusion

Theeffortsofthisdissertationproducedanovelpilot-scoringtoolforusein

identifyingGMEgraduateswhoarelikelytopracticemedicineinMichiganpost-training.

Targetedrecruitmentofidentifiedindividualsmayleadtoincreasedin-stateretention

rates,whichcouldtranslateintoameansofdemonstratingROItoGMEfundingsources,

particularlystateandlocalsources.

90

REFERENCES

1. AssociationofAmericanMedicalColleges.WhatDoesMedicareHavetoDowith

GraduateMedicalEducation?aamc.org.

https://www.aamc.org/advocacy/campaigns_and_coalitions/gmefunding/factsh

eets/253372/medicare-gme.html.AccessedOctober23,2015.

2. AssociationofAmericanMedicalColleges.WhyTeachingHospitalsAre

ImportanttoAllAmericans.aamc.org.

https://www.aamc.org/advocacy/campaigns_and_coalitions/gmefunding/factsh

eets/253374/teaching-hospitals.html.AccessedOctober23,2015.

3. AmericanHospitalAssociation.FastFactsonUSHospitals.2013AnnualSurvey

Data.http://www.aha.org/research/rc/stat-studies/fast-facts.shtml.Accessed

October23,2015.

4. AmericanMedicalNews.UncapGMEfundingtobreakadangeroustraining

bottleneck.amednews.com

http://www.amednews.com/article/20130204/opinion/130209978/4/.Accessed

October23,2015.

5. AssociationofAmericanMedicalColleges.2013StatePhysicianWorkforceData

Book.aamc.org

https://www.aamc.org/download/362168/data/2013statephysicianworkforceda

tabook.pdf.AccessedOctober23,2015.

6. Henderson,TM.MedicaidGraduateMedicalEducationPayments:A50-State

Survey.AssociationofAmericanMedicalColleges.

91

https://members.aamc.org/eweb/upload/Medicaid%20Graduate%20Medical%2

0Education%20Payments%20A%2050-State%20Survey.pdf.AccessedOctober

23,2015.

7. AccreditationCouncilforGraduateMedicalEducation.ReportofACGME

AccreditatedSponsoringInstitutionsinMichigan.acgme.org.

https://apps.acgme.org/ads/Public/Reports/Report/16.AccessedOctober23,

2015.

8. French,R.Michigangetsmed-schoolboom,doctorbust.BridgeMagazine.

2012:1-6.http://bridgemi.com/2012/06/michigan-gets-med-school-boom-

doctor-bust/.AccessedOctober23,2015.

9. GeorgiaStatewideAreaHealthCareNetwork.2013PrimaryCareSummit

Summary.GeorgiaRegentsUniversitywebsite.

http://www.gru.edu/ahec/documents/fy15gme_request.pdf.AccessedOctober

23,2015.

10. BerwickD,WilenskyGR,AlexanderB,etal.GraduateMedicalEducationThat

MeetstheNation’sHealthNeeds.InstituteofMedicine.

http://www.iom.edu/Reports/2014/Graduate-Medical-Education-That-Meets-

the-Nations-Health-Needs.aspx.AccessedOctober23,2015.

11. MetzlerI,GanjawallaK,KaupsK,MearaJG.TheCriticalStateofGraduate

MedicalEducationFunding.BulletinoftheAmericanCollegeofSurgeons.

http://bulletin.facs.org/2012/11/critical-state-of-gme-funding/.Accessed

October23,2015.

92

12. O’SheaJS.ReformingGraduateMedicalEducationintheUS.Backgrounder.

2014;(2983):1-14.Retrievedfromhttp://report.heritage.org/bg2983.Accessed

October23,2015.

13. ConnollyB,DeVriesJ,GiffordB;WorkgrouponGraduateMedicalEducation

Funding.ReportonWorkgroupEffortsonGMEFunding.michigan.govwebsite.

https://www.michigan.gov/documents/mdch/18463_04_01_12_379998_7.pdf

AccessedOctober23,2015.

14. GreeneJ.Snyder'sProposedStateBudgetCutsRuralHospitals,Graduate

MedicalEducationFunding.Crain’sDetroitBusiness.

http://www.crainsdetroit.com/article/20150211/NEWS/150219947/snyders-

proposed-state-budget-cuts-rural-hospitals-graduate-medical.AccessedOctober

23,2015.

15. MichiganPrimaryCareAssociation.ConferenceCommitteeFinalizesFY2016

MichiganDepartmentofCommunityHealthBudget.mpca.net.

http://www.mpca.net/news/234177/Conference-Committee-Finalizes-FY-2016-

Michigan-Department-of-Community-Health-Budget.htm.AccessedOctober23,

2015.

16. AssociationofAmericanMedicalColleges.PreserveMedicareSupportfor

PhysicianTraining.aamc.org.

https://www.aamc.org/download/262676/data/gmefactsheet.pdf.Accessed

October23,2015.

93

17. AssociationofAmericanMedicalColleges.PhysicianShortagestoWorsen

WithoutIncreasesinResidencyTraining.KaiserHealthNews.

https://kaiserhealthnews.files.wordpress.com/2011/01/md-shortage.pdf.


18. Meehan,M.HospitalssufferingunderACA,newKentuckyHospitalAssociation

reportsays.kentucky.com

http://www.kentucky.com/2015/05/08/3842438_hospitals-suffering-under-

aca.html?rh=1.AccessedOctober23,2015.

19. CenterforHealthWorkforceStudies.MichiganPhysicianSupplyandDemand

Through2020.InstitutueforHealthPolicywebsite.

http://www.ihcs.msu.edu/documents/MichiganPhysicianSupplyandDemandthro

ugh2020-Final.pdf.AccessedOctober23,2015.

20. SperoJC,FraherEP,RickettsTC,RockeyPH.GMEintheUnitedStates:Areview

ofStateInitiatives.CecilG.ShepsCenterforHealthServicesResearch,The

UniversityofNorthCarolinaatChapelHill.

https://www.shepscenter.unc.edu/product/gme-in-the-united-states-a-review-

of-state-initiatives/.AccessedOctober23,2015.

21. TrippUmbach.ExpandingGraduateMedicalEducationinIndiana.Indiana

UniversitySchoolofMedicinewebsite.

http://evansville.medicine.iu.edu/files/9314/3215/2212/State_of_Indiana_GME

_Expansion_Approach_with_Southwest_IN_as_Pilot.pdf.AccessedOctober23,

2015.

94

22. CommonwealthofMassachusettsExecutiveOfficeofHealthandHumans

Services.ReportoftheSpecialCommissiononGraduateMedicalEducation.

mass.govwebsite.http://www.mass.gov/eohhs/docs/eohhs/gme-final-

report.pdf.AccessedOctober23,2015.

23. RabinowitzHK,DiamondJJ,MarkhamFW,PaynterNP.CriticalFactorsfor

DesigningProgramstoIncreasetheSupplyandRetentionofRuralPrimaryCare

Physicians.JAMA.2001;286(9):1041-1048.

24. ChanBT,DeganiN,CrichtonT,etal.FactorsInfluencingFamilyPhysiciansto

EnterRuralPractice.CanadianFamilyPhysician,2005;51:1246-1247.

25. PellegrinKL.ABriefSurveytoIdentifyPrioritiesforImprovingClinician

RecruitmentandRetention:ResultsfromHawaiiIslandPhysicians.Hawaii

JournalofMedicine&PublicHealth.2012;71(4Suppl1):45-49.

26. RobertGrahamCenter.WhatInfluencesMedicalStudentandResidentChoices?

RobertGrahamCenterwebsite.http://www.graham-

center.org/dam/rgc/documents/publications-reports/monographs-

books/Specialty-geography-compressed.pdf.AccessedOctober23,2015.

27. HancockC,SteinbachA,NesbittTS,AdlerSR,AuerswaldCL.WhyDoctorsChoose

SmallTowns:ADevelopmentalModelofRuralPhysicianRecruitmentand

Retention.SocialScience&Medicine.2009;69(9):1368-1376.

28. MayoE,MathewsM.SpousalPerspectivesonFactorsInfluencingRecruitment

andRetentionofRuralFamilyPhysicians.CanJRuralMed.2006;11(4):271-276.

95

29. CameronPJ.Professional,PersonalandCommunity:3DomainsofPhysician

RetentioninRuralCommunities.CanJRuralMed.2012;17(2):47-54.

30. BurfieldWB,HoughDE,MarderWD.LocationofMedicalEducationandChoice

ofLocationofPractice.JMedEduc.1986;61:545-559.

31. SeiferSD,VranizanK,GrumbachK.GraduateMedicalEducationandPhysician

PracticeLocation:ImplicationsforPhysicianWorkforcePolicy.

JAMA.1995;274(9):685-691.

32. OwenJA,HaydenGF,BowmanRC.InfluenceofPlacesofBirth,Medical

Education,andResidencyTrainingontheEventualPracticeLocationsofFamily

Physicians:RecentExperienceinVirginia.SouthernMedicalJournal.2005;98(6):

674-674.

33. FaganEB,GibbonsC,FinneganSC,etal.FamilyMedicineGraduateProximityto

TheirSiteofTraining:PolicyOptionsforImprovingtheDistributionofPrimary

Care.AccessFamMed.2015;47(2):124-30.

34. OfficeforHealthcareWorkforceAnalysis&Planning.RetainingPhysicians

EducatedinSouthCarolinaDataBrief.OfficeforHealthcareWorkforceAnalysis

&Planningwebsite.

http://www.officeforhealthcareworkforce.org/docs/Retaining%20Physicians%20

Educated%20in%20South%20Carolina.pdf.AccessedOctober23,2015.

35. ArmstrongD,ChungR,ForteGJ.2014NewYorkResidencyTrainingOutcomesA

SummaryofResponsestothe2014NewYorkResidentExitSurvey.Centerfor

HealthWorkforceStudieswebsite.

96

http://chws.albany.edu/archive/uploads/2015/04/2014_NY_Residency_Training

_Outcomes.pdf.AccessedOctober23,2015.

36. HowtheDataElementsontheAMAPhysicianMasterfileAreCollected,

MaintainedandVerified.PDFfillerwebsite.http://www.pdffiller.com/51185066-

primarysourcepdf-How-the-Data-Elements-on-the-AMA-Physician-Masterfile-

are--Various-Fillable-Forms.AccessedOctober23,2015.

37. AmericanMedicalAssociation.AMAPhysicianMasterfile.AmericanMedical

Associationwebsite.http://www.ama-assn.org/ama/pub/about-ama/physician-

data-resources/physician-masterfile.page.AccessedOctober23,2015.

38. KonradTR,SlifkinRT,StevensC,MillerJ.UsingtheAmericanMedicalAssociation

PhysicianMasterfiletoMeasurePhysicianSupplyinSmallTowns.TheJournalof

RuralHealth.2000;16(2):162-167.

39. SunGW,ShookTL,KayGL.(1996).InappropriateUseofBivariableAnalysisto

ScreenRiskFactorsforUseinMultivariableAnalysis.JournalofClinical

Epidemiology.1996;49(8):907-916.

40. HarrellJrFE.RegressionModelingStrategiesWithApplicationstoLinearModels,

LogisticRegression,andSurvivalAnalysis.NewYork,Springer-Verlag;2001.

41. DillmanDA,SmythJD,ChristianLM.Internet,MailandMixed-ModeSurveysThe

TailoredDesignMethod(3rded.).Hoboken,NJ:JohnWiley&Sons,Inc;2009.

42. HoughCL,CaldwellES,CoxCE,etal.DevelopmentandValidationofaMortality

PredictionModelforPatientsReceiving14DaysofMechanical

Ventilation.CriticalCareMedicine.2015;43(11):2339-2345.

97

43. ZodpeySP,DeshpandeSG,UghadeSN,KulkarniSW,ShrikhandeSN,HingeAV.A

predictionmodelformoderateorseveredehydrationinchildrenwith

diarrhoea.JDiarrhoealDisRes.1999Mar;17(1):10-16.

44. ImperialeTF,MonahanPO,StumpTE,GlowinskiEA,RansohoffDF.Derivation

andValidationofaScoringSystemtoStratifyRiskforAdvancedColorectal

NeoplasiainAsymptomaticAdults:ACross-sectionalStudy.AnnIntern

Med.2015;163(5):339-346.

45. SullivanLM,MassaroMM,D'AgostinoRB.PresentationofMultivariateDatafor

ClinicalUse:TheFraminghamRiskScoreFunctions.StatMed.2004;23(10):1631-

1660.

46. RosenthalTC,RosenthalGL,LucasCA.FactorsinthePhysicianPracticeLocation

Puzzle:ASurveyofNewYorkStateResidency-TrainedFamilyPhysicians.The

JournaloftheAmericanBoardofFamilyPractice.1992;5(3):265-273.

47. AllisonPD.HandlingMissingDatabyMaximumLikelihood.SASGlobal

Forum2012.StatisticalHorizonswebsite.http://statisticalhorizons.com/wp-

content/uploads/MissingDataByML.pdf.AccessedOctober23,2015.

48. RubinDB.InferenceandMissingData.Biometrika,1976;63(3):581-592.

49. AllisonPD.ImputationofcategoricalvariableswithPROCMI.InSASUsersGroup

International,30thMeeting(SUGI30).SASwebsite.

http://www2.sas.com/proceedings/sugi30/113-30.pdf.AccessedOctober23,

2015.

50. HilbeJM.LogisticRegressionModels.CRCPress;2009.

98

51. CallaghanKJ,ChenJ.(2008).RevisitingtheCollinearDataProblem:An

AssessmentofEstimator‘Ill-Conditioning’inLinearRegression.Practical

AssessmentResearch&Evaluation.2008;13(5):1-6.

http://pareonline.net/getvn.asp?v=13&n=5.AccessedOctober23,2015.

52. BelsleyDA.AGuidetoUsingtheCollinearityDiagnostics.ComputerSciencein

EconomicsandManagement.1991;4(1):33-50.

53. SneeRD,MarquardtDW.(1984).CollinearityDiagnosticsDependontheDomain

ofPrediction,theModel,andtheData.TheAmericanStatistician.1984;38(2):

83-87.

54. LongJS,FreeseJ.Regressionmodelsforcategoricaldependentvariablesusing

Stata.StataPress;2006.

55. HosmerDW,LemeshowS,SturdivantRX.AppliedLogisticRegression(3rded.).

Hoboken,NJ:JohnWiley&Sons,Inc;2013.

56. LindseyC,SheatherS.VariableSelectioninLinearRegression.Stata

Journal.2010;10(4):650.

57. HockingRR.ABiometricsinvitedpaper.TheAnalysisandSelectionofVariables

inLinearRegression.Biometrics.1976;32(1):1-49.

58. BurnhamKP,AndersonDR.MultimodelInferenceUnderstandingAICandBICin

ModelSelection.SociologicalMethods&Research.2004;33(2):261-304.

59. HarrellFEH,LeeKL,MarkDB.TutorialinBiostatistics:MultivariablePrognostic

Models:IssuesinDevelopingModels,EvaluatingAssumptionsandMeasuring

andReducingError.StatisticsinMedicine,1996;15:361–387.

99

60. TabachnickBG,FidellLS.UsingMultivariateStatistics(6thed.).UpperSaddle

River,NJ:PearsonEducation,Inc;2013.

61. PerkinsNJ,SchistermanEF.TheInconsistencyof“Optimal”CutpointsObtained

usingTwoCriteriaBasedontheReceiverOperatingCharacteristicCurve.

AmericanJournalofEpidemiology.2006;163(7):670–675.

62. KrupaC.HowStatesareKeepingDoctorsfromMovingOut.AmericanMedical

Newswebsite.

http://www.amednews.com/article/20111219/profession/312199944/1/.


63. TheHenryJ.KaiserFamilyFoundation.PrimaryCareHealthProfessional

ShortageAreas(HPSAs).TheHenryJ.KaiserFamilyFoundationwebsite.

http://kff.org/other/state-indicator/primary-care-health-professional-shortage-

areas-hpsas/.AccessedOctober23,2015.

64. MichiganCountiesList.Michigan.govwebsite.

http://www.michigan.gov/som/0,1607,7-192-29701_31713_31714-97053--

,00.html.AccessedOctober23,2015.

65. HealthResourcesandServicesAdministration.ShortageAreas.HRSAData

Warehousewebsite.

http://datawarehouse.hrsa.gov/Tools/DataByGeographyResults.aspx?geoTyp=St

ate&geoCd=26.AccessedOctober23,2015.

100

AppendixA

Pilot-ScoringTool

101

A.Pilot-ScoringTool

Points

Wastheresident/fellowborninMichigan?

Yes=3points;No=0points

Willtheresident/fellowgraduatefromaprimarycareprogram?


Didtheresident/fellowreceiveabachelor’sdegreefromaMichigan-basedcollege?


Wastheresident/fellowevermarried?


Didtheresident/fellowgraduatefromaMichigan-basedmedicalschool?


Didtheresident/fellowcompletetheirGMEtraininginMichigan?


TotalScore

Scores>4=likelytopracticeinMichigan;Scores<4=lesslikelytopracticeinMichigan

102

AppendixB

HumanSubjectsInstitutionalReviewBoardLetter

103

i i i t i " \ , , , , , i , i - , , ; i r r l ' i - i+, , - { i ' { - " ' i=:,r,,. :'

Re:

Human Subjects lnstitutional Review Board

Date: Mav 12.2015

To: Jessaca Spybrook,Tracy Frieswyk, St

Principal Investigator

From: Amy Naugle, Ph.D.,

HSIRB Proiect Number 15-05-13

This letter will serve as confirmation that your research project titled "GME GraduateRetention Rates: A Single Institution Study" has been approved under the exemptcategory of review by the Human Subjects Institutional Review Board. The conditionsand duration of this approval are specified in the Policies of Western MichiganUniversity. You may now begin to implement the research as described in theapplication.

Please note: This research may only be conducted exactly in the form it was approved.You must seek specific board approval for any changes in this project (e.g., you mustrequest a post approval change to enroll subjects beyond the number stated in yourapplication under "Number of subjects you want to complete the study)." Failure toobtain approval for changes will result in a protocol deviation. In addition, if there areany unanticipated adverse reactions or unanticipated events associated with the conductof this research, you should immediately suspend the project and contact the Chair of theHSIRB for consultation.

Reapproval of the project is required if it extends beyond the termination datestated below.

The Board wishes you success in the pursuit of your research goals.

Approval Termination: May 11,2016

1903 W. Michigan Ave., Kalamazoo, Ml 49008-5456pH0nE: (269) 38i-8293 rnx: (269) 387-8276

cAMPUs srTr: 251 W. Walwood Hall

Documents

GME Graduate Retention Rates: A Single Institution Study