Evaluating Health Policy Evaluating Health Policy through Natural Experimentsthrough Natural Experiments

Andrew B. Bindman, MDAndrew B. Bindman, MD

Professor Medicine, Professor Medicine, Health Policy, Health Policy, Epidemiology & Epidemiology & BiostatisticsBiostatistics

UCSFUCSF

Insurance Reforms and Insurance Reforms and Other Imminent Policy Other Imminent Policy ChangesChanges Many of you identified specific Many of you identified specific policies that are or are about to policies that are or are about to be implemented that are relevant be implemented that are relevant to your research intereststo your research interests

Opportunity to study this change Opportunity to study this change as a way to inform the policy as a way to inform the policy processprocess

Learning About PoliciesLearning About Policies

Read the newspaperRead the newspaper– NY Times, Washington Post, PoliticoNY Times, Washington Post, Politico

Health newswire servicesHealth newswire services– California Healthline (California Healthline (www.chcf.org))– Kaiser Health NewsKaiser Health News

(www.kaiserhealthnews.org)(www.kaiserhealthnews.org) Academic faculty Academic faculty Professional Societies/Scientific Professional Societies/Scientific OrganizationsOrganizations

Community-based organizationsCommunity-based organizations Directly from policy decision-makersDirectly from policy decision-makers

My Research InterestMy Research Interest

Health consequences of Health consequences of public policiespublic policies

Access to and quality of Access to and quality of care for low-income, care for low-income, diverse, and patient diverse, and patient populations vulnerable to populations vulnerable to poor health because of their poor health because of their social circumstancessocial circumstances

Medi-Cal: Medi-Cal: CaliforniaCalifornia’’s Medicaid s Medicaid ProgramProgram ~8 million beneficiaries~8 million beneficiaries $41 billion last year$41 billion last year 22ndnd largest use of general fund (17%) largest use of general fund (17%) Pays for 1 in every 2 births in the Pays for 1 in every 2 births in the statestate

Approximately half of beneficiaries Approximately half of beneficiaries are Latinoare Latino

Provides 2/3rds of safety net Provides 2/3rds of safety net fundingfunding

Medicaid Population: Medicaid Population: Clinical QuestionClinical Question

Many Medicaid beneficiaries have Many Medicaid beneficiaries have interruptions in coverage interruptions in coverage (churning)(churning)

Many uninsured gain Medicaid Many uninsured gain Medicaid coverage when hospitalizedcoverage when hospitalized

Does Medicaid coverage provided at Does Medicaid coverage provided at the time of a hospitalization the time of a hospitalization adequate or do interruptions in adequate or do interruptions in Medicaid enrollment have a negative Medicaid enrollment have a negative impact on the health of impact on the health of beneficiaries?beneficiaries?

Designing a Research StudyDesigning a Research Study

Randomized trialRandomized trial– Feasibility?Feasibility?

Designing a Research StudyDesigning a Research Study

Randomized trialRandomized trial– Unethical and impracticalUnethical and impractical

Observational studyObservational study– Compare the experiences of Compare the experiences of beneficiaries who have interruptions beneficiaries who have interruptions in coverage with those who have in coverage with those who have continuous coveragecontinuous coverage

Reverse CausalityReverse Causality

Interruption in coverage might Interruption in coverage might not predict worse health not predict worse health outcome so much as worse health outcome so much as worse health might predict whether or not might predict whether or not have interrupted coveragehave interrupted coverage

Bias of higher admissions among Bias of higher admissions among those with continuous coveragethose with continuous coverage

Designing a Research StudyDesigning a Research Study

Randomized trial- is it Randomized trial- is it feasible?feasible?

Observed variation - is it Observed variation - is it biased?biased?

Natural experiment - does a Natural experiment - does a good one exist?good one exist?

Natural ExperimentsNatural Experiments

A A naturalnatural or q or quasi-experimentuasi-experiment is a is a naturally occurring instance of naturally occurring instance of observable phenomena which observable phenomena which approximate or duplicate the approximate or duplicate the properties of a controlled properties of a controlled experiment. In contrast to . In contrast to laboratory experiments, these events , these events aren't created by scientists, but aren't created by scientists, but yield data which nonetheless can be yield data which nonetheless can be used to make causal inferences. used to make causal inferences.

What Are the Elements of a What Are the Elements of a Good Natural Experiment in Good Natural Experiment in Health PolicyHealth Policy

Policy implementation not Policy implementation not biased by patient biased by patient characteristics such as health characteristics such as health statusstatus

Policy can be effectively tied Policy can be effectively tied to a to a ““treatmenttreatment”” exposed group exposed group

Access to before/after dataAccess to before/after data

Medicaid Population: Medicaid Population: Policy QuestionPolicy Question

Federal law requires re-determination of Federal law requires re-determination of eligibility for beneficiaries at a eligibility for beneficiaries at a minimum of every 12 months but states minimum of every 12 months but states have option to do more frequentlyhave option to do more frequently

Beneficiaries who do not Beneficiaries who do not ““re-sign upre-sign up”” are are dropped from programdropped from program

Does frequency of a stateDoes frequency of a state’’s re-enrollment s re-enrollment process increase the number of process increase the number of beneficiaries with interruptions in beneficiaries with interruptions in coverage and if so is this in turn coverage and if so is this in turn associated with patientsassociated with patients’’ health? health?

Natural Experiment of Natural Experiment of Interrupted Medicaid Interrupted Medicaid CoverageCoverage

California extended Medicaid re-California extended Medicaid re-enrollment period for enrollment period for allall children in children in California from every 3 to every 12 California from every 3 to every 12 months on January 1, 2001months on January 1, 2001

Extension of eligibility re-Extension of eligibility re-determination period should be determination period should be associated with an increase in associated with an increase in continuity of Medicaid coverage, but continuity of Medicaid coverage, but should not except through its influence should not except through its influence on continuity of coverage be associated on continuity of coverage be associated with the health status of children. with the health status of children.

Challenging Issues in Challenging Issues in Studying Natural Studying Natural ExperimentsExperiments Learning about a policy Learning about a policy change as it is about to change as it is about to happen or after the fact happen or after the fact makes it harder to collect makes it harder to collect baseline databaseline data

Primary Data CollectionPrimary Data Collection

Can be challenging to organize in Can be challenging to organize in time to assess pre-policy time to assess pre-policy condition condition

Lots of work but lots of control Lots of work but lots of control over data collection (eg surveys, over data collection (eg surveys, physiological measures, etc)physiological measures, etc)– Time consumingTime consuming– ExpensiveExpensive

Difficult to maintain over timeDifficult to maintain over time

Secondary DatabasesSecondary Databases

Pre-existing data that are often Pre-existing data that are often collected for an alternative collected for an alternative purposepurpose

Individual level or sometimes Individual level or sometimes aggregate dataaggregate data

Examples:Examples:– National surveys National surveys – RegistriesRegistries– Study cohortsStudy cohorts– Administrative dataAdministrative data

Secondary Data:Secondary Data:Advantages/ChallengesAdvantages/Challenges

Efficient - cheap, fast and often very Efficient - cheap, fast and often very largelarge

Little control on what was collectedLittle control on what was collected Collection is often Collection is often longitudinal/repeated cross-sectional longitudinal/repeated cross-sectional

Potential to analyze temporal changesPotential to analyze temporal changes If source is a payer or a provider may If source is a payer or a provider may have incomplete capturehave incomplete capture

Can be scooped by others with access Can be scooped by others with access to same datato same data

Finding Secondary DataFinding Secondary Data

www.ctsi.ucsf.edu/research/celdac

www.phpartners.org/health_stats.html

Medicaid Data for Studying Medicaid Data for Studying Interruptions in CoverageInterruptions in Coverage

Comprehensive and detailed Comprehensive and detailed regarding eligibilityregarding eligibility

Fee for service claims complete Fee for service claims complete Missing claims information for Missing claims information for beneficiaries in managed carebeneficiaries in managed care

WonWon’’t reflect experience of t reflect experience of beneficiaries when they arenbeneficiaries when they aren ’’t t coveredcovered

Statewide Hospital Patient Statewide Hospital Patient Discharge AbstractsDischarge Abstracts

Comprehensive capture of all Comprehensive capture of all hospitalizations in state hospitalizations in state regardless of payerregardless of payer

Includes information on hospital Includes information on hospital admission diagnosesadmission diagnoses

Provides payer source at time of Provides payer source at time of hospitalizationhospitalization

Ambulatory Care Sensitive Conditions:Ambulatory Care Sensitive Conditions:AHRQ Prevention Quality IndicatorsAHRQ Prevention Quality Indicators

1.1. Condition with acute course and window for Condition with acute course and window for interventionintervention

2.2. Condition with chronic course amenable to self-Condition with chronic course amenable to self-managementmanagement

ACS Conditions

Acute Conditions:Acute Conditions:– DehydrationDehydration– Ruptured Ruptured Appendicitis Appendicitis

– CellulitisCellulitis– Bacterial Bacterial PneumoniaPneumonia

– Urinary Tract Urinary Tract InfectionInfection

Chronic Chronic Conditions:Conditions:– AsthmaAsthma– Hypertension Hypertension – COPDCOPD– Diabetes Diabetes MellitusMellitus

– Heart FailureHeart Failure– AnginaAngina

Statewide Hospital Patient Statewide Hospital Patient Discharge AbstractsDischarge Abstracts

Provides payer source at time of Provides payer source at time of hospitalization but not over timehospitalization but not over time

Critical question for hospitalizations Critical question for hospitalizations for ambulatory care sensitive for ambulatory care sensitive admissions is what the insurance admissions is what the insurance status was prior to the admission status was prior to the admission since many uninsured gain coverage in since many uninsured gain coverage in association with the hospitalizationassociation with the hospitalization

Linked CA Hospital Linked CA Hospital Discharge and Medicaid Discharge and Medicaid Eligibility FilesEligibility Files

OSHPD: Hospital Discharge Data

1998 2003

DHS: Medi-Cal Enrollment Database

1998 2003• Demographics• Monthly enrollment history• Aid Category (e.g. TANF or SSI)• FFS, managed care • Other insurance

• Diagnosis (ICD-9 Code)• Month/Year of admission

Linkage

Pre/Post Study of Re-Pre/Post Study of Re-Enrollment Policy Change for Enrollment Policy Change for ChildrenChildren

Children 1-17 years with a Children 1-17 years with a minimum of 1 month of Medicaid minimum of 1 month of Medicaid coverage in California coverage in California

Outcome = time to a hospital Outcome = time to a hospital admission for an ambulatory care admission for an ambulatory care sensitive condition sensitive condition

Main predictor = time periodMain predictor = time period– Pre policy change = Jan Pre policy change = Jan ‘‘99-December 99-December

‘‘0000– Post policy change = Jan Post policy change = Jan ‘‘01-December 01-December

‘‘0202

Children 1-17 Years in California Medicaid Children 1-17 Years in California Medicaid Before and After Policy to Change EnrollmentBefore and After Policy to Change Enrollment

1999-2000 1999-2000 2001-20022001-2002

NN 3,288,171 3,288,171

3,230,120 3,230,120

Mean Age (yrs)Mean Age (yrs) 99 99

% Female% Female 5050 5151

Ethnicity (%)Ethnicity (%)

HispanicHispanic 5454 5656

Black Black 1313 1212

AsianAsian 88 88

OtherOther 2525 2424

Aid Group (%)Aid Group (%)

TANFTANF 4747 5050

SSISSI 33 33

OtherOther 5050 4747

Managed Care (%)Managed Care (%) 4747 4141

Children with Continuous Children with Continuous Medicaid Enrollment by Time Medicaid Enrollment by Time PeriodPeriod

0%

10%

20%

30%

40%

50%

60%

70%

Pre: 1999-2000 Post: 2001-2002

49

62

Years of Enrollment

Percenta

ge

Probability of a Probability of a Hospitalization for an ACS Hospitalization for an ACS Condition Over TimeCondition Over Time

0.00

0.05

0.10

0.15

0.20

0.25

0.300.35

0.40

0 3 6 9 12 15 18 21 24

Before 2001Enrollment Extension After 2001 EnrollmentExtension

Months

Children: Adjusted Risk of ACS Children: Adjusted Risk of ACS HospitalizationHospitalization

Relative HazardRelative Hazard P-ValueP-Value

Post policy Post policy 0.740.74

<.0001<.0001

AgeAge 0.880.88

<.0001<.0001

FemaleFemale 0.970.97

0.01750.0175

EthnicityEthnicity

HispanicHispanic

3.263.26

<.0001<.0001

Black Black

4.704.70

<.0001<.0001

AsianAsian

1.101.10

0.09260.0926

OtherOther

2.972.97

<.0001<.0001

Aid GroupAid Group

TANFTANF

1.471.47

<.0001<.0001

SSISSI

24.9024.90

<.0001<.0001

Managed Managed CareCare

0.820.82

<.0001<.0001

Quasi- (natural) Quasi- (natural) ExperimentsExperiments

"Estimating the internal validity of a "Estimating the internal validity of a relationship is a deductive process in relationship is a deductive process in which the investigator has to which the investigator has to systematically think through how each of systematically think through how each of the internal validity threats may have the internal validity threats may have influenced the data. Then the investigator influenced the data. Then the investigator has to examine the data to test which has to examine the data to test which relevant threats can be ruled out. . . . relevant threats can be ruled out. . . . When all of the threats can plausibly be When all of the threats can plausibly be eliminated it is possible to make confident eliminated it is possible to make confident conclusions about whether a relationship is conclusions about whether a relationship is probably causal." probably causal."

Cook and Cook and CampbellCampbell

LimitationsLimitations

Could secular changes other Could secular changes other than the policy change than the policy change explain the observed explain the observed differences? differences?

Strengthening the Design Strengthening the Design of a Natural Experimentof a Natural Experiment

Pre/post changes ideally with a Pre/post changes ideally with a comparison group not exposed to policy comparison group not exposed to policy

““Difference in differencesDifference in differences””

Need to establish conceptual basis for Need to establish conceptual basis for selection of specific comparison group selection of specific comparison group

Potential Comparison Potential Comparison Group: Adults in Medi-CalGroup: Adults in Medi-Cal

Medicaid eligibility re-Medicaid eligibility re-determination period did not determination period did not change during study period for change during study period for adultsadults

Therefore, would not expect a Therefore, would not expect a decrease over time in decrease over time in hospitalizations for ambulatory hospitalizations for ambulatory care sensitive conditions among care sensitive conditions among adultsadults

Comparison Group: Comparison Group: Adults in MedicaidAdults in Medicaid

Adults with Medicaid coverageAdults with Medicaid coverage– 1999-2000 = 62%1999-2000 = 62%– 2001-2002 = 60%2001-2002 = 60%

Adjusted relative hazard of a Adjusted relative hazard of a hospitalization for an ACS hospitalization for an ACS condition for adults in post vs condition for adults in post vs pre period= 1.11pre period= 1.11

Second Comparison Group: Second Comparison Group: Children with Continuous Children with Continuous CoverageCoverage

Comparison of children with Comparison of children with continuous coverage in each time continuous coverage in each time period revealed no significant period revealed no significant difference in hospitalizations for difference in hospitalizations for ambulatory care sensitive ambulatory care sensitive conditionsconditions

Suggests no difference in treatment Suggests no difference in treatment approach to ambulatory care approach to ambulatory care sensitive conditions over time sensitive conditions over time period of studyperiod of study

How Do We Know This is How Do We Know This is About Access to Ambulatory About Access to Ambulatory Care?Care?

Hospitalization rates among Hospitalization rates among children for non ambulatory children for non ambulatory care sensitive conditions care sensitive conditions (appendicitis and (appendicitis and gastrointestinal gastrointestinal obstruction) did not change obstruction) did not change over timeover time

Most with Interruption in Most with Interruption in Medicaid Coverage Do Not Medicaid Coverage Do Not Have Alternative for Have Alternative for Ambulatory CareAmbulatory Care At the time of At the time of hospitalizationhospitalization–59% regain Medi-Cal with 59% regain Medi-Cal with admissionadmission

–7% remain uninsured 7% remain uninsured –33% had another form of 33% had another form of insuranceinsurance

Policy ImplicationsPolicy Implications

States need to become more aware States need to become more aware of the hidden costs in their of the hidden costs in their Medicaid policiesMedicaid policies

Continuity of Medicaid coverage Continuity of Medicaid coverage can support better health and can support better health and decrease wasteful spending on decrease wasteful spending on hospitalizations that could have hospitalizations that could have been avoided with less costly been avoided with less costly outpatient careoutpatient care

Translating Research into Translating Research into PolicyPolicy

Results of study used Results of study used – in testimony to California in testimony to California legislature to prevent more frequent legislature to prevent more frequent eligibility re-determination as part eligibility re-determination as part of budget cut processof budget cut process

– in Congress to support Maintenance of in Congress to support Maintenance of Effort requirements as a part of CHIP Effort requirements as a part of CHIP reauthorizationreauthorization

Published in scientific journals for Published in scientific journals for other states to consider in their other states to consider in their policy makingpolicy making

Are There Opportunities Are There Opportunities for Randomized Evaluations for Randomized Evaluations of Health Policies?of Health Policies?

Randomized designs are least Randomized designs are least susceptible to biassusceptible to bias

Political considerations often Political considerations often make this approach impractical make this approach impractical in health policy interventionsin health policy interventions

May be opportunities to use a May be opportunities to use a lottery in implementing lottery in implementing policies that have more demand policies that have more demand than supply (a wait list)than supply (a wait list)

Oregon Health Study: Oregon Health Study: Randomized ImplementationRandomized Implementation

Opportunity for those 19-64 yrs Opportunity for those 19-64 yrs <100% FPL otherwise ineligible <100% FPL otherwise ineligible to obtain Medicaidto obtain Medicaid

85,000 applied but only 85,000 applied but only available for 30,000available for 30,000

Lottery used to randomly select Lottery used to randomly select who gained Medicaid coveragewho gained Medicaid coverage

Oregon Health Study: Oregon Health Study: Analytic PlanAnalytic Plan

Comparisons made in utilization and Comparisons made in utilization and outcomes between those offered outcomes between those offered Medicaid coverage through lottery and Medicaid coverage through lottery and those who were notthose who were not

Intention to treat analysis- some Intention to treat analysis- some offered did not accept offered did not accept – ~10,000 of 30,000 selected enrolled~10,000 of 30,000 selected enrolled

Some not offered may have gained Some not offered may have gained coverage through other meanscoverage through other means

Oregon Health Study: Oregon Health Study: Results Results

Those offered Medicaid were Those offered Medicaid were – 70% more likely to have a regular 70% more likely to have a regular source of caresource of care

– 60% more likely to have a mammogram60% more likely to have a mammogram– 20% more likely to have cholesterol 20% more likely to have cholesterol screeningscreening

Also improvements in self reported Also improvements in self reported health statushealth status

HomeworkHomework

Identify or design a Identify or design a plausible natural experiment plausible natural experiment to evaluate a policy relevant to evaluate a policy relevant to your area of researchto your area of research

Describe data you could use Describe data you could use to study it and possible to study it and possible comparison group(s)comparison group(s)

Not All Policy Changes Not All Policy Changes Make Good Natural Make Good Natural Experiments Experiments

Voluntary Medicare managed careVoluntary Medicare managed care–voluntary implementation can voluntary implementation can have health selection biashave health selection bias

Expansion of public insurance Expansion of public insurance coveragecoverage–uptake by uninsured and uptake by uninsured and ““crowd crowd outout”” of privately insured can of privately insured can make it hard to isolate who make it hard to isolate who got got ““treatmenttreatment”” of insurance of insurance