Heterogeneity in Trial Data: Learning from Difference

Rick Rheingans, PhDUniversity of Florida

SHARE Research Consortium

What Works Best? It Depends

• Sector debate at the interface of science and policy• Based on reasonable questions: what will work

best here?

• Differences between studies– Meta-analyses

• Differences within studies– Analytical focus on main effects

• Differences outside of studies

Meta-analyses and Heterogeneity

• Variability across and within studies• Depends on behaviors• Depends on who – higher protection among the most

vulnerable• Depends on initial water quality and other exposures

Sources of Variability within Trials

• Different effect levels in different sub-populations due to behavior or vulnerability– Opportunity for targeting

• Spatial differences environmental conditions affecting exposure– Opportunity for geographic targeting

• Differences between settings based on implementation– Opportunity to adjust adjust the intervention

Analytical Tools for Teasing Out Difference

• Random effects models– Did the intervention work differently in different

communities – especially for cluster randomized trials• Effect modification– Are there characteristics of individuals or

communities that change the impact of the intervention

– Stratification to look at discrete populations

• Focus is usually on the main effect

Grappling with Differences: School Water, Sanitation and Hygiene Impacts

• SWASH+ – Collaborative applied research and advocacy project led by

CARE in western Kenya• Cluster-randomized trial in 185 schools– Included hygiene promotion, water treatment, sanitation

infrastructure, and water supply• Objective:– Estimate the impact of school WASH interventions on

health (helminthes and diarrhea), educational outcomes (absenteeism and performance), and behaviors (e.g., diffusion to homes)

What’s the Question?

• National and global policy and advocacy interest in estimating the main effects– Days of absence avoided– Percent reduction in diarrhea

• Compare it to other school investments• Compare it to other WASH investments

• What if the most important answer is - it depends?

Differential Impacts of School Water, Sanitation and Hygiene

• Absenteeism (Freeman et al, 2011)– Strong impact for girls (Odds Ratio 0.4), no measureable impact for boys

• Helminths reinfections - – Differences by gender

• Ascaris for girls; especially poorest• Hookworm for boys; especially poorest

– Differences by behavior• Reduced hookworm reinfection among boys without shoes

• Diffusion of behavior change (water treatment) to homes– Strongest effect among the poorest households

• Differences between schools and regions

• Conduct across 3 Districts in western Kenya

• Differing socio-economic and exposure conditions

Trying to Explain Differences in School-Cluster Performance

• Reveals challenges in sustaining hand washing facilities and water treatment

• In compliance adjusted analysis, both having HW facilities and treated water are associated with reduced absence

Trying to Explain Differences: New Pathways

• In schools receiving new latrines, children had increases in fecal hand contamination

• Suggests – Importance of latrine cleanliness– Interdependence of hand-washing and sanitation– Need for anal cleansing materials

Implications?

• What to invest in: – De-worming? – School uniforms? – More teachers? – School WASH?

Different Conditions and Impact Variability: A Hypothetical Exercise

• Overall impact estimates provide us with the ‘average’ setting, but what will it be in a particular setting?

• Assume a setting where on average 35% of under-5 diarrhea preventable through improved sanitation

• Part of diarrhea burden is due to non-sanitation related exposures

• Some due to whether the household has sanitation

• Some due to whether they have to share that facility

• Some due depending how community’s coverage

Different Conditions and Impact Variability: A Hypothetical Exercise

• Overall impact estimates provide us with the ‘average’ setting, but what will it be in a particular setting?

• Assume a setting where on average 35% of under-5 diarrhea preventable through improved sanitation

• Part of diarrhea burden is due to non-sanitation related exposures

• Some due to whether the household has sanitation

• Some due to whether they have to share that facility

• Some due depending how community’s coverage

Household

Other

Comm35% Preventable

What Happens with Greater Community Exposures

• If there is heterogeneity in level of community exposure –– Would severe

diarrhea rates go up?

– Would the preventable fraction with sanitation go up?

Household

Other

25% Preventable?

Household

Other

Comm

Household

Other

Comm

Household

Other

Comm35% Preventable

50% Preventable?

65% Preventable?

Variability in Community Level Exposure

• One measure may be population density of people without sanitation

• Based on cluster-level coverage and population density

• Varies widely within countries and provinces

Variability in Community Level Exposure

• One measure may be population density of people without sanitation

• Based on cluster-level coverage and population density

• Varies widely within countries and provinces

Does sanitations impact change?

Other Sources of Differences

• Could also consider heterogeneity in vulnerability (e.g., nutritional status)– Increased odds diarrhea mortality with low

weight for age (Caulfield et al, 2004)– Increased risk of illness, for a given exposure– Increased risk of mortality, given illness

• May not affect the fraction preventable through sanitation, but would increase the number of severe cases preventable

Differences in Impact?

Median 16.7%

Median

Differences in Impact?• How does

sanitation impact vary across the space?

• Could heterogeneity in impact trial data help us understand how much?

• Same is likely true for other WASH interventions

Salvador, Brazil Sanitation Trial

• Not a randomized trial – repeated cross-sectional study before and after city-wide sanitation project (Barreto et al, 2010)

• Took advantage of different levels of household and neighborhood change to estimate impact on childhood diarrhea and helminth infections

• Lessons from heterogeneity within the trial– Changes in community sanitation coverage were more important

than whether households received a connection– Impacts on helminth reduction were strongest among the poorest– Showed that intervention reduced the impact of SES on diarrhea

disparities

Trial Heterogeneity and Generalizability

• Trials often focus on settings with high levels of burden and homogeneity– Increase the measureable impact– Reduce the size of the intervention needed

• However lack of heterogeneity within the trial can make it hard to generalize to a broader setting– External validity

Example Deworming and Soil Transmitted Helminths

• Miguel and Kremer tested the impact of deworming for STH on educational outcomes in western Kenya

• Found that deworming can significantly reduce absenteeism (Miguel and Kremer, 2004); spillover effects; and increased long-term earnings (Baird et al, 2011)

• However the prevalence of STH was uniformly high within the study, compared to the rest of the country

Translating to Heterogeneous Settings• Pullan and colleagues developed spatial estimates

of national burden to identify where mass treatment would be most appropriate

Schistosomiasis Control in China

• Liang et al examined the impact environmental and chemotherapy interventions for Schisto control

• Developed mathematical models of transmission

• Used data from intervention trials to calibrate the models in different settings

• Identified patterns for generalizing

Using Variability to Make a Difference

• Getting more out of trials– Analyzing factors modifying intervention effect

– Better characterizing mechanisms – connecting interventions to outcomes• Exposure and environmental studies

• Modeling and new analytical techniques

– Deliberate attention to external validity and generalizability in trial design

• Better translation– Using non-trial outcome data to better understand what happens in

more diverse settings

– Better characterizing contexts for which we would like to know the effect of interventions

– Better policy signals - to encourage more effective intervention selection