Predicting Panel Attrition on a National Study

How We Can Optimize Locating Resourcesand Methods

Lauren M. McNamaraNed EnglishMelissa Heim VioxKatie DekkerRon Hazen

Predicting Panel Attrition ona National Study

•Attrition is non-random– Refusal vs. unlocatable/missing– Focus on participants lost to follow-up

•Certain types of participants more likely tobecome lost

•Goal: Identify high-risk participants before theybecome ‘lost’

Predicting Panel Attrition on a National Study: How We Can Optimize Locating Resources and Methods 2

Problem Statement

Retention challenges longitudinal surveys

• Goal to improve the health and well-being of children

• Examines environmental influences ondevelopment and health

• Broad definition of “environment”

• Congress authorized Study in Children’s Health Act of2000

• Vanguard Phase/Pilot Study• Approx. 5,000 children• Recruited pre-birth or at birth from 2009 to 2013• Recruitment methods: in-person enumeration, provider-based

recruitment, direct outreach


About the National Children’s Study (NCS)

Outcome:Lost To Follow-Up (LTFU)• Unlocatable respondents

Predictors of Attrition• Individual Characteristics• Household Characteristics• Operational Indicators• Interviewer Characteristics*

*Out of scope for this presentation

4

Becoming “Lost”

In-person locating

Interactive locating

Batch locating

Predicting Panel Attrition on a National Study: How We Can Optimize Locating Resources and Methods

Tracing contacts

• NCS population – July 2014• All consented NCS children (N = 5,500) linked to primary

caregivers• 200 (3.6%) lost to follow-up

• Analysis conducted on N=4,700 with complete data• 150 (2.8%) lost to follow-up

• NCS data sources• Last known data point retained for time-varying data• Preliminary, unweighted data

• Geocoded 7,200 household addresses• Calculated total distance moved (Point A to Point B to Point C)• Neighborhood (tract) variables associated with location

• American Community Survey (ACS)

Predicting Panel Attrition on a National Study: How We Can Optimize Locating Resources and Methods

Data Sources

5

Variable LTFU Not LTFU Odds RatioRecruitment Method: Initial – HH Based 6.8% 20.1% 0.29*Recruitment Method: Enhanced Household 31.6% 19.2% 1.95*Recruitment Method: Provider with Area

Sample 20.3% 17.8% 1.18

Recruitment Method: Direct Outreach 35.3% 29.0% 1.34Recruitment Method: Provider Sample 6.0% 13.9% 0.40*Race: Non-Hispanic White 30.8% 61.5% 0.28*Race: Non-Hispanic Black 21.8% 11.5% 2.14*Race: Hispanic 33.8% 17.0% 2.50*Race: Non-Hispanic Other** 13.5% 10.0% 1.41


Descriptive Statistics: Percentages

*Statistically significant, p<0.05**Includes American Indian or Alaska Native, Asian, Native Hawaiian or Pacific Islander, MultipleRace, Other Race

Variable LTFU Not LTFU Odds RatioPrimary Caregiver <25 Yrs Old at Screening 42.1% 20.2% 2.88*High School Diploma or Less Education 51.5% 29.9% 2.49*Resides in Metropolitan Statistical Area 86.5% 81.9% 1.41Married or Living with Partner 75.2% 85.3% 0.52*English-Speaking 81.2% 89.8% 0.49*Currently Employed 34.6% 62.8% 0.31*Household Income ≥ $30k 23.3% 65.8% 0.16*Homeowners 15.4% 56.1% 0.14*Any Tracing Contacts Collected 6.0% 28.5% 0.14*Two or More Addresses Collected (Movers) 31.6% 36.8% 0.79


Descriptive Statistics: Percentages

*Statistically significant, p<0.05

Variable LTFU Not LTFUDistance Moved 128.6 (46.94) 68.0 (5.49)Percent 18+ with High School Diplomaor Less Education in Census Tract 0.5 (0.02)* 0.4 (0.00)

Percent Homeowners in Census Tract 0.5 (0.02)* 0.6 (0.00)


Descriptive Statistics: Means

Data are Mean (Standard Error)*Statistically significant, p<0.05

• N=300 (5.7%) respondents did not provide any address• 18.5% of LTFU• 5.2% of Not LTFU

• Respondents with no address collected were much morelikely to attrite (OR = 4.1, p<0.05)

• These participants were not included in the modeling dueto missing data on address-related variables


No Address Collected

• Respondent-reported variables with high missing rate(>12%) replaced with comparable ACS-derived data (6%missing)

• Education• Home Ownership• Akaike Information Criterion (AIC) and Schwarz Criterion (SC)

support this choice

• Logistic regression models participants’ LTFU status• Full model fit to data• Reduced model selected

• Create streamlined model to score probability respondentbecomes lost to follow-up

• SAS stepwise model selection


Model Building

Variable Odds Ratio Confidence IntervalRace: Non-Hispanic Other 1.9* (1.1, 3.2)Race: Hispanic 1.8* (1.2, 2.7)Resides in Metropolitan Statistical Area 1.8* (1.1, 3.1)Primary Caregiver <25 Years Old atScreening 1.7* (1.1, 2.5)

Two or More Addresses Collected(Movers) 0.5* (0.3, 0.7)

Currently Employed 0.4* (0.3, 0.7)Percent Homeowners in Census Tract 0.3* (0.1, 0.6)Recruitment Method: Provider-Based 0.2* (0.1, 0.4)Household Income ≥ $30k 0.2* (0.1, 0.4)Any Tracing Contacts Collected 0.2* (0.1, 0.4)


Reduced Model



Discussion

• Recruitment Method• Different recruitment methods associated with increased and

decreased risk of attrition• Increased Risk of Attrition

• No Address Collected• Hispanic and Non-Hispanic Other Race• Residency in Metropolitan Statistical Area• Primary Caregiver < 25 years old at Screening

• Decreased Risk of Attrition• Stability: currently employed, increasing percentage homeowners

in census tract, higher income• Movers: collection of addresses indicates engagement?• Any tracing contacts: indicator of engagement, assist locating

• Logistic regression does not account for time-varying data• Most recent reported value used• Multilevel, mixed-effect regression model would be more

appropriate– Sparse and scattered data at the study event level– Preliminary analysis to identify potential important factors

• Missing, discrepant data• Unweighted data, not representative of the general

population• No official definition of lost to follow-up

• Who to include in denominator?• Excluded deceased and withdrawn participants


Limitations

• Can calculate the probability a given respondent willbecome lost to follow-up

• Inform retention methods for participants• Tune locating methods depending on participant• Check in more often, differently• Incentivize move notification

• Direct efforts toward participants at higher risk ofbecoming lost


Application

Thank You!

Contact:Lauren [email protected](312) 325-2553

Variable Odds Ratio Confidence IntervalRecruitment Method: Initial – HH Based --- ---

Recruitment Method: Enhanced Household 2.1 (1.0, 4.5)

Recruitment Method: Provider with Area Sample 1.7 (0.8, 3.9)

Recruitment Method: Direct Outreach 2.3* (1.1, 4.9)

Recruitment Method: Provider Sample 0.4 (0.1, 1.0)

Race: Non-Hispanic White --- ---

Race: Non-Hispanic Black 1.7 (0.9, 3.0)

Race: Hispanic 2.6* (1.5, 4.6)

Race: Non-Hispanic Other (AIAN, Asian, NHPI, Multiple, Other) 2.4* (1.3, 4.5)

Resides in Metropolitan Statistical Area 1.7* (1.0, 3.0)

Primary Caregiver <25 Years Old at Screening 1.5* (1.0, 2.3)

English-Speaking 1.5 (0.8, 2.6)

Married or Living with Partner 1.3 (0.8, 2.1)

Percent 18+ with High School Diploma or Less Education in Census Tract 1.0 (0.3, 3.3)

Two or More Addresses Collected (Movers) 0.5* (0.3, 0.7)

Currently Employed 0.5* (0.3, 0.7)

Percent Homeowners in Census Tract 0.3* (0.1, 0.6)

Household Income ≥ $30k 0.2* (0.2, 0.4)

Any Tracing Contacts Collected 0.2* (0.1, 0.5)


Full Model


• Increased risk of attrition (OR 1.7 – 2.6)*:• Recruitment Method: Direct Outreach• Race: Hispanic• Race: Non-Hispanic Other• Resides in Metropolitan Statistical Area• Primary Caregiver <25 Years Old at Screening

• Decreased risk of attrition (OR 0.2 – 0.5)*:• Two or More Addresses Collected (Movers)• Currently Employed• Percent Homeowners in Census Tract• Household Income ≥ $30k• Any Tracing Contacts Collected


Full Model


• Recruitment Method: Initial – HH Based (ReferenceGroup)

• Recruitment Method: Enhanced Household• Recruitment Method: Provider with Area Sample• Recruitment Method: Direct Outreach*• Recruitment Method: Provider Sample• Race: Non-Hispanic White (Reference Group)• Race: Non-Hispanic Black• English-Speaking• Married or Living with Partner• Percent 18+ with High School Diploma or Less Education

in Census Tract

Footer Information Here 18

Not Significant Predictors

*Statistically significant in full model, p<0.05

• Initial – Household Based• Launched in 2009• Eligibility determined – in part – by

geography of residence

• Enhanced Household• Launched in 2011• Household based (like Initial) with

additional community and provideroutreach

• Eligibility determined – in part – bygeography of residence

• Provider Based Recruitment• Launched in 2011• Recruited through health care

provider;• Eligibility determined – in part – by

geography of residence

• Direct Outreach• Launched in 2011• Used public outreach campaigns• Participants initiated contact with

study

• Provider Based Sampling• Launched in 2012• Sampled physicians and hospitals

serving area


Recruitment Methods

Documents

Predicting Panel Attrition on a National Study