Motivation for Nonprobability Methods

icf.com

we areCan We Do This Another Way? Potential Nonprobability Sample Sources for Social and Health Survey

Matt Jans, Davia Moyse, Yang Yang Deng, Ronaldo Iachan, Lee Harding, Kristie Healey, James Dayton - ICFScott Worthge – Quest MindshareLaura O'Campo, Sarah Chung - MFour FedCASIC 2021 - Virtual

ICF proprietary and confidential. Do not copy, distribute, or disclose.

Motivation for Nonprobability MethodsEase of data collection Lower cost than probability surveysQuicker to collectUsed in many scientific fields

FedCASIC 2021 - Virtual


Do They Work? Yes, for simple data collection Fit-for-use surveys and studies

Yes, as supplemental sampling frames Though with impact on effective sample size

Maybe, as replacements for probability surveys Different panels can obtain different results (Kennedy, Mercer, Keeter, et al, 2016) Weighting helps

Two possible methods Geofenced sampling Amazon MTurk


https://assets.pewresearch.org/wp-content/uploads/sites/12/2016/04/Nonprobability-report-May-2016-FINAL.pdf


About GeofencingSampling Geographical “fence” drawn around a point Respondents with geo-tracking app invited to a

survey when crossing the fence

Measurement Asking questions in the moment of an activity or

just after Could reduce recall bias and make out-of-reach

topics and experiences measurable


Panel Member Dashboard A Single Question


About the Amazon Mturk Method


Sampling US and international samples

– “Workers”

Measurement Questionnaires (“tasks”) programmed in MTurk or third-party software All the usual benefits of electronic/web questionnaires Workers may be more amenable to unconventional measurement tasks than conventional

respondents


Research Questions1. Can a geofenced sample of grocery, convenience, and home improvement

stores produce demographic and health estimates similar to a gold-standard probability sample health survey?

2. Which estimates?

3. When estimates differ, do they tend to be over- or underestimates, and what is their magnitude?



BRFSS Benchmark Pilot Methodology



BRFSS Benchmark Pilot Geofenced SamplesSampling MFour’s Survey on the Go® mobile opt-in panel Geofenced grocery, convenience, and home improvement stores

– 50m geofence from store entry Reminders at 1hr, 24hrs, and 36hrs

Questionnaire Demographics Health survey topics (wording taken from BRFSS) Image capture (tobacco, alcohol, or sugar-sweetened beverage display or product; or

anything convenient!)



Is MFour Surveys on the Go® Representative?


MFour US

60%

50%

40%

30%

20%

10%

0%Sex

(Female)50+

YearsHispanic/

LatinoEducation

(% College +)Black 35–49

Years25–34Years

18–24 Years

≤17 Years

Race/Ethnicity Age

53%51%

21%18%

14% 13%

35%

48%

41%

27%30%

23%

10%

31%

14%

9%

20% 19%

8%

31%

Marital Status(% Married)

+ + -

-

-

- MFour Underestimates

+ MFour Overestimates


Benchmarking MethodMFour (geofenced) sample weighted to very similar dimensions as BRFSS

MFour (geofenced) estimate is within 95% CI of BRFSS (RDD) estimate (gold standard)

Three Tiers 1. Tier 1: “Nail on the head” (geofenced w/in BRFSS bounds)

2. Tier 2: “Close but no cigar“ (95% CIs overlap)

3. Tier 3: “Close only counts in horseshoes and hand grenades” (95% CIs do not overlap)



Initial Results



Tier 1: “Nail on the head” (Geofenced w/in BRFSS bounds)



Tier 2: “Close but no cigar“(95% CIs overlap)

It’s a long list…



Tier 2: “Close but no cigar“(95% CIs overlap)



Tier 2: “Close but no cigar…Maybe a cigarillo?“(95% CIs overlap)



Tier 3: “Close only counts in horseshoes and hand grenades” (95% CIs do not overlap)Smoker everydaySmokeless tobacco at least some daysAlcohol: Days/week drank; Avg. drinks/day past month, Average drinks is

binge drinking, Binge drank past monthSugar sweetened bev. Times per day, At least 1 serving juice per day, At

least 1 serving sugar sweetened per dayNo health care due to cost



When dissimilar, more “bad health behavior”(over or under as % of BRFSS estimate)


020406080

100120140160180

Geofenced sample overestimates

Geofenced sample underestimates

Double


Next StepsExplore other benchmarking methods

When and where did people respond? When…how far after notification Where…at/near point of sampling v. at home Effects on estimate quality (i.e., “Are any of the Rs similar to BRFSS Rs”)

Compare to MTurk sample or other nonprobability sources

Seeking collaborators to provide sample sources



Thank you!Matt Jans

[email protected] Davia Moyse

[email protected] Dayton

[email protected] 2021 - Virtual

mailto:[email protected]




Three Additional Implementation Case Studies?



Hookah Establishments - Design Target population: Bars and restaurants in New York City that sell hookahGeography: New York CitySample size – 2,500 completes across 75+ NYC Hookah Bars Geofenced sample: ~50 Intercept sample: 2,250 completes Other sample – 200 completes from refusals - self administered cards with study URL provided to intercept

interviewers

Geofencing parameters: Upon panel member exit (3-5 meters from bar entrance) Limit potential for duplicates by turning off geofence when intercept interviews are being conducted Rescind invitation after 24 hours



Hookah Establishments - LessonsWhat Worked Well? Geofencing is inexpensive Modest changes required between the three survey versions Few false positives Inexpensive project supplement

Challenges? Target audience very small - limited benefits Coordination of intercept interviewer schedule with geofence deactivation

Other Takeaways More effective if recall period exceeded 24 hours

– Geotracking can track geolocation prior to current survey date



Hurricane Florence - Design Target population: Panel members in path of storm who also visited home improvement store to repair storm-related damage

Geography: Stores in North Carolina, South Carolina, Virginia and Coastal Georgia

Sample size Geofenced sample: 250

Geofencing use: Target households impacted by Hurricane/Tropical Storm Florence

– Determined nature of repairs needed– Insurance status– Financial hardship



Hurricane Florence - LessonsWhat Worked Well? Existing geofence (used for commercial market research) Geo-targeting look-back to any visit between storm landfall and today Screening for storm-related visit Fast Linkage to pre and post surveys

Challenges? Larger N Not all home improvement locations geofenced Not all repairs completed by DIY



Littering in NYC - Design Target population: NYC Residents age 18-65Geography: NYC’s 5 BorrowsSample size and response rate Geofenced sample: 75 completes from 20 bus shelters, 2 subway exits/entrances, and 3 poster locations Other sample: ~1,000 NYC residents sampled by ZIP Code

Geofencing use: Capture people likely to have seen ads posted about 9 months prior to survey Sampled people who had passed by anti-littering ads at bus shelters, subway stations, and along boardwalk

during the past year



Littering in NYC - LessonsWhat Worked Well? Geofencing to target NYC residents who were likely to have seen anti-littering adds

Challenges? Small sample size for this method due to few geofenced points

– Supplementary, not primary sampling method– Had to increase radius of geofence to gain additional sample mid-field



Case Study SummaryGeofencing can be used in myriad contexts Primary data collection v. supplement

Most useful when sampling involves known geographic points Need a centroid around which to draw the geofence

Potentially representative Skewing younger (including age-related characteristics like marriage) Likely due to mobile device ownership and use Providing devices to participants could reduce this overrepresentation


Documents

Motivation for Nonprobability Methods