Investigating the Potential of Using Non-Probability Samples

Debbie Cooper, ONS

Summary

• Project aims• What constitutes non-probability sampling?• Advantages of probability sampling• Growing interest in non-probability sampling• Types of non-probability sampling• Key challenges• Overcoming challenges • When is the use of non-probability sampling

justified?• Recommendations

Project aims

1. Provide a concise review of the types of non-probability samples

2. Highlight the key challenges associated with non-probability sampling

3. Increase awareness of techniques available to potentially overcome these challenges

4. Provide guidance to help inform decision-making on whether a non-probability sample is justified

What constitutes non-probability sampling?

• Non-probability sampling has two distinguishing characteristics:

one cannot specify the probability of selection for each unit that will be included in the sample

it is not possible to ensure that every unit in the population has a nonzero probability of inclusion 

(Frankfort-Nachmias and Nachmias, 1996)

Advantages of probability sampling

• the ability to calculate selection probabilities allows researchers to create design weights which result in an unbiased estimator

• allows for representativeness as each unit in the target population has a nonzero probability of selection

• allows for the estimation of sampling variabilityBUT

• non-random nonresponse and undercoverage violate the assumptions of probability sampling, giving them a non-probability element

Having said that.....

• Methods developed to deal with coverage and nonresponse issues in probability sampling:

using multiple sampling frames

adjusting weights for nonresponse and, if relevant, attrition

calibrating weights to population totals

Why is there a growing interest in non-probability sampling

• concerns about increasing nonresponse rates

• high costs associated with probability sampling

• ease of carrying out web surveys

• sometimes no other option available (e.g. hidden population)

Types of non-probability sampling

• Convenience/accidental sampling

• Purposive sampling

• Sample matching

• Chain referral methods

Convenience/accidental sampling

“Convenience sampling is a form of non-probability sampling in which the ease with which potential participants can be located or recruited is the primary consideration.”

(Baker et al., 2013)

Types of convenience sampling:

mall-intercept sampling volunteer sampling

Purposive sampling

• Consists of the researcher approaching people who they decide are most appropriate to participate in the study e.g. a sample of experts on a particular topic

Sample matching

• involves selecting a sample that matches a set of population characteristics of interest

• most common type of sample matching is quota sampling

• good estimates of the population characteristics used for matching need to be available

Chain referral methods

• Tend to be used for researching rare or hard-to-reach populations.

Types of chain referral methods:

Snowball sampling Respondent-driven sampling (RDS)

Non-probability sampling: key challenges

1. Greater likelihood of selection bias

2. Impossible to utilise unbiased estimators and associated quality measures

1. Selection bias

“The error introduced when the study population does not represent the target population”

(Delgado-Rodriguez and Llorca, 2004)

Some causes:

undercoverage volunteer bias interviewer/researcher unconscious bias

2. Unbiased estimators and associated quality measures

• Standard practice in official statistics is to use probability sampling and design-based estimation

• Advantages: resulting estimator is unbiasedsampling variability can be estimated directly

These advantages are lost when non-probability samples are used.

Overcoming challenges at:

1. Sampling stage

2. Weighting and estimation stage

1. Overcoming challenges at sampling stage

• Main challenge at this stage is obtaining a representative sample.

• Two popular non-probability sampling strategies developed to obtain a representative sample are:

Sample matching (quota sampling)Respondent-Driven Sampling

Sample matching (quota sampling)

• Aim is to obtain responses from a specific number of units that match the target population

• Quota sampling: interviewers asked to interview a certain number

of people with particular characteristics final sample ‘mirrors’ the target population in

terms of these characteristics

Sample matching problems

• Choice of who to interview still in the hands of the interviewer (unconscious bias)

• Undercoverage

• Nonresponse

• Therefore, recommended to use sample matching along with weighting

Respondent-driven sampling (RDS)

• Used for hidden target populations

• Two distinct phases:

1. Initial sample (the seeds at Wave 0) selected using convenience sampling

2. Rest of the sample is selected by following links from previous respondents

2. Overcoming challenges at weighting and estimation stage

• Different methods developed for use with the various sampling techniques.

Estimators and quality measures when using RDSUsing weightingAdditional quality measures

Estimation and quality measures for use with RDS

“For many years, researchers thought it was impossible to make unbiased estimates from this type of sample. However, it was recently shown that if certain conditions are met and if

the appropriate procedures are used, then the prevalence estimates from respondent-

driven sampling are asymptotically unbiased.”

(Salganik, 2006)

Estimation and quality measures for use with RDS

• Heckathorn (2011) – describes various estimators developed for use with RDS

Caution: these estimators require a number of assumptions to be made and biased estimates may result if assumptions are not met

• Chow and Thompson (2003) proposed a Bayesian approach for estimation for use with link-tracing designs

Estimation and quality measures for use with RDS

• Bootstrap method to construct confidence intervals around estimates (Salganik, 2006)

• Recommends sample size twice as large as that required under SRS

• Interval estimates when using Bayesian approach proposed by Chow and Thompson (2003)

Using weighting with non-probability sampling• Propensity-score adjustments (PSA) to

approximate design-based approach

• Valliant and Dever (2011) approach:construct pseudo design weightsuse covariates from reference survey to adjust these

design weightsCaution: Lee (2006) found that although PSA tends to

reduce nonresponse bias it seems to increase varianceCaution: Lee (2006) recommends that covariates highly

related to study outcomes should be used in the PSA

Additional quality measures

• Credibility Intervals – popular with opt-in panels

• Participation rates

Response rate = no. of respondents total no. of eligible units

Participation rate = no. with useable response total no. of initial invitations

Quality measures for non-probability sampling

• Essential for researchers to report on the quality of their estimates

• Currently no widely acceptable framework for assessing quality of estimates from non-probability samples – development needed

• Important to use different terminology to that used for probability sampling

When is use of non-probability sampling justified?

• Fitness for purpose

• ‘Modellers’ vs ‘Describers’

• No single correct approach

• Decision boils down to desired outcomes and resources

• Communicate clearly the quality of estimates and issues/limitations

Recommendations

• Fitness for purpose should be used to drive survey design

• Non-probability sampling does not necessarily equate to lack of quality

• Transparency is essential

Questions?

Debbie.cooper@ons.gsi.gov.uk