Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University

Do Response Rates Matter inOnline Panels?

Representativity at Different Levels ofCumulative Response Rates

Johan MartinssonUniversity of Gothenburgwww.lore.gu.se

About response rates

• RR used to be the main indicator of data quality

• Perhaps today more questionable (e.g. Groves &Peytcheva 2008)

• Moving to online surveys and panels, even more so

• Are participation rates for online panels meaningful?

www.mod.gu.sewww.mod.gu.se

Yeager et al. 2011, pp. 731


0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

No

ofsu

rvey

s

Participation rate (%)

Participation rates in 163 surveys from theCitizen Panel at the University of Gothenburg

The most important predictor is the demographiccomposition of the invited sample

www.mod.gu.se

Moving to more controlled comparison: theprobability based part of the Citizen Panel

• The emerging indicator is now Cumulative ResponseRate for probability based panels

• CRR = Recr.Rate * Profile rate * Participation rate

• Both recruitment rates and CRRs are usually low

• And it is not clear how important they are

www.mod.gu.se

Briefly about the Citizen Panel and LORE

• Started in 2010 with support from the University ofGothenburg

• The Citizen Panel has approx. 9,000 members fromprobability based recruitments from population samples

• And approximately 57,000 members from opt-inrecruitments

• You can read more at www.lore.gu.se

www.mod.gu.se

Response rates in web panels

• In 2012 LORE conducted a large probability recruitmentwith an experimental design

• This results in variation in recruitment rates


Recruitment rates in 9 different treatment groups

www.mod.gu.se

Questions

• What happens to response rates and CRR over time ina panel?

• How are they related to accuracy/representativity?

www.mod.gu.se

How to examine representativeness/accuracy

• We calculate average absolute deviations from a goodbenchmark

• 6 demographic indicators– Gender, age, country of birth, marital status,

education, labour market situation– These benchmarks come from Statistics Sweden– Four of these also register data in the sample

www.mod.gu.se

How to examine representativeness/accuracy

• 2 political indicators (quasi benchmarks)

– Interest in politics (from the SOM Institute survey,mail, n=6000)

– Vote intention (Statistics Sweden, RDD, n=9000)

www.mod.gu.se

www.mod.gu.se

1. What happens to CRR over time in a panel?

www.mod.gu.se

www.mod.gu.se

• Initial span 8.4 pct

• After two yrs 2.7 pct

• Higher RR => higherattrition

• Incentives recruitment=> higher attrition

www.mod.gu.se

2. How are RR and CRR related toaccuracy/representativity?

• First, we examine the accuracy at the recruitment

• Next, accuracy over time in the panel

www.mod.gu.se


• At recruitment we have 11 treatment groups withdifferent recruitment rates between 6 and 21 percent

• We examine the average the AAD for the 8 indicators

www.mod.gu.se

All 8 indicators

r=-.61

www.mod.gu.se

r=-.30

All 8 indicators

www.mod.gu.se

Vote intention (8 parties)

r=-.44

www.mod.gu.se

Vote intention (8 parties)

r=-.14

www.mod.gu.se


• We observe a weak correlation at the recruitment stage

• But what about later on? For that purpose we have asmaller set of groups that allow strict comparisons

• We examine representativeness 2 years later (8 panelwaves later)


Standard postcards: all 8 indicators

Standard cardno reminder

Standard card1 reminder

CRR AAD CRR AADWave 0 5.8 9.8Wave 1 4.7 7.3Wave 8 2.7 4.8


Standard postcards: all 8 indicators



CRR AAD CRR AADWave 0 5.8 7.2 9.8 6.9Wave 1 4.7 7.5 7.3 6.6Wave 8 2.7 8.4 4.8 8.2


Standard postcards: 2 political indicators





Incentive postcards: all 8 indicators

Incentive cardno reminder

Incentive card1 reminder

CRR AAD CRR AADWave 0 9.3 14.2Wave 1 6.4 9.5Wave 8 3.6 5.2


Incentive postcards: all 8 indicators





Incentive postcards: 2 political indicators





Did you notice something peculiar?








Did you notice something peculiar?

www.mod.gu.se

Summing up

• High initial recruitment rates deteriorate substantiallyover time and differences diminish

• Higher cumulative response rates consistently yieldlower average errors ...– ... when comparing over time within a recruitment

cohort– ... and when comparing between cohorts at same

wave / panel age• But panel attrition seems to play a part independent of

cumulative response rates (because not at random)• Overall, correlations btw CRR and accuracy seem low• Caveats: small samples for accuracy, unweighted data,

limited variation in recruitment rates and CRRs

www.mod.gu.se

Read more at

www.lore.gu.se

Laboratory of Opinion Research(LORE)

Documents

Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University