Does the Sample Source Matter? Wave 2 of a comparison of leading sample sources

Sept 07, 2012

Sample is the soil and water of research. The quality of it determines the value of what you produce.

We compared other sample sources that VC

regularly uses

We conducted a five minute survey April 12, 2012 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “River C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)

Methodology cont.

The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI), and looked at drop-out rates, straight lining, and the use of “don’t know”.

Sample Quality: Dropout rate during the survey Drop-outs as a percent of all who started the survey

4%

0

8% 10%

7%

16%

0

5% 5% 4%

5%

9%

19%

2%

0%2%4%6%8%

10%12%14%16%18%20%

SBA -Targeted

SBA -Routed

Lightspeed SSI ResearchNow

River UnitedSample

Nov-11 Apr-12

Panel E

Panel D

Panel C

Panel B

Data quality: straight-lining

0.5%

6.5%

7.9%

3.9%

7.8%

0.7%

5.5% 5.6%

4.8%

2.7% 2.1%

3.1%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

SBA SBA Routed Lightspeed SSI ResearchNow

SB River uSamp

NA

Panel B

Panel C

Panel D

Panel E

Data quality: use of “don’t know”

Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".

4%

0

6% 6%

4%

6%

0

4% 4%

5% 4%

4% 4%

3%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

SBA -Targeted

SBA -Routed

Lightspeed SSI ResearchNow

SB River uSamp

Nov-11 Apr-12

Panel B

Panel C

Panel D

Panel E

Differences between SBA and other panels in

how people answered all questions

Count of the number of statistically significant differences (95% CI) as a percent of the total number of questions

4%

2%

3%

7%

4%

2%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

SBA Routed Lightspeed SSI ResearchNow

uSamp RiverPanel B

Panel C

Panel D

Panel E

Expected by chance

Differences between SBA and other panels in

how people answered all questions

While, for the most part, these differences appear within the realm of random, there were some differences that seem to be very real. For example, in both April and November Panel B respondents were less likely to purchase snack products and had lower purchase intent scores for a snack concept.

Differences between panels in how people

answered the purchase intent question

Which statement below best describes how likely you would be purchase this snack if it were available today?—top 2 box: definitely and probably

34%

0

20%

33%

41%

29%

0

31%

39%

30%

44% 46%

40% 44%

0%5%

10%15%20%25%30%35%40%45%50%

SBA -Targeted

SBA -Routed

Lightspeed SSI ResearchNow

River UnitedSample

Nov-11 Apr-12

Panel B

Panel C

Panel D

Panel E

Conclusions

Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test. SBA also shows consistency in how people answered, as illustrated by reaction to the concept.

Conclusions

The same cannot be said for the other panels, however. They showed considerable variation between time 1 and time 2 in how they reacted to the snack concept, which was a non-trendy concept. This may be a concern for tracking. I will dig deeper into the comparison between waves in another report.