How non-native English speakers respond to English surveys

Please tweet! @PeanutLabsMedia @LoveStatsDIY Sample and PollingPlease tweet! @LoveStats @PeanutLabsMediaPlease tweet! @LoveStats @PeanutLabsMedia

I. Can’t.

Understand. You: How non-native

English speakers

respond to English

surveys

Presented by Annie PettitChief Research Officer at Peanut Labs

DIY Sample and Polling

Please tweet! @PeanutLabsMedia @LoveStatsDIY Sample and Polling

20% of US residents don’t speak English at

home

9% of US residents don’t speak English

well


Survey

•12 minutes, US, Census Rep

•Driving, shopping, electronics, eating out,

working


N=1212

N=228

N=204

Method

Simple Language


DATA QUALITY


Native Speakers Are Click Happy% of respondents choosing many answer options

0%

5%

10%

15%

20%

Q8 Q11 Q16 Q24 Q26 Q30

Somewhat Good Very Good Extremely Good


ESL Speakers Are Click Averse% of respondents choosing minimal answer options

0%

20%

40%

60%

Q8 Q11 Q16 Q24 Q26 Q30



ESL Speakers Choose Red HerringsSounds familiar? Looks familiar?

0%

2%

4%

6%

8%

10%



ESL Speakers Don’t Follow InstructionsFailure rate on “Choose 3 items”

0%

20%

40%

60%

80%

Choose 3 Choose 3



ESL Speakers Choose None of Above% choosing many None of Above answers

0%

4%

8%

12%

1 2 3



ESL Provide Shorter Verbatims% providing shortest responses

0%

5%

10%

15%

20%

25%

30%

1 2 3 4 5

Series1 Series3 Series5Blank % in Q1 % in Q2 % in Q3 % in Q4

Answer Shortest Answers Longest Answers


ESL are really fast or really slow% in Slowest to Fastest Deciles of Responses

0%

5%

10%

15%

20%

25%

1 2 3 4 5 6 7 8 9 10

Poor or Somewhat Good

Very Good

Extremely Good

Slowest Decile Fastest Decile


ESL Responders Make More Errors

0%

5%

10%

15%

20%

25%

30%

1 2 3 4 5 6 7 8 9 10 11 12

Total Number of Errors

Somewhat Good

Very Good

Extremely Good

0 1 2 3 4 5 6 7 8 9 10 11Number of errors made


RESULT OF THESE

DATA QUALITY MEASURES


Higher % of ESL People Cut

Are their

opinions less

important?

Is it fair to cut

them?0%

2%

4%

6%

8%

10%

12%

14%

1 2 3Somewhat good English

Good English Excellent English


Honest ESL People Get Cut

Any final comments on this survey?

• great and interesting survey

• Have nothing for u guy, sorry im

dumb lol lmao XD

• i 'm very active

• I enjoyed this survey

• it was a good survey

• it was very great to fill all ur surveys

• very nice survey, not enough

points

• Well nothing to share now but their

are something friendly way of

living now.

What 3 things do you do with your phone?

• 1..To call someone 2..To know what is

going on around the world. 3..to text my

friends

• coupons, texts, facebook

• i phone, awesome, can download

games and I used iPhone and mobile

• internet and check emails and talk and

text

• Looking information on the internet. Call

the doctors.

• Play games, Watch Youtube, and Watch

read, Talk Text Listen to music

• text. chatting with friends and talking on

the phone


Most Language Discriminatory

Variables Beta Sig

Under Click V1 -.283 .000

Contradict V1 -.150 .002

Can’t Rank -.137 .006

Lies -.151 .011

Under Click V2 -.132 .030

Eliminate

based on

language


Eliminate

based on

quality

Least Language Discriminatory

Variables Beta Sig.Over Click V1 .004 .944Over Click V2 -.005 .927Straightlining V1 -.005 .926Over Click V3 .005 .923Don’t Follow Instructions -.008 .903Verbatim V1 .011 .838Invalid Phone Number .015 .756Contradict V2 -.019 .704


CONCLUSIONS


Recommendations

•Choose less language discriminative data

quality questions

• Overclicking

• Straightlining

• Follow instructions

• Verbatim quality

• Contradictions

• Red herrings

• None of the above


Recommendations

• Avoid cultural references, slang, norms

• Consider forcing minimum responses

• Keep questions short

• Create white space

Have you seen any advertising for this brand?

Please consider TV ads, billboards, internet ads, and other ads.

“Costs an arm and

a leg”


Thank you!

Annie Pettit

Chief Research Officer

[email protected]

ca.linkedin.com/in/AnniePettit/

facebook.com/AnniePettit

twitter.com/LoveStats

Jonathan Cheriff

Director of Sales & Marketing

[email protected]

Find PeanutLabs on

LinkedIn Facebook Twitter YouTube

DIY SURVEY PROGRAMMING DIY POLLINGDIY SAMPLE

Data & Analytics

How non-native English speakers respond to English surveys