The Promises and Perils of Pushing Respondents to the Web ...sondages2016.sfds.asso.fr/wp-content/uploads/2016/... · The Promises and Perils of Pushing Respondents to the Web in

The Promises and Perils of Pushing Respondents to the

Web in Mixed-Mode Surveys For

9th Colloque Francophone sur les sondagesGatineau, Quebec, Canada

11 au 14 Octobre 2016

ByDon A. Dillman, Regents Professor

Department of Sociology and the Social and Economic Sciences Research Center

Washington State University, Pullman, WA 99164-4014 [email protected]

mailto:[email protected]

A brief look to the past

• In 1989 an iconic American movie, Field of Dreams, was created around the idea that if a baseball field were to be built in a rural Iowa corn field, “They (the players and people) will come.”

• In the late 1990’s a similar idea developed about conducting surveys by email with responses sent over the Internet.

• Hopefully, thepeople would not only come, but the cost of surveying would decline dramatically, making it a surveyor’s field of dreams.

• This reality has yet to be fully realized.

c Don A. Dillman, September 1, 2016 2

Three parts to this presentation

• Part I. The decline of telephone and difficulty with finding alternatives.

• Part II. A decade of research aimed at developing an effective web-push methodology.

• Part III. Specific challenges associated with the promises and perils of web-push surveying.


My specific focus

• My focus in this talk is on probability sampling of the general public.

• In this context I am also focused on simultaneously reducing four types of error: coverage, sampling, nonresponse and measurement.

• By web-push I am referring to initially asking people to respond only over the web, while reserving alternative response modes until later in the data collection process.

• Mixed-mode refers to using more than one survey mode as contact and/or response mode.


Part I.

The Decline of Telephoneand difficulty with finding

alternatives


Around 2005 it became clear that the voice telephone era of RDD household surveys

was coming to an end

• The presenting problems for RDD surveys were:– Low response rates.– Need to combine individual (cell) and household (land) lines.– Need for brevity.– Need to devote questions to correcting for area code transportability,

ownership of cell/landline access, not driving a vehicle, age of respondent.

– People answered phone only once, so that it is becoming a “one chance to persuade” (in 5 seconds or less) methodology

• However, cultural change is the biggest problem; the telephone is no longer used for ordinary conversations.

• Most telephone communication is asynchronous; texts and emails.


The hoped for Email contact/web response replacement, is not yet working

well. • Coverage for household populations remains

limited (<80%), some on slow connections.• Individual internet skills were limited among some

adults, especially the less educated, older and lower income individuals.

• No email sample frame for household selection comparable to telephone RDD.

• Email only contact produces response rates similar to phone.


The hoped for Email contact web response replacement, is not yet working well (2).

• Lack of TRUST on multiple levels; sponsorship, misrepresentation of purpose, threat of malware, and consequences of making a mistake.

• We are moving from age of desktops, to laptops, (and now) to devices, which are not optimally designed for answering surveys—screen size affects when and how they get used.

• The “purse and pocket” problem combined with the smartphone response situation of people being “on the move” from one task to another makes getting lengthy and thoughtful survey responses very difficult.


I am uncomfortable with some of the assessments and proposed solutions

• “Response rates no longer matter” (Higher the response, the better our chance of going after non-responders)

• “All response modes are facing declining response rates” (less true for mail than telephone or the Internet)

• “Repair the telephone” (Too many problems: coverage, brevity, extra questions, minimal opportunity to persuade, CULTURE moving away from voice exchanges)

• “Give households the Internet and rely on web-only panels” (Too many people don’t trust or are uncomfortable with computers and internet; results are biased)

• “Give people a choice of modes” (Most choose mail)


Postal addresses have emerged as sample frame with most complete household

coverage• Early focus (2007) was on drawing area-probability samples for personal

interviewing.• I am struck by the high coverage rates for households—98 %—in the U.S. Postal

Service “Delivery Sequence File” available as addresses without names.• If this could work for personal interview samples, why not mail surveys.• Reason for my interest in mail surveys was:

– I could deliver incentives with the survey request (much more effective that post payments)– I could deliver follow-up contacts in different forms to offer additional arguments for

responding.– Research done in the 90’s at the Census Bureau convinced me that we didn’t need names

for mail methods to be effective.– I had not seen evidence of a precipitous decline in mail response rates as happened for

telephone.– I could try to “push” people to the web instead of asking for a paper response.


In order to replace telephone with a web-push methodology…

• We needed a better sample frame than “random email addresses”. Postal mail appeared to be the only realistic choice.

• Using postal addresses would require mixing survey modes:– a different contact mode than response mode. – And might involve obtaining responses from more than one mode (e.g.

mail or telephone)– Mixing modes could introduce measurement issues

• Potential cost savings provided a powerful motivation for finding ways to make this work.

• Acceptance by the survey methodology community could be challenge because of losing “interviewer assistance”, our standard way of eliciting answers to survey questions.


Part II.

A decade of research aimed at developing an effective web-

push methodology


The Prototype (Lewiston-Clarkston) Experiment

• Address-based sample.• Push-to-web with mail contact (send mail response option in

third contact). • Undo old mail strategies, e.g. name personalization, envelopes,

use of postcard.• Token cash incentives with response request• Unified mode construction to reduce measurement differences.• Tailor to population with graphic design and color.• Reasonable length: At least 20 minutes of questions• My background model was, “What I thought could be applied

to the American Community Survey”


Conceptually, I was trying to undo earlier mail procedures (1978-2000) and use the visual design and mixed-mode concepts

introduced in (2000 and 2009) versions of book. Each revision was substantial (see% below). New concepts are coming at us faster and

faster

1978 2009~75%

2014~65%

2000~95%


c Don A. Dillman, September 1, 2016

2007 Lewiston-Clarkston Study

• 51 numbered questions, 90 responses, 10 pages (paper)—a 20 minute (if it were telephone) conversation

• Four contacts.1. Pre-notice letter.2. Questionnaire (or web request).3. Thank-you post card.4. Replacement questionnaire (adjusted by treatment).

• $5 token cash incentive included with mail questionnaire or web contact

(Later studies would change number and nature of contacts)

15

We compared four treatments

1. Mail preference with web mention: Send mail questionnaire and mention web with initial request

2. Push-to-mail: Send mail questionnaire but withhold mention of web for about two weeks

3. Push-to-web: Web invitation with no mail questionnaire, but explain that mail questionnaire will be sent in about two weeks

4. Equal preference: It is your choice!


Larger plain envelopes (even for web-only request) to get them opened

• Normal business stationery more likely to be ignored.• Used a return label showing the photo from survey cover and the

survey title to increase familiarity.

© Dillman17c Don A. Dillman, September 1, 2016

Correspondence tailored to city address

• All letters used WSU stationary (legitimation).

• $5 affixed to letter (to get it read)

• Photo of questionnaire cover used to tie different elements together (interest).

18c Don A. Dillman, September 1, 2016


A picture of the Lewiston and Clarkston region identified the survey area (confluence of the

Snake and Clearwater Rivers)

19


Additional pictures identified the survey area

• Photos taken of local landmarks, artwork, and symbols to make survey recognizable and visually attractive.

20


Mail Questionnaire tailored with broad survey topic, who should respond, and back-page pictures

Use of tailored images to help connect respondents to survey and to place an emphasis on study area instead of on survey source. 90 responses requested

21


Web questionnaire was similarly tailored

• Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images.

22


Web survey followed unified mode construction principles to link modes

• Question 2– Similar design format to paper survey, and use of familiar image in upper left-hand corner of

the screen. Emphasized respondents over sponsors (using other communications to do that).

23


Second example of unifying web presentation with mail presentation

Question 13 The small image would change every ten questions. Trying to show all of the

same images as the paper survey, as well as keep interest and familiarity.

24

Mail (on left) and web (on right) unified to control measurement


Initial withholding of mail drove 41% to the web; paper follow-up added 14%. Offering

choice drove ~80% to mail .

Treatments Web (%) Paper(%) Total (%)

Mail preference with web mention

4 58 62

Push-to-Mail(web in third contact)

1 70 71

Push-to-webMail questionnaire sent in 3rd of 4 contacts

41 14 55

Equal preference (choice) 13 50 63


High response rates are desirable but we need also to focus on nonresponse error

• It does not help much to improve response rates if our respondents are different from non-respondents on variables important to the study objectives

• Thus, we need to compare respondent characteristics on web vs. mail within the different treatment groups.

• If respondents are different from non-respondents in ways that affect results of the data analysis, then we have a significant problem


In the push-to-web treatment, web and mail respondents

were quite different


0

10

20

30

40

50

60

70

80

90

100

* p ≤ .05

** * *

*

*

*

28

But the complete push-to-web group was quite similar to the complete

push-to-mail treatment


*

* p ≤ .0529

0

10

20

30

40

50

60

70

80

90

100

Push-to-web

Push-to-mail

Initial success encouraged us to do four more large-scale tests to evaluate the web-push methodology

0 Lewiston, ID-Clarkston, WA Mail Test 20051. Lewiston, ID-Clarkston, WA Survey 20072. Washington Community Survey 20083. Washington Economic Survey 20094. WA, PA, AL Tri-state Electricity Survey20115. WA and NE Water Management Survey2012• Needed multiple topics and populations• Research goal was to refine through subtraction and addition experiments,

e.g. token cash incentives, other state populations, questionnaire design, respondent selection, location of sponsor, etc.)

• ~ 28 additional experiment treatments, with successful panels being carried forward.

For summarized detail, see Chapter 11 of Dillman, D., Smyth, J.D., Christian, L.M. 2014. “Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition. (John Wiley, Hoboken, NJ)


Tests 2-5; Changes in burden, topic, and procedures (e.g. no pre-notice, no postcard reminder)

• Questionnaire— 20-30 minutes to complete. (equivalent to ten 8 ½ x 11” pages of paper. Up to 140 individual questions

• General Implementation Protocol : specifics varied somewhat across studies.Week 1: Postal request that includes $4- $5 token cash incentive with

request to respond by web.Week 2: A thank you/reminder letter (not postcard).Week 5: Another letter request with a mail questionnaire and, in some cases, an additional smaller token cash incentiveWeek 6: A final thankyou/reminder.


2008 Washington Community Survey (WCS): first statewide test

• Example of the mail version:


A $5 cash incentive (WCS) more effective for web-push than paper-only

$5 Mail Mail w/o $50

10

20

30

40

50

60

39,2

Mail

$5 Internet Internet w/o $50

10

20

30

40

50

60

31,3

13,4

(+17.9*)

(+13.3*)

* p ≤ .05 c Don A. Dillman, September 1, 2016 33

Push-to-web had lower total response, but 2/3 of responses came over the web. For push-to-mail, a

late web request had very small effect.

$5 Push-to-web w/card $5 Push-to-mail $5 Mail-only0

10

20

30

40

50

60

31,3

51,9

$5 Mail-only; 0

15,0

3,1

56,7


46%

55%57%

34

Introducing a paper questionnaire in third contact brought in different kinds of respondents than did the initial push-to-web effort. (Messer

and Dillman, 2011)

Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)0

10

20

30

40

50

60

70

80

90

Web Mail follow-up

**

** *

*

* p ≤ .05


The combined push-to-web group (web plus mail respondents) was demographically

similar to the mail-only treatment


Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)0

10

20

30

40

50

60

70

Push-to-web Mail-only

Complete push-to-web group was more representative than the web-only

respondents(comparison to U.S. Census Bureau’s American Community Survey)


Education (HS or less)Children in HH # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)5

15

25

35

45

55

65

75

Web-only Push-to-web ACS

2009 statewide economic survey, tailored to state with map and pictures

• 11” X 8.5” booklet with ‘personalized’ images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey. (Messer and Dillman, 2012)

38c Don A. Dillman, September 1, 2016

2011 Three state Electricity Survey, with tailored covers

• Examples of the mail covers:


2012 water management survey: tailored to state by pictures and sponsorship• Examples of the mail covers:


Response rates for 10 statewide web-push (mean, 43%, with 62% over the web) vs. mail out/mail back (mean, 53%) surveys,

2007-2012 (Dillman, Smyth and Christian, 2014, Chapter 11).


Does mail item non-response negate gains in response rates?

• If mail higher item non-response rates then perhaps that cancels out the benefits of gaining the additional responses in web-push designs.

• Thus, it was important to evaluate this. (See Messer, Edwards and Dillman, Survey Practice, 2012).


Web vs. mail item non-response in the push-to-web groups for LCS, WCS, and WES

• Item nonresponse rates lower for web

2007 LCS 2008 WCS 2009 WES0

2

4

6

8

10

12

14

2,7 2,7

6,16,26,9

11,6


BUT, push-to-web and mail-only groups have similar overall item nonresponse rates; Late mail

respondents in push-to-web groups are probably less capable respondents(older with less education).

2007 LCS 2008 WCS 2009 WES0

2

4

6

8

10

12

14

3,64,2

8,0

5,04,2

8,1

Push-to-web


Summary of Major findings 1. 10 Postal-only response = 53% (38% to 71%). 2. 10 Push-to-web response = 43%(31% to 55%).3. Offering initial choice pushes most people (~80%)

to mail.4. Web-push produces about 60% of responses

over the web.5. Token cash incentives with request significantly

improves web and overall response rates.


Summary of Major findings (2)

1. Significant differences exist between web and mail responders (education, age, income, marital status).

2. Web+mail treatment respondents are quite similar to all mail-only treatment respondents.

3. Item non-response to mail is a less of a problem than we had anticipated when visual design principles are applied.

4. Response decreases when surveying in distant states.5. Web-push costs slightly more per respondent than mail-only

because of lower response rates.


Web-push methods being used for many national surveys

• 2015 Japanese Census.• American Community Survey (3.5million

households per year) • Screen and collect data for U.S. child education

survey.• Screen and collect household data for U.S. child

health survey.• 2016 Australian Census • 2020 U.S. Census


Part III.

Additional challenges associated with the promises

and perils of web-push surveys


Promises and Perils of Web-Push

• In the long run, we need to push respondents to the web, but we need to do it with eyes wide-open.

• There are significant promises to be realized.• There are also some perils in going this

direction that we need to contemplate


12 ways Web-push exhibits promise

1 Address-based sampling provides superior coverage.

2 Response rates to web-push are dramatically higher than RDD telephone.

3 Cost reductions seem likely.4 Quicker responses from “most” people seems

likely.5 A better fit with society’s “do-it-myself” culture.


Web-push exhibits promise (2)

6 Reduction in socially desirable answers (vs. interviews)7 Reduction in extremeness of scalar responses.8 More effective branching (vs. mail)9 Initial mail contact increases feelings of trust.10 Pre-incentives improve likelihood of response.11 Two-phase data collection may be collapsed into one

phase (screen, plus longer questionnaire).12 Better fit with organizational preferences and

pressures.


Web-push perils calling for research and innovation

1 How to overcome “irritation” of not providing immediate choice.2 Pre-incentives needed for getting people to transfer across modes

(i.e. mail contact to web response).3 Decisions on communication access have switched from

organizational control to individual preference and behavior.4 Pocket/Purse devices are dominant (and perhaps only) interface

some people have with computers.5 Web response is not sufficient for most surveys—bias towards

higher education, income and younger households with kids.6 Mail follow-up to get balance faces constraints—branching plus

need for unified mode construction.


Additional web-push perils calling for attention (2)

7 Mixing of response modes are likely to create measurement differences, if allowed to ask questions in different ways.

8 Switching to mail in 3rd or 4th contact is not being handled well, e.g. still emphasizing web, minimal incentive and/or follow-up.

9 Not developing coordinated or mutually supportive communications focused on the process of responding


Additional web-push perils calling for attention (3)

10 Not treating mail and web as help-mates. Respondents can use mail to think through answers before entering them on web, but different construction makes that difficult.

11 Are we on a slippery slope because of do it yourself software? Let’s use only people contactable by emailLet’s only obtain web responsesIncrease sample size because of low responseTreating large sample sizes as if they have sampling error.Use of weighting to create data for underrepresented

categories.A catch, release, and catch-again mentality develops.


Some additional steps

• Introducing more effective mail follow-up to the initial web-push strategy, e.g. multiple contacts with additional incentives to get those communications read.

• Dealing with unified design between mail follow-up and smartphones that are requiring redesign of laptop/desktop surveys.

• Dealing with branching issues in visual design of paper surveys.

• Is there a role for telephone follow-up despite social desirability and aural vs. visual design effects?

• There is a lot more that can and needs to be done.


Selected references

1. Smyth, J.D., Dillman, D.A., Christian, L.M., & O’Neill, A. (2010). “Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century.” American Behavioral Scientist 53: 1423-1448.

2. Dillman, D.A., Smyth, J.D., Christian, L.M. 2014. Internet, Phone, Mail and Mixed-Mode Surveys; The Tailored Design Method 4th edition. (John Wiley Co Hoboken, NJ)

3. Messer, Benjamin L. and Don A. Dillman. 2011. “Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures.” Public Opinion Quarterly 75(3):429-57.

4. Millar, Morgan M. and Don A. Dillman. 2011. Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly 75 (2): 249-269

5. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): 734-750.


Selected references, page 2

6. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): 734-750.

7. Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012). “Determinants of Web & Mail Item Nonresponse in Address-Based Samples of the General Public.” Survey Practice, April:. http://wwww.surveypractice.org

8. Messer, Benjamin L. 2012. “Pushing households to the web: Results from Web+Mail experiments using address based samples of the general public and mail contact procedures.” Ph.D. Dissertation. Washington State University, Pullman.

8. Edwards, Michelle L. 2013. “Measuring Public Perceptions of Water Governance in Nebraska and Washington.” Ph.D. Dissertation. Washington State University, Pullman.


http://wwww.surveypractice.org/

Selected references page 3• Dillman, Don A. and Michelle L. Edwards. In Press. Chapter 17. Designing a Mixed-Mode

Survey. Sage Handbook of Survey Methodology. Sage Publications Wolf, Joye, Smith and Fu. Thousand Oaks. CA

• Dillman, Don A., Feng Hao, Morgan M. Millar. In Press. Chapter 15. Improving the

Effectiveness of Online Data Collection by Mixing Survey Modes. In Fielding, Nigel, Raymond M. Lee and Grant Blank. The Sage handbook of Online Research Methods, 2nd edition. Sage Publications, London.

• Harter, Rachel, Battaglia, Michael P., Buskirk, Trent D., Dillman, Don A., English, Ned, Mansour Fahimi, Frankel, Martin R., Kennel, Timothy, McMichael, Joseph, McPhee, Cameron Brook, Montaquila, Jill, Yancey, Tracie, Zukerberg, Andrew L. 2016. Address-base Sampling. American Association for Public Opinion Research Task Force Report 133 pp. http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL.pdf


http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL.pdf

http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL.pdf

Thank you!For additional information contact:

[email protected] Wilson-Short Hall

Washington State UniversityPullman, WA 99164-4014

[email protected]

mailto:[email protected]

Documents

The Promises and Perils of Pushing Respondents to the Web ...sondages2016.sfds.asso.fr/wp-content/uploads/2016/... · The Promises and Perils of Pushing Respondents to the Web in