Carma internet research module: Encouraging responding

Encouraging Responding

CARMA Internet Research ModuleJeff Stanton

May 15-17, 2008 Internet Data Collection Methods (Day 2-2)

Dillman Vs. Groves

Tailored Design - DillmanUse an advance email contact or

pre-screening procedureSecond contact: Email a link to the

questionnaire with a carefully constructed recruitment message; include pre-study incentive

Third contact: Reminder message targeted to non-respondents

Subsequent contacts: Thank you messages, or second reminders

Provide any post-study incentives for participation

Leverage-Saliency – GrovesFollow-up procedures do not

work in a vacuum; their effectiveness depends upon the survey context and characteristics of the respondent

Topic interest moderates the effectiveness of response-enhancing techniques

Community involvement moderates the effectiveness of response-enhancing techniques

Elias Veris: Survey Gamification


Strategies for Encouraging Resp.

Use multiple response enhancing strategies to increase the proportion of the sample that responds. The single most powerful technique for increasing response is having an interesting topic. The second most powerful technique is having interesting questions. Each technique may only increment response rate by a few percentage point. Each increment decreases the potential for bias, although response rates under 95% do not generally eliminate it.

1. Pre-notification2. Publicity3. Instrument Design4. Incentives5. Survey length6. Reminders7. Opportunity8. Response Monitoring9. Establish Importance10. Provide Feedback

Response Enhancing Techniques and Pct Increases in Mail Surveys

Monetary incentives: 12% (Yammarino et al., 1991) to 19% (Hopkins & Gullickson, 1993)

Advance notice: 13% (Roth and BeVier, 1998)Follow-up/reminder: 6% (Yu and Cooper , 1983)Personalization: 9% (Martin, Duncan, Powers &

Sawyer, 1989)Each strategy has been tested and found effective

in Internet surveys, although different improvement %s

Anseel: Org Surveys 1995-2008Advance notice, personalization, topic salience, identification numbers (to

improve anonymity), and sponsorship all have generally positive effectsCombining several response enhancing techniques is advisableResults consistent with leverage-saliency

Consider how respondent characteristics relate to each response enhancing technique

Surveys of managers: Mention sponsorshipSurveys of executives: Mention business relevanceConsumers: Don’t mention sponsorshipNon-managerial employees: Incentives ineffectiveNon-working respondents: Incentives are effective

Personal distribution of surveysAdvance notice and follow-up have become less important over the years

Thorpe et al. 2009 (Family Practice)

Improving response rates when surveying physicians:Used Dillman’s Total Design Method including four

contacts by mail, RR = 48%With the addition of registered mail with return

receipt plus $25 gift cards (unconditional presentation), response rates for subsequent survey studies rose to over 75%

Personally notify your potential participants that they will be receiving a survey in the near future.

Actively publicize the survey. Be sensitive to the actual physical design of your

survey. For example, how questions are ordered may impact respondent participation. – A study by Roberson and Sundstrom (1990)

suggests placing the more interesting and easy questions first and demographic questions last.

Provide incentives, if appropriate. Inexpensive items such as pens, key chains, or certificates for free food/drink can increase responses.

Keep the survey to a reasonable length. A theory-driven approach to survey design helps determine what is absolutely necessary to include in the survey instrument.

Send reminder notes. Response rates may bump up 3-7% with each reminder note, but keep in mind that there's a point of diminishing returns when you irritate people who have chosen not to participate.

Give everyone the opportunity to participate (e.g., paper surveys where required, scheduling time off the phone in the call centers, etc.). At one company for example, most surveys run for 10 business days and span across three work weeks.

Track response rates so that HR generalists and/or the survey coordinators can identify units with low response rates and contact the responsible manager to increase responses.

Foster commitment to the survey effort. For example, you can involve a wide range of employees (across many levels) in the survey development process. Link the content of the survey to important business outcomes.

Provide respondents with survey feedback after the project is completed. Be careful not to abandon your participants once getting the data you wanted from them. You are paving the way (or failing to do so!) for future survey efforts.

Personalization of the survey invitation. Personal signature as part of cover letter.

Topic salience

Even when controlling for the presence of other techniques, advance notice, personalization, identification numbers, and salience, are associated with higher response rates.

Need to do more to just get the same results as the past.

Target facilitation strategy to who you are surveying.– For top executives, salience of the survey topic was

most key. Incentives were counterproductive– Incentives worked for unemployed individuals


Pre-screening w/Active Response

Dear sampled individual:

In one week, we will begin a study on this topic where you will have the opportunity to earn $X for completing a survey that is Y minutes long. Click here if you expect to participate:

http://mysrv.com/screen.htm?participate=yes

Please click here if you plan to decline participation or do not wish to receive the recruitment message:

http://mysrv.com/screen.htm?participate=no


Dealing with Non-Contacts

AdvanceScreening

Cullrefusals

RecruitingMessage

Cullbad

addrs

1st RemMessage

Cullgoodresps

2nd RemMessage

DistributeRewards

Ethics question: Who gets rewards? How much missing

data allowed?

Education

Carma internet research module: Encouraging responding