John Barrett Rudly Raphael

President, QuestionPro AudienceCEO, Priority Metrics Group

www.pmgco.com

QuestionPro Audience

20 Million Respondents

B2B and B2CSolutions

Mobile Research Panels

10 SpecialtyPanels

• Global Scalability

• Quality Recruitment

• Ongoing quality control

• Panel Respondents

• Business Panels

• Customer Panels

• Real time results

• Mobile community

• Robust platform

• Veterinarians• Building

Contractors• Pet Owners• Registered Voters• And more!

#DataQualitytips

Agenda1. Data quality2. Data quality problems3. Conversion4. Interviewer training 5. Reducing non-response bias6. Questionnaire design7. Question Construction

Improving Data Quality

Importance of Data Quality

Quality Problems – Total Survey Error

Groves (1989) identifies three categories of error:

1. Coverage – some members of the population under study do not have a known nonzero chance of being included in the sample.

2. Measurement effect – the instrument or items on the instrument are constructed in such a way to produce unreliable or invalid data.

3. Non-Response effect – nonrespondents in the survey sample differ from respondents in ways that are germane to the objectives of the survey.

Total Survey Error … for the rest of us

Measurement Effect Non-Response Effect

Data Quality Problems

1.Sampling2.Non-Response3.Questionnaire Design4.Execution/Methodology5.Quality Control6.Analysis/Reporting7.Costs

Poll: Which of the following has a direct effect on data quality, as it relates to your research?Please select the top 3. a. Poor sampleb. Poor questionnaire designc. Poor execution/methodologyd. Poor quality control procedurese. Poor survey incentivesf. Poor open-end responses g. Limited analysis/reporting toolsh. Limited research budget http://buff.ly/2n7Lk30

Non-Response

Non-Response Documentation

• Document non-response using AAPOR categories:❏Interview❏Eligible, non-interview❏Unknown eligibility, non-interview❏Not eligible

• Identify correlates of non-response

Non-Response vs. Maximizing Response Rates

Survey of healthcare leaders: web-based, self-report, four follow-ups• Overall 95% response rate, examined

results in each of five waves

Response Wave and Evaluative Attitudes Assessed Using a Multi-Item Scale: Mean and 95 Percent Confidence Intervals for the Alignment and Commitment Scale (ACS).

Conclusion: Although high response rates are desirable because of their effect on precision and power, absolute thresholds representing “adequate” survey response rates may not be accurate.

Source: “Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare Leaders, “Meterko et al, Public Opinion Quarterly, January 27, 2015

Conversion: Non-Respondent Sees the Light!

✓ Different versions of survey introduction

✓ Study contact rules✓ Incentives✓ Interviewer style and training

Research literature reports telephone conversion rates of 5% to 40%.

Interviewer Training – The Often Overlooked Nugget

✓ Interviewer hiring practice

✓ Thorough interview training❏Survey background❏Questionnaire familiarity❏5-second rule: Introduction, survey purpose❏Timing of contact❏Personality of interviewer

✓ Multiple contacts❏May increase response rates dramatically The right person makes a

difference.

Reducing Non-Response Bias

• Good survey design increases response rates• Good survey communication increases response rates

Incentives can increase response rates

CleverForeshadow

✓ Do not cross “coercive threshold”✓ Donation to charity (business)✓ Prepaid (vs. promised) incentive

(consumer)✓ Differences decrease with follow-up✓ May lead to lengthier open-end

answers

Source: “National Survey of Physicians to Determine the Effect of Unconditional Incentives on Response Rates of Physician Postal Surveys,”Abdulaziz et al, BMJ Open, Volume 5, Issue 2

Reducing Non-Response Bias

Interviewers can gather information about non-respondents

• Observable characteristics

• Evaluations of engagement, honesty, ability

• Visual and verbal clues

• Observable characteristics

• Evaluations of engagement, honesty, ability

• Verbal clues

Questionnaire Design

Questionnaire Design Starts with Clear Objectives

Start End

Good Design Practices

✓ Promise (or at least offer) anonymity or confidentiality

✓ Some form of relationship (known brand or existing customer) increases response rate

✓ Simplify the perceived task• Preserve white space • Design a logical flow• Group common items together – similarity and proximity• Don’t ask unnecessary questions• Minimize task difficulty

✓ Avoid use of jargon and notation

✓ Include relevant questions using question and page logic based on previous answers

Good Design Practices

Improvements

Explanation of task

Question wording

Confidentiality

Positive Elements

Clean, simpleLogical flow

Comment space

Visual Display

✓ No more than four visual elements

✓ Identify new information with distinct colors and/or sizes

✓ Bold type used in favor of italics, underline, or upper case

✓ Use highlights and graphics sparingly

✓ Differentiate important elements with large, bright or distinctive colors

Visual Display

Positive Elements

Logical flow

Improvements

Jargon throughout• Contract• Self-discover• Influencing style

37 visual elements

Everything is brightEverything is newEverything is important

Question Construction

✓ Use conversational norms where possible

✓ Use common words with single primary definition

• Few letters and syllables

• Easy to pronounce

• Avoid abbreviations

✓ Avoid asking certain types of questions:

• Opinions held at prior times

• Explain prior behavior or thoughts

✓ Balance use of closed- and open-end questions appropriately

• Use open-end for numeric answers and categorical questions with unknown breadth of possible answers

Question Construction – Grice’s Maxims

The maxim of quantity, where one tries to be as informative as one possibly can, and gives as much information as is needed, and no more.

The maxim of quality, where one tries to be truthful, and does not give information that is false or that is not supported by evidence.

The maxim of relation, where one tries to be relevant, and says things that are pertinent to the discussion.

The maxim of manner, when one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity.

Question Construction – Response Process

(Tourangeau Model)

Response based on recall or educated guess from cues or inferences, often requires choice of answers to report and how to report

(agree or strongly agree)

Comprehension how respondents understand the

questions and infer the question’s point

Judgment assessment of completeness or sufficiency of

information/opinion, appropriateness and manner of using inferences, and how to

transform retrieved information into appropriate answer

Retrievalrecalling information from long-term memory (behavioral questions) or a

preformed opinion (attitude questions), but retrieval is rarely complete

Question Construction – Bad Examples

1. What is your frequency of utilization of retail travel agents?

2. Are you against the restrictive House Bill 935?

3. Have you visited ___ in the past and did you enjoy your visit?

4. Please rank order the following with 1 being the most important and 7 being the least important

5. When you purchased ____ what other options did you consider?

6. Did you use the features on our website?

7. In order to speed up your shopping experience, have you used the self check-out lanes?

8. How likely are you willing to pay $50 for a ____?

Rating Scales

✓ Label options with words, not numbers

✓ Ensure range of options covers all points on continuum

✓ For bipolar constructs use 7-point scale

✓ For single constructs use 5-point scale

✓ Don’t offer “Don’t Know” response

✓ Rotate order of response on categorical questions and wherever else it is practical

How satisfied were you with the speed of check-out?

Communication Plan

✓ Invitations and advance/introductory letters

✓ Self-administered questionnaires

✓ Reminder letters/emails

Self-Administered

Interviewer-Administered

Computer-Administered

Importance of Pre-Test

Traditional pre-test – match survey method to complexity

Cognitive pre-test – think-aloud

Conclusions – Part 1

To address non-response bias:

1. Document and categorize non-responses

2. Look for correlations

3. Don’t be fooled by high response rate

4. Convert!

5. Hire the best interviewers and train them well

6. Use well-designed questionnaires

7. Use incentives thoughtfully

8. Gather information from interviewers about non-respondents (back to #2)

Conclusions – Part 2

To build the best questionnaires:

1. Start with the end in mind

2. Follow GDPs

3. Establish relationship

4. Use visual elements in limits

5. Craft questions that are clear and direct

6. Test against Grice’s Maxims

7. Think like a respondent (Tourangeau Model)

8. Benefit from research learnings about rating scales

9. Develop and implement a communication plan

10. Pretest, and if necessary, pre-test again

Poll Results

Poll: Which of the following has a direct effect on data quality, as it relates to your research?

Thank You!!!Questions?

 questionpro.com

 @questionpro

sampleprojects@questionpro.comsales-team@qsample.com