Embed Size (px)
Citation preview
1
Summary of Instrument and Question Design Guidelines!
Professor Ron Fricker!Naval Postgraduate School!
Monterey, California!
3/26/13
Reading:!Dillman et al. chapters 4-6!
Goals for this Lecture!
• Review guidelines for instrument and question design!– General instrument guidelines!– General question guidelines!– General response scale guidelines!
• Discuss the types of errors that can contribute to total survey error!
• Understand what aspects of a survey can be tailored to a given situation!
3/26/13 2
Keep the Survey as Short as Possible!
• Minimize total number of questions!– Remember the Mark Twain quote: “I'm sorry this letter is so
long, but I did not have time to make it shorter.”!– Bottom line: If you aren’t going to analyze it, don’t ask it!!
• Corollary: If it doesn’t help answer the research objective, don’t ask it!
• Ensure respondents have ability to respond via chosen survey modes !– Web-based survey: do all potential respondents have
Internet access?!• Else, could result in selection bias!
– Allow for alternate response modes as necessary!
3/26/13 3
Physical Format Checklist!
• If numbering, number all questions sequentially !
• Use large, clear type!– Make it easy to read!
• Use ‘white space’ !– Don’t crowd questions and text !– Place more blank space between questions than between
subcomponents of questions!
• Put special instructions on questionnaire as needed, next to (or within) question!
• Clearly distinguish directions from questions!– Use different fonts, white space, boxes, etc.!
3/26/13 4
Physical Format, Continued!
• For paper surveys, don’t split questions across pages/screens!– If necessary (e.g., question requires 1.5 pages),
restate question and response categories on next page!
• If possible, list response categories vertically instead of horizontally!
• Be consistent with !– Direction of response categories!– Placement of response categories!
3/26/13 5
Self-Administered Surveys!
• Use visual elements in a consistent way to define the desired path through the survey!
• If you must change conventions part way through, prominent visual cues should alert respondents to the change!
• Place directions where they are to be used and where they can be seen!
• Present information that needs to be used together in the same location!
• Ask one question at a time!
3/26/13 6
Organizing the Questions!
• Segment questions by topic!• Ask about related topics together!• Salient questions take precedence over less
salient ones!• Use transitions when changing topics – give a
sense of progress through the questionnaire!• Leave objectionable questions (e.g., income)
for the end!• Put demographic questions at the very end!
3/26/13 7
Guidelines for Writing Good Questions!
• Questions must be:!– Easily and consistently understood!– Easy to answer:!
• Only ask one question at a time!• Break complicated questions/concepts up into a series
of simpler questions!
• Respondents have a common understanding of !– The meaning of each question!– What constitutes an adequate answer!
• Respondents also must:!– Have access to information needed to answer!– Be willing to provide answers called for by questions!
3/26/13 8 Fowler, F.J., Jr., Improving Survey Questions: Design and Evaluation, SAGE Publications and Fowler, F.J., Jr., Survey Research Methods, SAGE Publications.
Question Design Guidelines!
• Survey question design isn’t rocket science !– It’s mainly common sense combined with careful pre-testing
of questions and responses!
• Keep questions as simple and direct as possible!– Break complex queries into a series of questions!
• Avoid ambiguous terms and questions like the plague!– Corollary: Carefully and clearly define complex and
technical terms!
• Only ask open-ended questions if necessary!– Don’t ask respondents to do your work for you!– Exception: As appropriate, give respondents a chance to
provide input not otherwise asked for!3/26/13 9
Specific Response Scale Guidelines (1)!
• Response categories should be consistent with the question!– In the absence of another need/requirement,
default to five-point Likert scale!• Categories must be exhaustive, including
every possible answer!• Categories must be mutually exclusive (no
overlap)!• If appropriate, include a “don’t know” category!
3/26/13 10
Specific Response Scale Guidelines (2) !
• Clearly label the response scale(s)!• Be consistent with the scale labels and
ordering throughout the instrument!• Keep the number of response categories to
the necessary minimum:!– At what level of detail can respondents easily
recall/respond?!– How finely can respondents meaningfully answer?!– What is necessary for the analysis?!
• If “n/a” response needed/appropriate, visually distance it from the rest of the scale!
3/26/13 11
Don’t Forget to Collect Relevant Covariate (Often Demographic) Data!
• Usually it is important to collect covariate data for the analysis!– Depends on the research objectives, of course!
• For example, demographic data!– E.g., for individuals: age, gender, race, marital
status, military service, rank, etc.!– E.g., for organizations: size, type, mission, etc.!
• Usually these are important categories that you want to conduct the analysis by!
3/26/13 12
13
Design Goal: Reducing Total Survey Error!
Source of Error Definition
Coverage ‘...from the failure to give any chance of sample selection to some persons in the population’.
Sampling ‘...from heterogeneity on the survey measure among persons in the population’.
Nonresponse ‘...from the failure to collect data on all persons in the sample’.
Measurement ‘...from inaccuracies in responses recorded on the survey instruments. These arise from: (a) effects of interviewers on the respondents' answers to
survey questions; (b) error due to respondents, from the inability to answer
questions, lack of requisite effort to obtain the correct answer, or other psychological factors;
(c) error due to the weakness in the wording of survey questionnaires; and,
(d) error due to effects of the mode of data collection, the use of face to face or telephone communications’.
ü Figure 2.1 of Dillman et al. (2009) has nice examples of surveys that failed as a result of committing each of the above errors
3/26/13
Reducing Total Survey Error!
• Coverage and sampling errors are reduced with good sample design!
• Measurement errors are reduced with good question and instrument design!
• Nonresponse errors are reduced with good question, instrument, and fielding design and execution!– Focus of the Tailored Design Method!
ü Let’s look at the first two bullets first…!
3/26/13 14
Surveys from a Design Perspective!
3/26/13 15
This
sid
e is
abo
ut w
hat
is b
eing
mea
sure
d
This
sid
e is
abo
ut w
ho
is b
eing
mea
sure
d
Source: Survey Methodology, 1st ed., Groves, et al, 2004.
Where the Errors Come In!
3/26/13 16
This
sid
e is
abo
ut
indi
vidu
al m
easu
rem
ents
This
sid
e is
abo
ut
aggr
egat
e m
easu
rem
ents
Source: Survey Methodology, 1st ed., Groves, et al, 2004.
Developing a Survey Design Tailored for the Situation!
• Each survey situation unique!– Objective(s) and topic(s)!– Respondents!– Sponsor(s) and researcher(s)!
• What works well in one situation may or may not work well in another!
• However, there are proven principles and general strategies!– Must tailor them to each specific situation!
3/26/13 17
Survey Features That Can Be Tailored to the Situation!
• Questionnaire!– Mode, topics, length,
layout and organization, etc.!
• Individual questions!– Topics, type (open,
closed, etc), wording, design and layout, etc. !
• Additional Materials!
• Contact style!– Mode, number,
timing, wording, personalization, etc.!
• Incentive!– Type, amount, timing!
• Sample!– Type and size of
sample!
3/26/13 18
Tailored Design Method Overview!
• Customize surveys for each situation!– Based on knowledge of topic, types of
respondents, budget and other constraints!– Motivates various types of people to respond!
• Establishes trust, maximizes perceived benefits while minimizing costs!
• Seeks to address overall survey error!– Particular focus on minimizing non-response!– But deliberately considers all sources of error!
3/26/13 19
Take Away: Good Survey Design is a Science and an Art!
• For a good survey:!– You must ask the
right questions!– Respondents must
understand your questions!
– Respondents must know the answers!
– Respondents must be willing and able to tell you the answers!
• And remember, always pretest!!3/26/13 20
What We Have Covered!
• In this lecture, we have summarized some guidelines for instrument and question design!– General instrument guidelines!– General question guidelines!– General response scale guidelines!
• Discussed the types of errors that can contribute to total survey error!
• Described what aspects of a survey can be tailored to a given situation!
3/26/13 21
