Crafting Good Questions - 2-2 -- Crafting Good Questio¢  Crafting Good Questions! Professor Ron Fricker!

  • View
    4

  • Download
    0

Embed Size (px)

Text of Crafting Good Questions - 2-2 -- Crafting Good Questio¢  Crafting Good Questions!...

  • 1

    Crafting Good Questions!

    Professor Ron Fricker! Naval Postgraduate School!

    Monterey, California!

    6/24/12

  • 2

    Goals for this Lecture!

    •  Guidelines for writing effective questions! –  Lots of dos and don’ts! –  Mistakes and errors to avoid when writing

    questions! •  Guidelines for writing effective response

    scales! –  More dos and don’ts!

    Ø Goal is to advance our understanding of what makes a good survey question and how to write such questions!

    6/24/12

  • Question Components!

    6/24/12

    3

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Guidelines for Writing Questions!

    •  Lots of pithy dos and don’ts: use simple words, keep it short, be specific, etc.! –  Often they are too simple and/or contradictory!

    •  That said, here are some of Dillman’s guidelines that he has… !

    “…found useful in making the wording 
 and structural changes necessary for 
 turning an initial draft of respondent 
 queries into acceptable survey questions.”!

    6/24/12

    4

  • Use Simple and Familiar Words!

    6/24/12

    5

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Example!

    6/24/12

    6

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Don’t Be (Needlessly) Wordy!

    6/24/12

    7

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Keep the Questions as Short as Possible!

    6/24/12

    8

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Use Complete Sentences!

    6/24/12

    9

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Use Wording that Specifies 
 What You’re Asking for Clearly!

    6/24/12

    10

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Make Sure Question is Accurate!

    6/24/12

    11

    Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman, 2006.

  • Ask One Question at a Time!

    6/24/12

    12

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • 13

    Only Ask One Question at a Time (cont’d)!

    6/24/12

    Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.

  • Only Ask One Question at a Time (cont’d)!

    6/24/12

    14

    Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2009.

  • Avoid Double Negative Phrasing!!

    6/24/12

    15

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Make Sure Questions 
 Apply to All Respondents!

    •  All questions should require an answer of all respondents! –  Else, use a

    screening question and skip!

    •  This is not the same as saying all respondents must answer all questions!!

    6/24/12

    16

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • State Both Positive and 
 Negative Sides in Question Stem!

    6/24/12

    17

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Integrate Instructions into Question Stem!

    •  Put instructions at the point where respondents need them! –  Avoid putting them in a

    separate section up front! •  For questions, integrate

    into the question stem! •  For other instructions,

    integrate then into the survey where they are relevant!

    6/24/12

    18

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • 6/24/12

    19

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

    Avoid Bias From Unequal Comparisons!

  • 6/24/12

    20

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

    Avoid Bias From 
 Unequal Comparisons (cont’d)!

  • A Bit About Response Scales!

    •  There are many possible ordinal scales:!

    •  In structuring response scales, must balance:! –  Analytical needs (precision of information

    required)! –  Ease of respondent recall and answer!

    6/24/12

    21

    Fowler, F.J., Jr., Improving Survey Questions: Design and Evaluation, SAGE Publications.

  • The “Likert” Scale is Very Common!

    •  Wikipedia: “A Likert scale (pronounced 'lick-ert') is a type of psychometric response scale often used in questionnaires, and is the most widely used scale in survey research. When responding to a Likert questionnaire item, respondents specify their level of agreement to a statement.”!

    •  Five-point scale is frequently used, but it can vary! •  Example: How much to you agree or disagree with the

    following statement: Ice cream is good for breakfast! q Strongly disagree ! q Disagree ! q Neither agree nor disagree ! q Agree ! q Strongly agree !

    6/24/12

    22

    http://en.wikipedia.org/wiki/Likert_scale

  • Provide Balanced, 
 Evenly-Spaced Response Scales!

    6/24/12

    23

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Constructing Response Scales!

    •  Number of points in scale should be determined by how you plan to use the data!

    •  Respondents can generally only remember a maximum of 5 responses unless visual cues are used!

    •  With scales with few points, every scale can be labeled! –  In longer scales, only label midpoint and endpoints!

    •  Graphic images such as thermometers and ladders can be effective!

    6/24/12

    24

  • Poorly Designed Response 
 Categories Drive Respondents Nuts!

    •  Guidelines:! –  Answer categories

    must include all (reasonable) possible answers!

    –  Answer categories must be mutually exclusive!

    6/24/12

    25

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Be Sure Question Specifies 
 (and Matches) Response Task!

    •  Ensure question stem clearly states the response task! 1.  “Do you favor or oppose expanding the

    designated faculty parking at NPS?”! •  And make sure it matches the response scale!

    1.  “Do you favor or oppose expanding the designated faculty parking at NPS?”! q  Agree! q  Neutral! q  Disagree!

    6/24/12

    26

    Huh???!

  • Use Forced-Choice Vice 
 Check-All-That-Apply Questions!

    6/24/12

    27

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Use Forced-Choice Vice 
 Check-All-That-Apply Questions (cont’d)!

    6/24/12

    28

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Respondents Tend to 
 Engage More with Forced Choice!

    6/24/12

    29

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Use Construct-specific Questions and Responses Whenever Possible!

    6/24/12

    30

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Another Example!

    6/24/12

    31

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • And Another Example!

    6/24/12

    32

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

  • Answer Ordering Matters!

    6/24/12

    33

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

    •  Randomize response options if concerned about order effects!

  • Layout and Spacing Matter!

    6/24/12

    34

    Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.

    •  Guidelines:! –  Maintain spacing

    between answer categories consi