Upload
bryan-swindall
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Questionnaire Design Questionnaire Design ClinicClinic
Spring 2010 Seminar Series
Survey Research LaboratoryUniversity of Illinois
www.srl.uic.edu
2
Survey Research Laboratory
Cognitive Steps in AnsweringCognitive Steps in Answering QuestionsQuestions
1. Understand question.2. Search memory for information.3. Integrate information into
summary judgment.4. Translate judgment onto response
alternatives.
3
Survey Research Laboratory
Survey Intro/Cover LetterSurvey Intro/Cover Letter
Introduction should indicate: who is conducting the survey the topics to be covered in the survey an assurance of confidentiality any IRB stipulations whether you offer how long it will
take depends on mode, topic, population
4
Survey Research Laboratory
Survey Intro/Cover Letter - Survey Intro/Cover Letter - exampleexample
The goal of this survey is to understand the patterns of interactions between the groups and individuals who will be attending ….. The results of this survey will be used in part for University of Illinois at Chicago research purposes. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the ____________ region. By completing this survey, you are agreeing to participate in the research.
5
Survey Research Laboratory
Cover letter - exampleCover letter - example
The Department of _________at the University of Illinois at Chicago, is conducting to understand the patterns of interactions between the groups and individuals ….… The research team is headed by…
You are receiving this questionnaire because you are . . . It should take you about 10 minutes to complete the questionnaire. Please respond by August 1.
Participation in this survey is voluntary, and you may choose to stop answering questions at any time. Your choice to participate or not will have no effect on your course grade or on your relation with the University of Illinois. Your response is anonymous as we will not be asking for any personally identifying information – please do not write your name anywhere on this questionnaire.
We plan to use the results of this survey in part for University of Illinois at Chicago research purposes. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the Calumet region.……. Only aggregated results will be presented.
Completing this questionnaire indicates your consent to participate in the survey.
Please contact ________ if you have any questions. If you have any questions about your rights as a research participant, please contact the IRB (OPRS) at….
Thank you for your participation!
NamesAffiliationEmail / phone number
6
Survey Research Laboratory
Ordering the QuestionsOrdering the Questions
First questions should be directly related to the topic as described in the introduction or advance/cover letter
Put demographic questions at the end
Leave objectionable questions (e.g., income) for the end
7
Survey Research Laboratory
Characteristics of a good questionCharacteristics of a good question
One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that accommodates all possible contingencies of response
One that uses specific, simple language One that produces variability in
response One that minimizes social desirability One that is pretested
8
Survey Research Laboratory
Acquiescence - exampleAcquiescence - example
Would you be interested in learning about courses from the past experiences of other students who have taken them?
• Yes• No
Do you feel ready to apply what you have learned today?
• Yes• No
9
Survey Research Laboratory
Acquiescence - exampleAcquiescence - example
Administrators (e.g., principal, assistant principal, dean of students ) at this school ………………
a. Collaborate with faculty and staff to make the school run effectively.
• Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree
10
Survey Research Laboratory
Form A (agree I’s version):Individuals are more to blame than social conditions for crime and lawlessness in this country.
Form A (Agree I’s) Agree (I): 59.6%
Disagree (SC): 40.4 100 (473)
Form B (agree SCs version):Social conditions are more to blame than individuals for crime and lawlessness inthis country.
Form B (Agree SCs) Agree (SC): 56.8%
Disagree (I):43.2 100 (472)
AcquiescenceAcquiescence
11
Survey Research Laboratory
Do you feel ready to apply what you have learned today?
• Yes• No
Do you feel ready to apply what you have learned today, or do you not feel ready to do that?
• Feel ready• Do not feel ready
Acquiescence - exampleAcquiescence - example
12
Survey Research Laboratory
Would you be interested in learning about courses from the past experiences of other students who have taken them?
• Yes• No
How interested are you in learning about courses from the past experiences of other students who have taken them?
• Not at all interested, Slightly interested, Moderately interested, Very interested, Extremely interested
Acquiescence - exampleAcquiescence - example
13
Survey Research Laboratory
Administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively.
Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree
Characteristics of a good question - Characteristics of a good question - exampleexample
How well do administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively? Not well at all, Slightly well, Moderately well, Very well, Extremely well
14
Survey Research Laboratory
Characteristics of a good questionCharacteristics of a good question
One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that uses specific, simple language One that accommodates all possible
contingencies of response One that produces variability in
response One that minimizes social desirability One that is pretested
15
Survey Research Laboratory
The instructor was able to answer questions clearly and concisely in this course.
Participants materials (high quality; easy to follow; helpful, etc)
Characteristics of a good question - Characteristics of a good question - exampleexample
16
Survey Research Laboratory
Characteristics of a good questionCharacteristics of a good question
One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that uses specific, simple language One that accommodates all possible
contingencies of response One that produces variability in
response One that minimizes social desirability One that is pretested
17
Survey Research Laboratory
Indicate how the amount of use of the following types of classroom assessment have changed in your classroomAssessment during instruction to determine:degree of comprehension on part of an individual student…..
Characteristics of a good question - Characteristics of a good question - exampleexample
18
Survey Research Laboratory
Has your use of the following types of assessments increased, decreased or stayed the same?
Assessment during instruction to find out how well a
student has understood the material…..
Characteristics of a good question - Characteristics of a good question - exampleexample
19
Survey Research Laboratory
Characteristics of a good questionCharacteristics of a good question
One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that uses specific, simple language One that accommodates all possible
contingencies of response One that produces variability in
response One that minimizes social desirability One that is pretested
20
Survey Research Laboratory
How often do you get in touch with your academic advisor in the Department of Communication?
• Never• Once a school year or less often• Once a semester • Multiple times a semester or more often • In emergencies only
How often do you go talk with your academic advisor in the Department of Communication?
Multiple times a semester Once a semester Once a school year For emergencies only
Characteristics of a good question - Characteristics of a good question - exampleexample
21
Survey Research Laboratory
Characteristics of a good questionCharacteristics of a good question
One that yields a truthful, accurate answer
One that asks for one answer on one dimension
One that uses specific, simple language One that accommodates all possible
contingencies of response One that produces variability in
response One that minimizes social desirability One that is pretested
22
Survey Research Laboratory
Preferable to test the questionnaires with people like those in your main study population
Test in same mode to be used for the study
Consider cognitive pretesting
PretestingPretesting
23
Survey Research Laboratory
Other characteristics of a good Other characteristics of a good questionquestion
Specify who, what, when, where and how.
Over what period of time? What’s included?
Provide cues
24
Survey Research Laboratory
Does this table of contents cover recent developments? Does this table of contents cover recent research / topics in this field, or
does it not?
How many people live in your household? How many people live in your household?
Please include yourself in the count, and also those who people who live with you but are away temporarily. Don’t forget to count any babies and children.
How many years have you been enrolled in a college or University? Not counting summers, for how many years have you been enrolled as a
student in a college or university?
Please count each year that you were enrolled even if these are not consecutive years.
Other characteristics of a good Other characteristics of a good question - examplequestion - example
25
Survey Research Laboratory
Open vs. Closed QuestionsOpen vs. Closed Questions
Closed questions are usually better Easier for the respondent Less coding later Better to have respondent do
categorizing Categories help define the
question
26
Survey Research Laboratory
What grade level(s) have you worked with in the past 12 months? (e.g., 6th, 7th, 8th) _________________
Which grade level(s) have you worked with in the past 12 months? (please select all that apply).
Grade 6 Grade 7 Grade 8
How often do you send text messages? _________________
How often do you send text messages? You can answer in any of the three different ways below, whichever one would be most accurate for
you. You can tell us how many times per DAY you send text messages, or how many times per WEEK
you send text messages, or how many times per MONTH you send text messages.
Open vs. Closed Questions - Open vs. Closed Questions - exampleexample
27
Survey Research Laboratory
Disadvantages of Disadvantages of Closed QuestionsClosed Questions
Categories may be leading to respondents
May make it too easy to answer without thinking
May limit spontaneity Not best when
asking for frequency of sensitive behaviors
there are numerous possible responses• role of initial question development and
pretesting
28
Survey Research Laboratory
Response OptionsResponse Options
Response categories should be consistent with the question
Categories should typically be exhaustive, including every possible answer
Categories should be mutually exclusive
“Don’t know” category usually not appropriate knowledge questions
29
Survey Research Laboratory
Did the facilitator have a thorough understanding of the subject matter?Very Poor, Needs Improvement, Okay, Good, Excellent
How good or bad was the facilitator’sunderstanding of the subject matter?
Extremely good, Somewhat good, Neither good nor bad, Somewhat bad , Extremely bad
Response Options - exampleResponse Options - example
30
Survey Research Laboratory
Please indicate below which of the following organizations you interact with on a regular basis. Please check all that apply.
Response Options - exampleResponse Options - example
How often do you exchange ideas or advice with members of each of the following organizations?Organization A: Never, Rarely, Sometimes, Very Often, Extremely OftenOrganization B: Never, Rarely, Sometimes, Very Often, Extremely Often
Do you have at least one personal friend in each of the following organizations, or do you not?Organization A: Have, Do not haveOrganization B: Have, Do not have
Organization A• I regularly exchange ideas or advice with members of this organization.• I regularly collaborate on projects with members of this organization.• I have personal friends in this organization.• I provide funding or receive funding from this organization.
31
Survey Research Laboratory
5-7 response options
General guidelines• number of scale points (sensitivity vs.
meaning)• labeling of scale points
Response OptionsResponse Options
32
Survey Research Laboratory
How would you rate the instructor’s ability to explain the material in this course?Extremely poor, Somewhat poor, Fair, Somewhat good, Extremely good (or Excellent)
Response Option Labels - Response Option Labels - exampleexample
33
Survey Research Laboratory
Physical Format ChecklistPhysical Format Checklist
Number all questions sequentially
Use large, clear type; don’t crowd
‘White space:’ Place more blank space between questions than between subcomponents of questions
List answer categories vertically instead of horizontally
Avoid double/triple ‘banking’ of response choices
Be consistent with direction of response categories
Be consistent with placement of response categories
34
Survey Research Laboratory
Physical Format Checklist (continued)Physical Format Checklist (continued)
Don’t split questions across pages. If necessary (e.g., question requires 1.5 pages), restate question and response categories on next page
Put special instructions on questionnaire as needed, next to question
Distinguish directions from questions – be consistent
35
Survey Research Laboratory
Key References To Get StartedKey References To Get Started
Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, 1999.
Bradburn, N, Sudman, S. and Wansink, B. Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004.
See also past issues of Public Opinion Quarterly
36
Survey Research Laboratory
Other ReferencesOther References
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. New York: Academic Press.
Krosnick, J.A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., et al. (2002). The impact of "No Opinion" response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371–403.
Saris, W. E., Krosnick, J. A., & Schaeffer, E. M. (2005). Comparing questions with agree/disagree response options to questions with construct-specific response options.Unpublished manuscript, Political, Social, Cultural Sciences, University of Amsterdam.