Designing Effective QuestionnairesDesigning Effective Questionnaires
Survey/Questionnaire Data …Survey/Questionnaire Data …
Use type appropriate to population, topic, and budget.– In-person– Telephone– E-mail / web– Self Administered / Mail
Survey/Questionnaire Data …Survey/Questionnaire Data …
Outputs (Activities)– Teaching and facilitation– Participant background– Location– Time and date of program– Marketing
Learning outcomes and intended behavior change Future programming (needs assessment)
Don’ts!Don’ts!Don’t ask a question if you already know
the answer!Don’t ask a question if you’re not prepared
to do anything about the answer!Don’t ask “nice to know” questions. Ask questions only if you know what
you will do with the answers.
What do you really want to What do you really want to know?know?
Take three minutes and write down four or five things you would really like to know from a questionnaire you are designing.
Asking good questionsAsking good questions
Questions should be clear and preciseMake it easy to completeKnow your audienceProtect anonymity of respondentsProof read and edit Pilot test!
Caution – End-of-Session Caution – End-of-Session QuestionnairesQuestionnaires
Self-reportsOne-point-in-timeAlthough they may measure perceptions
and intentions, they won’t measure behavior change
Other considerationsOther considerations
Data analysisHow will you use the results?Who needs the results?Timing of the survey/questionnaire
Designing QuestionnairesDesigning Questionnaires
Keep it short and on point. Use questions that have been used and tested by
others if possible Be sure questions have “face validity”. Include important demographic questions. Make sure ALL questions are germane to the
evaluation. Pre-test all questionnaires on same population.
Types of Question/FormatTypes of Question/Format
Open-endedYes or noOne best answerSelect all that applyRating or RankingRetrospective (post-then-pre)
Open-endedOpen-ended
Free textProvide richer detailMust be clear in what you are looking forMore difficult to get people to fill out
sometimesTakes more time to analyze
Yes or NoYes or No
SimpleUse with information that has no middle
ground– Do you own your own home?– Are you a U.S. citizen?
One-best answerOne-best answer
Provide only one of many possible answers
“Of the following, which topic would you MOST like to receive information about?”
a. Pasture Management
b. Farm Estate Transfer
c. Dairy Cattle Nutrition
Select all that applySelect all that apply
When you want to have more than one answer to a question
Which appliances do you own? (check all that apply)
__ Blender __ Waffle Iron
__ Toaster __ Coffee Maker
__ Mixer __ Food Processor
Rating QuestionsRating Questions
To what extent do you think you can apply the information learned here today?
Not at all A Great Deal 1 2 3 4 5 6
Ranking QuestionsRanking Questions
Of the topics covered today, please rank their usefulness in order, with 1 being most useful and 4 being least useful.
___ Setting limits for teenagers
___ Challenging behaviors
___ Consequences
___ Listening
Scavenger HuntScavenger Hunt
Using the UW Extension Publication – Collecting Evaluation Data: End-of-Session Questionnaires
An example of a grid with dichotomous pairs of words to rate an instructor
An example scale asking how the level of the workshop fit participant experience level
An example of a retrospective survey on public policy education
Sample category scales
Find the following:Find the following:
More about questionsMore about questions
Avoid double barreled questions– “Did you find the workshop helpful and
interesting?”
Don’t use leading questions. – “Do you agree that funding for Extension
should increase?” yes or no
Don’t use abbreviations and jargon
Questions… continuedQuestions… continued
Use ranges for choices instead of precise numbers. Put choices in logical order.
Be very clear– “How many times did your club meet last
year?”
Avoid questions that are too demanding.– “Rank the following 15 items in order of
importance to you.”
Yet more on questions…Yet more on questions…
Avoid making assumptions.– “How many children do you have?”
Avoid bias. – “How would you rate the workshop? a.
excellent; b. good; c. satisfactory”Make response categories clear.
– “How many acres do you own?” 0-10, 10-20, 20-50, 50-100
Question OrderQuestion Order
Question order does influence results. Begin a questionnaire with something that isn’t
too difficult to answer, but will be of interest to the participant.
Put easiest (demographic information) at the end – this is where people sometimes lose interest and if the questions at the end are less demanding, they are more likely to finish.
Pre/post TestsPre/post Tests
Use an identical knowledge test, before the educational program and then after.
Might ask specific questions about a subject, like, “How many servings of fruits are recommended for an adult every day by the new food pyramid?”
Might ask them to rate their knowledge of something, “Please rank your understanding of the new food pyramid recommendations.”
Pre/post test considerationsPre/post test considerations
Need to pair the tests (make sure you match up pre and post responses for each individual – not average scores) if you want to report on individual knowledge gained. – Use name, id number, birthday, etc.
Internal validity might be affected – testing influences.
Retrospective SurveyRetrospective Survey
Sometimes called a post-then-pre designBetter way to measure self-reported
behavior changesAvoids the issue of a participant not
knowing what they don’t know.– “Do you eat the appropriate number of fruits
and vegetables daily?”
Example retrospective questionExample retrospective question
After today, will you
Often Sometimes Never
Before today, did you
Often Sometimes Never
Check equipment calibration?
Use protective gloves?
Scout fields for specific pest problems?
Types of ErrorTypes of Error
Don’t get hung up on sampling error. (But be sure to have enough cases for statistical power.)
Non-sampling error is more critical.– Non-response– Question wording effects
Recommended
