Upload
merry-crawford
View
223
Download
1
Tags:
Embed Size (px)
Citation preview
Survey Design and Measurement
Jeremy Kees, Ph.D.
Some practical issues….
• Qualtrics Research Platform– Free you under VSB’s “site license”– Extremely user friendly, but also very
robust– www.qualtrics.com
Some practical issues….
• Amazon Mechanical Turk– The most inexpensive way to collect
consumer data– Extremely user friendly, but also very
robust– www.mturk.com
Online Survey (created by you and housed on Qualtrics’ server)
Create HIT (Human Intelligence Task) on Mturk•Description of your study and a (Qualtrics) link to it
Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey)
Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid.
Everyone is happy
The Mturk data I collected today….
• N=200
• Cost = $100
• Data collected in less than 1 hour
• Demographics– Mean age = 36
– 56% male
– 76% Caucasian
– 80% at least some college• 41% are college grads
– Median income = $35-50k
– Highly engaged!
Stages in theResearch Process
Determine Research Design
Analyze and Interpret the Data
Design Sample and Collect Data
Formulate Problem
Design Data Collection Method and Forms
Prepare the Research Report
Surveys / Questionnaires
• The most common measurement instrument when quantitative data is sought– Descriptive research– Experiments– Modeling– Etc….
Developing Surveys
• Good, well-specified research objectives lead to good surveys
• Research design dictates what types of questions should be used– Exploratory research = unstructured
script– Confirmatory research = structured
survey
Desirable Characteristics
• Brief
• Objective
• Specific
• Relevant
Survey Methods
• Usually should determine administration method prior to developing items– Can dictate what types of questions you
should ask
• Internet panels have become the most efficient and versatile method to collect data– Phone is still a viable option
– Mall intercepts can still be useful
– Mail/fax makes little sense anymore
Mall Intercepts vs. E-Panels
mall internet paneltests tests members
household size 2.8 2.9 3.0average age 40.5 39.2 37.2employed 71% 72% 69%white 86% 88% 89%male 20% 21% 15%college 40% 43% 46%
Correlation between Responses:
mall vs. internetinternet test/retest reliability
purchase intent .86 .94frequency .94 .97liking .85 .91price / value .90 .99
Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254.
Internet PhoneTime survey took to administer 12.5 19.4 minutes
Upon completion, would respondent participate in future studies? 35% yes 26% yes
More experienced Internet Users x
Used rating scale extreme “endpoints”more frequently x
E-Panels vs. Phone
Types of Questions
• Screening Variables
• Independent Variables
• Dependent Variables
• Classification Variables– Segmentation– Moderators
• Attention Filters
Primary Data: Overview
• Types of Primary Data– Demographic / Socioeconomic Characteristics– Psychological / Lifestyle Characteristics– Attitudes / Opinions– Awareness / Knowledge– Intentions– Motivation– Behavior
• What, how much, where, when, how, who• Purchase behavior vs. use behavior
• E.g., --- basic hierarchy of effects models• Example (CWL Study)
QUESTION WORDING - General Guidelines • Use simple words and questions• Avoid ambiguous words and questions• Avoid leading questions---be objective• Avoid implicit alternatives• Avoid generalizations and estimates ---
Be specific • Avoid double-barreled questions
1. What is your income?
$10,000 or less………………….1
$10,000 to $25,000……………..2
$25,000 to $50,000……………..3
$50,000 to $75,000………..…….4
$75,000 to $100,000..……..…….5
$100,000 or more…………..……6
What is the problem and how would you revise the question?
4. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES)
Very Satisfactory………………………4
Somewhat satisfactory…………………3
Somewhat unsatisfactory………………2
Very unsatisfactory……………….……1
Question Wording
• It is good practice to use scales whenever possible– Likert or semantic differential– Multi-item
Itemized Rating Scales
• The respondents are provided with a scale that has a number or brief description associated with each category.
• The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated.
• The commonly used itemized rating scales are the Likert and semantic differential
Types of Scales
• Nominal scales: those that use only labels
• Ordinal scales: those with which the researcher can rank-order the respondents or responses
• Interval scales: those in which the distance between each descriptor is equal
• Ratio scales: ones in which a true zero exists
21
IntervalPlease indicate how much you like each soft drink by checking the appropriate position on thescale: dislike like
a lot dislike like a lotCoke ____ ____ ____ ___Dr. Pepper ____ ____ ____ ___Mountain Dew ____ ____ ____ ___Pepsi ____ ____ ____ ___Seven Up ____ ____ ____ ___Sprite ____ ____ ____ ___
NominalWhich of the soft drinks in the following list do you like? (Check ALL that apply):
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite
OrdinalRank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6):
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite
RatioPlease divide 100 points among these soft drinksTo represent how much you like each:
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite100
Examples…
Itemized Rating Scales
• Likert Scales– requires the respondents to indicate a degree of
agreement or disagreement with each of a series of statements about the stimulus objects
Strongly Disagree Neither Agree Strongly
disagree agree nor agree
disagree
1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5
2. Wal-Mart has poor in-store service.1 2X 3 4 5
3. I like to shop at Wal-Mart . 1 2 3X 4 5
4. Wal-Mart has low prices . 1 2 3X 4 5
Itemized Rating Scales
• Semantic Differential Scales– End points associated with bipolar labels that have
semantic meaning
SEARS IS:
Powerful --:--:--:--:-X-:--:--: Weak
Unreliable --:--:--:--:--:-X-:--: Reliable
Modern --:--:--:--:--:--:-X-: Old-fashioned
Decisions for Itemized Scales
• Number of scale items• More is better, but there is a diminishing return
around 11 points (Nunnally 1978)• 7-point scales are customary
• Enough to discriminate• Allows for a scale midpoint• Manageable
• Odd/even number of categories
• Forced vs. non-forced
Why Multi-Item Scales??
Construct Abstract Concept
“Unobservable”
“Latent”
“Psychological”**Single items are typically not
sufficient to assess unobservable constructs
Multi-Item Scales are More “Reliable”• True Score Test Theory
– All measures have1. “True” Score2. “Error” (Random and Systematic)
• Good measures minimize the systematic error component of the score
• Types of Reliability– Inter-Rater– Test-Retest– Internal Consistency (Cronbach’s Alpha)
Developing Sound Measures
Step 2:
Specify Domainof the Construct
Generate Sampleof Items
Collect Data
Purify Measure
Assess Validity
Step 1:
Step 3:
Step 4:
Step 5:
Question Sequencing
• After you have developed your measures, think about the order in which they should be asked
QUESTION SEQUENCING - General Guidelines
• Use (more) simple, interesting opening questions
• Use the funnel approach, asking broad questions first, and follow with more specific questions
• Carefully design branching questions– Skip/display logic
– Ask for classification information last• Place more difficult or sensitive
questions near the end
Question ordering #1
1 – EVALUATION OF FAT LEVEL OF A PRODUCT
2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS
3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE
Question ordering #2
1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE
2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS
3 – EVALUATION OF FAT LEVEL OF PRODUCT
QUESTION SEQUENCING - General Guidelines
FOP Labeling Study
• We were interested in consumer evaluations of:1. Facts Up Front
2. All On-Package Labeling
3. Front-of-Package Nutrition Info
Why was question sequencing critical??
Tips for Maximizing Participation
• Offer an incentive ($$$)• Importance/relevance of the research
project and its purpose• Completing the questionnaire will take
only a short time• Answers are anonymous or
confidential• Reminder 2-3 days after the initial ask
Attention Filters
• Always include an attention filter to ensure that you are getting “quality” respondents– Eliminate “click throughs”
Attention Filters (Case Study)
• Advertising Experiment– Very stringent screening criteria
• Total # that started the study = 15,458• Number that qualified = 870
– Incidence Rate (IR) = 5.6%
• Number that qualified and passed the attention screener = 451– 48% failed the attention filter!!!
• NOT GOOD, criticalmix!
“Easy” Attention Filter
“Difficult” Attention Filter
And finally, remember the golden rule….
Do unto your respondents as you would have them do unto you!!
Team Assignment #2• Refine your research questions
– Need to be clear, concise, and “testable”
• Based on your research questions1. Design 2 potential studies that could
address your research questions1. Explain the benefits and weaknesses of each
approach
2. Pick the “best” design and explain your decision
(Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later)
Team Assignment #3
** Don’t start on this assignment until you’ve read Fowler (CH 6-7)
• Based on your research design1. Write a paragraph about what your
measurement instrument is supposed to accomplish
2. Make a list of what should be measured to accomplish the goals of the study
3. Develop your measurement instrument
Team Assignment #3
• Deliverables include:1. A very clean, polished version that you
could use to actually collect data• This means you will need to carefully think
through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.)
2. Intro paragraph and variable list (see previous slide)
(Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)