View
227
Download
0
Category
Preview:
Citation preview
What are we measuring?
- Whether people are
doing things differently
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
doing things differently
How can we measure it?
- Measuring changes in levels of reducing/ reusing/
recycling
- Observing/recording behaviours
- Asking about change
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
- Each is a form of survey or data collection exercise
SampleDesign
PilotSurvey
Conductof
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
ofSurvey
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
The survey
process
Richardson, Ampt,
Meyburg 1995
Each is a form of survey Method of measurement Aspects of survey design needed
Measuring changes in levels of
reducing/ reusing/ recycling
- Sample selection
- Pilot
- Survey
- Expansion/weighting
- Analysis
Observing/recording behaviours - Sample selection
- Pilot
- Survey
- Expansion/weighting
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
- Expansion/weighting
- Analysis
Asking about change - Sample selection
- Survey design
- Pilot
- Survey
- Expansion/weighting
- Analysis
Key elements of survey design
1. Preliminary planning
2. Selection of method
3. Sample design
4. Survey instrument design
5. Pilot
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
5. Pilot
6. Survey implementation
7. Expansion/weighting
• Analysis – over to you..
1. Preliminary planning
• Define survey objectives
– Very specific: what, by whom, over what period, where
• Review of existing information
– Useful methodologies from elsewhere; use of stated
preference?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
preference?
• Define terms
– From your objectives and from respondent’s perspective
• Survey content
– Dot points
7
Exercise
• Think of 2 terms you would want to use in a survey
related to your work
– Write down clear definitions for both
– Ask another person (not from your organisation) to do the
same with your terms
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Compare
2. Selection of a Survey Method for Measurement
• Observation surveys
• Intercept surveys
• Self-administered surveys
• Telephone surveys
• Personal interview surveys
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Personal interview surveys
• Internet/online surveys
9
Observation methods
• Chosen when
– Possible to count accurately
– Possible to count all or select a representative sample
• Can be manual, automatic, video
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
10
Intercept Surveys
• Intercepting people
– At an activity centre (e.g. workplace, transfer station, shopping centre)
• Possible methods
– distribution - mail-back/on-line
– personal interview
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– collect phone no.
• Always needs a total classification count
11
Intercept Surveys
• Advantages
– Able to reach specific populations
– Can combine with observational counts
– Can use multiple survey methods
• Disadvantages
– Generally low response rates (20-30% for self-completion)
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Generally low response rates (20-30% for self-completion)
– Hurried conditions
– Must allow for non-random sampling
– No follow-up possible in most cases
12
Self-administered Surveys
• Possible targets
– households
– activity centres/workplaces/transfer stations
• Method of Distribution
– mail-out vs. hand delivered
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Method of Collection
– mail-back vs. hand collection
13
Self-administered Surveys
• Advantages
– Can get extensive geographical coverage
– No interviewer effects
– Can obtain considered responses
– Hand-collection to good response rate
• Disadvantages
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Disadvantages
– Layout and wording must be clear - hard to design
– No probing possible
– Answers not independent
– Response rates lower than face to face
14
Telephone Interviews
• Advantages
– Wide geographic coverage
– Intermediate costs
– Good supervision - CATI
– Multilingual capabilities
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Computerised
Telephone Interviews
• Disadvantages
– Sample usually weak
• Low phone ownership for some groups
• Answer-phones, mobile phones, screening devices
• Hard to know how it represents the population
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Hard to know how it represents the population
– Credibility of interviewer (confusion with telemarketing)
– Low response rate
– No follow-up for refusals
Personal Interviews
• Can be paper or computer
• Advantages
– Generally higher response rates (60-90%)
– Flexibility of information
– Presence of interviewer
Maintain interest
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Maintain interest
– Spontaneous answers
Personal Interview
• Disadvantages
– High costs
– Interviewer influence
• personal characteristics
• interrupt household/work routine
opinions of interviewers
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• opinions of interviewers
• interpretation of vague answers
– Considered response difficult
On-line Surveys
• Advantages
– Low costs
– Can use elaborate visual effects
– Can use adaptive techniques (can give different scenarios
for different responses)
– Good for workplaces if sufficient follow-up
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Good for workplaces if sufficient follow-up
• Disadvantages
– Usually very biased sample
– Low response rate
– Hard to get all people in household if needed
Exercise
• Think of a behaviour you would like to measure
• Discuss with a partner
– Best method of collection
• Strengths
• Weaknesses
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Weaknesses
3. Sample Design in the Survey Process
SampleDesign
PilotSurvey
Conductof
Survey
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
Sampling Methods
Preliminary Concepts
• What is a sample?
– a collection of things which is some part of a larger population and which is selected so as to be representative of some or all of that population
• Target Population
– who are we trying to survey?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Sampling Units
– what are we going to sample?
• Sampling Frame
– where are we going to get a list of these things?
Sampling Methods
Sampling Frame
• a base list to identify the sampling units
• should contain all the sampling units
• examples,
– all households on a street (e.g. Council records)
– telephone directories
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– telephone directories
– mailing lists
– maps
– electoral rolls
– blocklists
Sampling Methods
Sampling Frame Problems
• inaccuracy
• incompleteness
• duplication
• inadequacy
• out-of-date
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• out-of-date
• Must check the reason for which the list was
originally compiled to understand likely deficiencies.
Sampling Methods
Sampling Error & Sampling Bias
• Sampling Error
– due to the simple fact that we are taking a sample, and not
the population.
– can minimise error by taking larger sample.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Sampling Bias
– due to systematic omission of some elements from our
final sample.
– cannot minimise error by taking larger sample.
Sampling Methods
Random Sampling
• Each unit is selected independently
and each unit in the population
has an equal probability of being selected.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Must use random sampling to avoid sampling bias.
Sampling Methods
Random Sampling Methods
• Simple Random Sampling
• Stratified Random Sampling
• Variable Fraction Stratified Random Sampling
• Multi-stage Sampling
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Multi-stage Sampling
• Cluster Sampling
• Systematic Sampling
• Note quotas not on list
Sampling Methods
Sample Size
• How much data do we need?
• Too much data >>> too expensive
• Not enough data >>> not able to draw conclusions
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Somewhere in the middle is a sample size which
enables us to draw sufficient conclusions at a
reasonable cost
Sampling Methods
Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.
Exercise
• Hand out random sampling sheet
– Explain
– Questions?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
4. Survey design in the survey process
SampleDesign
PilotSurvey
Conductof
Survey
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
Instrument design for reliable measurement
• Question content
• Question types
• Physical design - also for observation/counting
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Question Content
• Reliability
– repeatable
– easy to answer
• Accuracy
– no question bias
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– measures what we want
• Relevance
– must appear relevant to respondent
Question Types
• Factual
– “What did you do?”
• Classification (e.g. socio-demographic)
– for comparing with secondary data
• Opinion and attitude questions
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Opinion and attitude questions
– “What do you think about ……?”
• Stated Response Questions
– “What would you do if ……?”
Physical Design of Forms/Apps
• Observational surveys
– Ergonomic
– Size/format – not too big or small
– Weather-proof
– Need log forms
Test under actual conditions
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Test under actual conditions
Physical Design of Forms
• Self-administered forms
– Layout vital
– Minimal writing should be required
– No coding aids should appear
– Instructions very clear
– Professional appearance
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Professional appearance
– Include ID number
35
5. Pilots in the survey process
SampleDesign
PilotSurvey
Conductof
Survey
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
Pilot Surveys
• Why not do a pilot survey?
– too expensive
– not enough time
• Why do a pilot survey?
– too expensive to omit it
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– too expensive to omit it
– not enough time to omit it
Pilot Surveys
• pilot survey is a test of ALL aspects of design
• scope for experimental design
• saves expensive mistakes
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Uses of the Pilot Survey
• "Skirmishing" of wording
• Adequacy of questionnaire– definitions clear?
– too many "don't knows“?
– too long?
– open to closed questions
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Efficiency of interview/surveyer training
• Non-response rate
• Analysis
• Cost and duration
6. Survey implementation
SampleDesign
PilotSurvey
Conductof
Survey
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
Conducting the Surveys
• Need high response rate for validity
• Consider
Gross sample . . . . . . . . . . . . . . . . . . = 100 (houses who are eligible to put out bin)
Sample loss (vacant, invalid phone no.) = 5 (vacant houses)
Net sample . . . . . . . . . . . . . . . . . . . . . . = 95
Responses . . . . . . . . . . . . . . . . . . . . . . = 50 (put out green bin)
Response rate . . . . . . . . . . . . . . . . . . . . = response/net sample = (50/95) = 53%
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
• Consider
– Announcement letter/message >> higher response
– Follow-up regime >> higher response rate
Exercise – dot points for a survey you need
• What method?
• How to get a sampling frame?
• What questions? Need any for weighting?
• Other issues/questions?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– Richardson, Ampt, Meyburg (1995)
– http://www.geog.ucsb.edu/~deutsch/geog111_211a/code_books/Survey_Methods_For_Transport_Planning.pdf
7. Weighting (correction)/Expansion
SampleDesign
PilotSurvey
Conductof
Survey
Selection ofSurvey Method
SurveyInstrument
Design
PreliminaryPlanning
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
DataCoding Data
Editing
Data Managementand Analysis
Data Correctionand Expansion
Presentationof Results
Tidying-Up
Weighting & Expansion of Data
• Getting the sample data to represent the population
from which it was drawn, as nearly as possible
• Why? – systematic errors
– Non-response >> weighting certain type of respondents
higher
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
higher
– Missing Data >> can make assumptions – or note
– Inaccurate Reporting >> e.g. social desirability bias
Example of weighting
• Your response is 50/95 – what about the 45?
Say 30 males (67%) 15 females (33%)
- Your secondary data (e.g. counts, other data)
males 50% females (50%)
missing responses from females
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
– missing responses from females
- Responding females are therefore ‘weighted’ with a
slightly higher value (1.5) males (.75)
Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.
Summary
• To measure behaviour change it is essential to understand the data collection and survey process
• In particular need to understand:
– Survey method
– Sampling principles
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Sampling principles
– Importance of a pilot
– Implementation options
– Need for weighting
• Vital for future funding as well as for sharing methodologies
Recommended