Upload
curtis-thomas
View
219
Download
3
Embed Size (px)
Citation preview
JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEYBUSINESS AND CONSUMER TENDENCY SURVEY
Task Force on Improvement of Response Rates and Minimisation of Respondent Load
Part 1 – Relationship between response rates and data collection methods
Bianca Maria Martelli, Institute for Studies and Economic Analysis -ISAE – Rome, Italy
BRUSSELS, 14-15 November 2005
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20052
1. DATA COLLECTION METHODSwhich is the most effective for:
• Improving response rates• Reducing burden• Impact on costs• Sectoral effects• Size effects
2. COMMUNICATION METHODSmost effective and cost efficient for:
• Compulsory surveys• Voluntary participation
TERMS OF REFERENCETERMS OF REFERENCE
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20053
ACKNOLEDGEMENTSACKNOLEDGEMENTS
The present research has been realised with the substantial contributions of the members of the EC/OECD task force on RR who supported me with extremely useful material , references, information, personal experience and suggestions.
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20054
WHY IS THE RESPONSE RATE IMPORTANT?WHY IS THE RESPONSE RATE IMPORTANT?
• Response Rate (RR) one of the most important factors to ensure a high quality of the BTS results (Etter, 2002)
• Non response rate (NR) is one of the components of non-sampling error that may affect the survey and the most efficient strategy for reducing measurement errors is to eliminate possible causes of such errors during the whole survey design stage (OECD 2003)
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20055
WHY FIRMS/CONSUMER RESPOND?WHY FIRMS/CONSUMER RESPOND?
Conceptual framework for COS/BTS Survey participation (proposed by Willimack 2002, reported by Petroni 2004):
• Businesses weight response burden against business goals when considering the decision to participate in a survey. Both burden and goals may be impacted by: – factors relating to the external environment, the business
and the respondent, which are not under control of the survey organisation
– survey design features, which are under the survey design features, which are under the survey organisation controlsurvey organisation control
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20056
FACTORS INFLUENCING RESPONSE RATESFACTORS INFLUENCING RESPONSE RATES
Subject Kind of Survey
Questionnaire
Reporting, Sampling, Response Unit
Target Universe
Frame Sampling design
RR
Data Collection Mode
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20057
WHAT IS (NON) RESPONSE RATE?WHAT IS (NON) RESPONSE RATE?
• CONFUSION ON THE TOPIC : people deals with RR/NR, but not explain what really apply
• NEED OF CLEAR RR/NR DEFINITIONThe response rate is the number of
complete/incomplete interviews with reporting units divided by the number of eligible
reporting units in the sample (AAPOR 2004)
• SEVERAL KIND OF RR/NR• UNIT / ITEM NR
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20058
KINDS OF NON RESPONSE RATESKINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003) (1/3):
where n’ = number of enterprises which did not submit useable information
n = number of enterprises in the survey
'1 *100nNR n
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
20059
KINDS OF NON RESPONSE RATESKINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003)(2/3):
In case of unequal sampling fraction but uniform weights a proper measure is the following:
where
is the sampling fraction of the ith unit
'
1
1 1
1
2 *1001
n
i i
n
i
fNR
f
i
i
nf
n
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200510
KINDS OF NON RESPONSE RATESKINDS OF NON RESPONSE RATES
OECD DEFINITIONS (2003) (3/3): In the most general case, of unequal sampling fraction and reporting units with different weights:
where wi is the size weight of the ith unit
'
1
1 1
1*
3 *1001
*
n
ii i
n
ii
wf
NRw
f
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200511
KINDS OF (NON) RESPONSE RATESKINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004)
• COOP= Co operation rate Proportion of all cases interviewed of all eligible units ever contacted
• REF = Refusal RateProportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases
• CON = Contact Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200512
KINDS OF (NON) RESPONSE RATESKINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS
• OVERALL COMPLETION RATE (Lozar Manfreda, 2003) % of responses among all sent invitations or exposed to invitation/eligible, including partial and complete respondentsThe author assumes that most often researchers when dealing with RR actually refer to it (similar to OECD NR1)
• FULL COMPLETION RATE Considers only the % of complete (unit) respondents • DROP-OUT RATEReferring only to item non response
• CLICK-OUT RATE For Web surveys (% of accessing the Web questionnaire among all invited)
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200513
RECOMMENDATION ON (NON) RESPONSE RATESRECOMMENDATION ON (NON) RESPONSE RATES• Clearly specify which kind of response rate is applied
According to the specific work purposes of this paper:
• OECD NR1 (or OVERALL COMPLETION RATE) is useful in the first steps of the survey design to check the robustness of the data collection process. to check the robustness of the data collection process.
• FULL COMPLETION RATE , DROP OUT RATE, CLICK TROUGH are also useful for analysing the DCM
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200514
DATA COLLECTION MODES TECHNIQUESDATA COLLECTION MODES TECHNIQUES
• FACE TO FACE –Historically one of the first applied (since 1912)–Mainly for COS–High quality, High costs
• CAPI/CASI Computer Assisted Personal Intervieving/Computer Asssisted Self-
Interviewing –“Computerised” version of face-to face
• MAIL–The first historically applied (1788 as reported by De Leeuw, 2005)– Implies the self-administration of questionnaire– Visual approach– “Low” costs, relatively low RR
•
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200515
DATA COLLECTION MODES TECHNIQUESDATA COLLECTION MODES TECHNIQUES
• FAX– Similar to MAIL– Widely used for BTS
• TELEPHONE– Useful as supporting measure– “Aural” (instead of “visual”)contact: risk of “loss of context”– Primacy/recency effects (for COS)– Primary role of interviewer
• CATI (Computer Aided Telephone Interview)– Widely used for COS, less for BTS (Italy, CB)– Allows to quickly collect information– Allows immediate control of inconsistencies– Expensive
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200516
Focus: CATI AT ISAE for BTSFocus: CATI AT ISAE for BTS
minus: • Costs (in comparison
with postal tecnique) • Possible subjective
biases
plus: • Rapidity• Reliability • High response rate• Less sample biases• Easy substitution
• Since 1988 for Business (since 2002 completely)• Since the beginning (1992) for Services
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200517
Focus: CATI AT ISAE for BTSFocus: CATI AT ISAE for BTS
ISAE Manufacturing and Services CATI Interviews structure
Outcome Number of phone calls
MANUFACTURING (sample size 4100 units)
SERVICES (sample size 2100 units)
1- Accepted interview (explanatory material received) 4032 1846
2 - Accepted interview (expl. material non received) 31 93
3 - Accepted first interview 46 36
4 - Refusal 69 64
5 - Unreachable 129 730
6 -Out of target 4 2
Failure
7 -No response
51
75
8 -Phone busy 37
9 - Automatic replayer 9
10 - Wrong phone number 72
11 - Already contacted 1
12 - No more active 7
13 – Exceeding quotas 105
Total phone calls 4359 3076
Monthly average January – June 2005
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200518
DATA COLLECTION MODES TECHNIQUESDATA COLLECTION MODES TECHNIQUES
• E - MAIL– “Visual”approach, similar to MAIL and FAX– Decreasing trend– Low cost– Confidentiality problems (not allowed at INSEE )
• TOUCH-TONE Data Entry/VOICE RECOGNITION Entry
– For COS– Low RR– Low cost
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200519
DATA COLLECTION MODES TECHNIQUESDATA COLLECTION MODES TECHNIQUES
• WEB/CAWI – Increasing relevance for BTS (less for COS)– Low cost– Coverage errors– Risk of loss of context in the questionnaire– Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005,
Etter)– Self selection bias– Only a part of participants (30-50%) freely choose this mode
(Etter, Stangl, Kershoff)
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200520
DATA COLLECTION MODES PROCESSDATA COLLECTION MODES PROCESS
STEPS (timing of interaction)
– CO OPERATION / PRE SURVEY NOTIFICATION
– OVERALL RESPONSE STAGE– FULL COMPLETION– REMINDINGS– FOLLOW UPs
DCM often is a MULTISTAGE PROCESS and for any step more than one mode can be applied
MODESthe different modes can be
applied :
- CONCURRENT
and/or in a:
- SEQUENTIAL way
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200521
DATA COLLECTION MODE: MIXED MODE DATA COLLECTION MODE: MIXED MODE (De Leeuw, 2005)(De Leeuw, 2005)
• MULTIMODE SYSTEM
• OPTIMIZE data collection procedures
• REDUCE total survey ERRORS within the available time and costs
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200522
MIXED MODE DATA COLLECTION MIXED MODE DATA COLLECTION
FIRM
REDUCES BURDEN
REDUCES PANEL ATTRITION (TIME IN SAMPLE)
INCREASES VOLUNTARY PARTICIPATION
EFFECTS ON BURDEN, COSTS, RR
INSTITUTION
INCREASES RR
INCREASES BURDEN
INCREASES COSTS ERRORS DUE TO MODE EFFECTS
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200523
SECTOR AND SIZE EFFECTSSECTOR AND SIZE EFFECTS
SECTOR• Firms specific effect is
present– Differences (for both Mail
and Internet) (Etter, 2002)
– Differences (for both CATI and FAX), Italy
• …..but limited• Retail, Services lower
RR (but also smaller size…)
SIZE• Generally inverse
correlation between RR and Size– medium/large firms respond
more regularly (Etter, 2002)– Burden is greater for small
firms (Petroni, 2003)
• Some exception– South Africa (Kershoff, 2005),
Latvia do not notice remarkable influence
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200524
COMMUNICATION METHODS (CO OPERATION STAGE)COMMUNICATION METHODS (CO OPERATION STAGE)
• Action useful in getting co operation (and reducing NR):– Tailoring contacts to the characteristics of the
establishment– Contacting the “right” person (mid level management)– Overcoming the lack of awareness (of the Survey /
Institution)– Building the initial relationship (personal contact
advisable)– Knowledge of survey characteristics – Survey pre notice
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200525
COMMUNICATION METHODS: INTERVIEWERS COMMUNICATION METHODS: INTERVIEWERS (CO OPERATION STAGE)(CO OPERATION STAGE)
It is strongly advisable:
• Use of interviewers– in the initial step, even if the ongoing DCM is different,
increases RR (BLS Survey)
• Training of interviewers– Mainly for BTS, less for COS– Overcome gate keepers: difficult first contact does not
necessarily mean unwillingness to participate
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200526
COMMUNICATION METHODS COMMUNICATION METHODS (CO OPERATION STAGE)(CO OPERATION STAGE)
all the above mentioned actions are essential also for effectively communicating:
• COMPULSION (OF RESPONSE)
as it may increase the (perceived) burden of respondent
(Mixed ) Effects of compulsions :– Increases the Overall Completion Rate (lowers the Unit
NR)– May increase the Drop Outs (Item NR)– Positive effects in France, mainly for large firms– Positive effects in Italy (Retail)
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200527
COMMUNICATION METHODS COMMUNICATION METHODS (CO OPERATION STAGE)(CO OPERATION STAGE)
• Voluntary participationmost effective approaches :
– Sending the business survey to named individuals and use telephone follow-up methods to encourage responses (Paxon, 1995)
– Lowering the response burden (Linn 2003)– Increasing the perceived relevance of the survey to
business goals (Linn, 2003)– Adopting a Mixed DCM
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200528
OVERALL RESPONSE STAGE : PANEL ATTRITIONOVERALL RESPONSE STAGE : PANEL ATTRITION
• Maintain contacts with participant
• Renewing interest
• Using simple and plain questionnaires– Avoiding open ended and “difficult to answer questions”
• Question wording– accuracy, mainly when using concurrent DCMs
• Remindings
• Follow – ups
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200529
CONCLUSIONS AND RECOMMENDATIONSCONCLUSIONS AND RECOMMENDATIONS
• Efforts should be made in improving preliminary contacts and remindings / follow up actions
• All the modes have plusses and drawbacks • The starting point of view should be the
firm (to a lesser extent the consumer): adopting a company-centric point of view – Efforts should be made to allow surveys’
participants to choose the mode they prefer
Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November
200530
CONCLUSIONS AND RECOMMENDATIONSCONCLUSIONS AND RECOMMENDATIONS
• Although Internet (online) seems to be the mode of the future, a large part of firms still prefer the most traditional mail questionnaire and a significant part of respondents prefer fax (habit persistence)
• The future appears to be even more bound to a mixed data collection mode – Mixed mode data collection should be carried
out both concurrently and in a multi stage way (as far as possible bound to BTS and COS timeliness)