Task Force on Improvement of Response Rates and Minimisation of Respondent Load

JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEYBUSINESS AND CONSUMER TENDENCY SURVEY

Task Force on Improvement of Response Rates and Minimisation of Respondent Load

Part 1 – Relationship between response rates and data collection methods

Bianca Maria Martelli, Institute for Studies and Economic Analysis -ISAE – Rome, Italy

BRUSSELS, 14-15 November 2005

Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November

20052

1. DATA COLLECTION METHODSwhich is the most effective for:

• Improving response rates• Reducing burden• Impact on costs• Sectoral effects• Size effects

2. COMMUNICATION METHODSmost effective and cost efficient for:

• Compulsory surveys• Voluntary participation

TERMS OF REFERENCETERMS OF REFERENCE


20053

ACKNOLEDGEMENTSACKNOLEDGEMENTS

The present research has been realised with the substantial contributions of the members of the EC/OECD task force on RR who supported me with extremely useful material , references, information, personal experience and suggestions.


20054

WHY IS THE RESPONSE RATE IMPORTANT?WHY IS THE RESPONSE RATE IMPORTANT?

• Response Rate (RR) one of the most important factors to ensure a high quality of the BTS results (Etter, 2002)

• Non response rate (NR) is one of the components of non-sampling error that may affect the survey and the most efficient strategy for reducing measurement errors is to eliminate possible causes of such errors during the whole survey design stage (OECD 2003)


20055

WHY FIRMS/CONSUMER RESPOND?WHY FIRMS/CONSUMER RESPOND?

Conceptual framework for COS/BTS Survey participation (proposed by Willimack 2002, reported by Petroni 2004):

• Businesses weight response burden against business goals when considering the decision to participate in a survey. Both burden and goals may be impacted by: – factors relating to the external environment, the business

and the respondent, which are not under control of the survey organisation

– survey design features, which are under the survey design features, which are under the survey organisation controlsurvey organisation control


20056

FACTORS INFLUENCING RESPONSE RATESFACTORS INFLUENCING RESPONSE RATES

Subject Kind of Survey

Questionnaire

Reporting, Sampling, Response Unit

Target Universe

Frame Sampling design

RR

Data Collection Mode


20057

WHAT IS (NON) RESPONSE RATE?WHAT IS (NON) RESPONSE RATE?

• CONFUSION ON THE TOPIC : people deals with RR/NR, but not explain what really apply

• NEED OF CLEAR RR/NR DEFINITIONThe response rate is the number of

complete/incomplete interviews with reporting units divided by the number of eligible

reporting units in the sample (AAPOR 2004)

• SEVERAL KIND OF RR/NR• UNIT / ITEM NR


20058

KINDS OF NON RESPONSE RATESKINDS OF NON RESPONSE RATES

OECD DEFINITIONS (2003) (1/3):

where n’ = number of enterprises which did not submit useable information

n = number of enterprises in the survey

'1 *100nNR n


20059


OECD DEFINITIONS (2003)(2/3):

In case of unequal sampling fraction but uniform weights a proper measure is the following:

where

is the sampling fraction of the ith unit

'

1

1 1

1

2 *1001

n

i i

n

i

fNR

f

i

i

nf

n


200510


OECD DEFINITIONS (2003) (3/3): In the most general case, of unequal sampling fraction and reporting units with different weights:

where wi is the size weight of the ith unit

'

1

1 1

1*

3 *1001

*

n

ii i

n

ii

wf

NRw

f


200511

KINDS OF (NON) RESPONSE RATESKINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004)

• COOP= Co operation rate Proportion of all cases interviewed of all eligible units ever contacted

• REF = Refusal RateProportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases

• CON = Contact Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases


200512

KINDS OF (NON) RESPONSE RATESKINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS

• OVERALL COMPLETION RATE (Lozar Manfreda, 2003) % of responses among all sent invitations or exposed to invitation/eligible, including partial and complete respondentsThe author assumes that most often researchers when dealing with RR actually refer to it (similar to OECD NR1)

• FULL COMPLETION RATE Considers only the % of complete (unit) respondents • DROP-OUT RATEReferring only to item non response

• CLICK-OUT RATE For Web surveys (% of accessing the Web questionnaire among all invited)


200513

RECOMMENDATION ON (NON) RESPONSE RATESRECOMMENDATION ON (NON) RESPONSE RATES• Clearly specify which kind of response rate is applied

According to the specific work purposes of this paper:

• OECD NR1 (or OVERALL COMPLETION RATE) is useful in the first steps of the survey design to check the robustness of the data collection process. to check the robustness of the data collection process.

• FULL COMPLETION RATE , DROP OUT RATE, CLICK TROUGH are also useful for analysing the DCM


200514

DATA COLLECTION MODES TECHNIQUESDATA COLLECTION MODES TECHNIQUES

• FACE TO FACE –Historically one of the first applied (since 1912)–Mainly for COS–High quality, High costs

• CAPI/CASI Computer Assisted Personal Intervieving/Computer Asssisted Self-

Interviewing –“Computerised” version of face-to face

• MAIL–The first historically applied (1788 as reported by De Leeuw, 2005)– Implies the self-administration of questionnaire– Visual approach– “Low” costs, relatively low RR

•


200515


• FAX– Similar to MAIL– Widely used for BTS

• TELEPHONE– Useful as supporting measure– “Aural” (instead of “visual”)contact: risk of “loss of context”– Primacy/recency effects (for COS)– Primary role of interviewer

• CATI (Computer Aided Telephone Interview)– Widely used for COS, less for BTS (Italy, CB)– Allows to quickly collect information– Allows immediate control of inconsistencies– Expensive


200516

Focus: CATI AT ISAE for BTSFocus: CATI AT ISAE for BTS

minus: • Costs (in comparison

with postal tecnique) • Possible subjective

biases

plus: • Rapidity• Reliability • High response rate• Less sample biases• Easy substitution

• Since 1988 for Business (since 2002 completely)• Since the beginning (1992) for Services


200517

Focus: CATI AT ISAE for BTSFocus: CATI AT ISAE for BTS

ISAE Manufacturing and Services CATI Interviews structure

Outcome Number of phone calls

MANUFACTURING (sample size 4100 units)

SERVICES (sample size 2100 units)

1- Accepted interview (explanatory material received) 4032 1846

2 - Accepted interview (expl. material non received) 31 93

3 - Accepted first interview 46 36

4 - Refusal 69 64

5 - Unreachable 129 730

6 -Out of target 4 2

Failure

7 -No response

51

75

8 -Phone busy 37

9 - Automatic replayer 9

10 - Wrong phone number 72

11 - Already contacted 1

12 - No more active 7

13 – Exceeding quotas 105

Total phone calls 4359 3076

Monthly average January – June 2005


200518


• E - MAIL– “Visual”approach, similar to MAIL and FAX– Decreasing trend– Low cost– Confidentiality problems (not allowed at INSEE )

• TOUCH-TONE Data Entry/VOICE RECOGNITION Entry

– For COS– Low RR– Low cost


200519


• WEB/CAWI – Increasing relevance for BTS (less for COS)– Low cost– Coverage errors– Risk of loss of context in the questionnaire– Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005,

Etter)– Self selection bias– Only a part of participants (30-50%) freely choose this mode

(Etter, Stangl, Kershoff)


200520

DATA COLLECTION MODES PROCESSDATA COLLECTION MODES PROCESS

STEPS (timing of interaction)

– CO OPERATION / PRE SURVEY NOTIFICATION

– OVERALL RESPONSE STAGE– FULL COMPLETION– REMINDINGS– FOLLOW UPs

DCM often is a MULTISTAGE PROCESS and for any step more than one mode can be applied

MODESthe different modes can be

applied :

- CONCURRENT

and/or in a:

- SEQUENTIAL way


200521

DATA COLLECTION MODE: MIXED MODE DATA COLLECTION MODE: MIXED MODE (De Leeuw, 2005)(De Leeuw, 2005)

• MULTIMODE SYSTEM

• OPTIMIZE data collection procedures

• REDUCE total survey ERRORS within the available time and costs


200522

MIXED MODE DATA COLLECTION MIXED MODE DATA COLLECTION

FIRM

REDUCES BURDEN

REDUCES PANEL ATTRITION (TIME IN SAMPLE)

INCREASES VOLUNTARY PARTICIPATION

EFFECTS ON BURDEN, COSTS, RR

INSTITUTION

INCREASES RR

INCREASES BURDEN

INCREASES COSTS ERRORS DUE TO MODE EFFECTS


200523

SECTOR AND SIZE EFFECTSSECTOR AND SIZE EFFECTS

SECTOR• Firms specific effect is

present– Differences (for both Mail

and Internet) (Etter, 2002)

– Differences (for both CATI and FAX), Italy

• …..but limited• Retail, Services lower

RR (but also smaller size…)

SIZE• Generally inverse

correlation between RR and Size– medium/large firms respond

more regularly (Etter, 2002)– Burden is greater for small

firms (Petroni, 2003)

• Some exception– South Africa (Kershoff, 2005),

Latvia do not notice remarkable influence


200524

COMMUNICATION METHODS (CO OPERATION STAGE)COMMUNICATION METHODS (CO OPERATION STAGE)

• Action useful in getting co operation (and reducing NR):– Tailoring contacts to the characteristics of the

establishment– Contacting the “right” person (mid level management)– Overcoming the lack of awareness (of the Survey /

Institution)– Building the initial relationship (personal contact

advisable)– Knowledge of survey characteristics – Survey pre notice


200525

COMMUNICATION METHODS: INTERVIEWERS COMMUNICATION METHODS: INTERVIEWERS (CO OPERATION STAGE)(CO OPERATION STAGE)

It is strongly advisable:

• Use of interviewers– in the initial step, even if the ongoing DCM is different,

increases RR (BLS Survey)

• Training of interviewers– Mainly for BTS, less for COS– Overcome gate keepers: difficult first contact does not

necessarily mean unwillingness to participate


200526

COMMUNICATION METHODS COMMUNICATION METHODS (CO OPERATION STAGE)(CO OPERATION STAGE)

all the above mentioned actions are essential also for effectively communicating:

• COMPULSION (OF RESPONSE)

as it may increase the (perceived) burden of respondent

(Mixed ) Effects of compulsions :– Increases the Overall Completion Rate (lowers the Unit

NR)– May increase the Drop Outs (Item NR)– Positive effects in France, mainly for large firms– Positive effects in Italy (Retail)


200527

COMMUNICATION METHODS COMMUNICATION METHODS (CO OPERATION STAGE)(CO OPERATION STAGE)

• Voluntary participationmost effective approaches :

– Sending the business survey to named individuals and use telephone follow-up methods to encourage responses (Paxon, 1995)

– Lowering the response burden (Linn 2003)– Increasing the perceived relevance of the survey to

business goals (Linn, 2003)– Adopting a Mixed DCM


200528

OVERALL RESPONSE STAGE : PANEL ATTRITIONOVERALL RESPONSE STAGE : PANEL ATTRITION

• Maintain contacts with participant

• Renewing interest

• Using simple and plain questionnaires– Avoiding open ended and “difficult to answer questions”

• Question wording– accuracy, mainly when using concurrent DCMs

• Remindings

• Follow – ups


200529

CONCLUSIONS AND RECOMMENDATIONSCONCLUSIONS AND RECOMMENDATIONS

• Efforts should be made in improving preliminary contacts and remindings / follow up actions

• All the modes have plusses and drawbacks • The starting point of view should be the

firm (to a lesser extent the consumer): adopting a company-centric point of view – Efforts should be made to allow surveys’

participants to choose the mode they prefer


200530

CONCLUSIONS AND RECOMMENDATIONSCONCLUSIONS AND RECOMMENDATIONS

• Although Internet (online) seems to be the mode of the future, a large part of firms still prefer the most traditional mail questionnaire and a significant part of respondents prefer fax (habit persistence)

• The future appears to be even more bound to a mixed data collection mode – Mixed mode data collection should be carried

out both concurrently and in a multi stage way (as far as possible bound to BTS and COS timeliness)

Documents

Task Force on Improvement of Response Rates and Minimisation of Respondent Load