Survey Design

Diane Steele, Juan Muñoz, Raka Banerjee, Talip Kilic

Module 1: Multi-topic Household Surveys

Poverty and Inequality Course

DECPI

March 7, 2012

Qualities of a good survey

Relevance Answer the policy questions of import

Quality Methodology Accuracy and reliability Adherence to international statistical guidelines

Timeliness Punctuality Production time Comparability over time

Sustainability Public dissemination Documentation

The total survey design

approach

Sampling

Data

management

Tools & Methods

Fieldwork

Or the integration of

the development of questionnaires and

methods,

sampling design,

the organization, training and

supervision of the field staff

and computerized quality controls of

fieldwork

to deliver analysts

and policy makers

reliable datasets on time

Survey design and implementation are

processes that require documentation

at all stages

Designing Household Survey

Questionnaires Diane Steele

Module 1: Multi-topic Household Surveys Poverty and Inequality Course

DECPI March 7, 2012

Questionnaire Design vs Sampling

Sampling

Complicated, in range of survey tasks –leave for specialist-

Has theory

Advantage: can tell if done right

Questionnaire design

No integrated theory, some techniques (cognitive testing)

Limited knowledge of impact of changes (LSMS IV need)

Cannot always tell when done right

Process is important

Process

• What are data needs?

• Can household survey meet needs?

• Is hhld survey best option?

Purpose of

Survey

Questionnaire Content

Questionnaire format

Organization of Field Work

Example: LSMS Surveys

Measure Welfare: Levels, Distribution, Causes

As there are multiple measures of interest:

– Consumption

– Income

– Wealth, Savings, Human Capital

– Objective measures

– Subjective measures

LSMS surveys use:

Multi-topic Household Questionnaire

Price Questionnaire

Example: LSMS Surveys

Measure Welfare: Levels, Distribution, Causes

Inform Policy Who Benefits/Uses Programs, public services

Impact of Programs

Use of public services

Outcomes policies designed to effect

Availability of Services to Household

Prices Charged

Quality of Services

LSMS surveys use: Household Questionnaire

Community Questionnaire

Facility Questionnaire

Example: LSMS Surveys

Measure Welfare: Levels, Distribution, Causes

Inform Policy

Determinants Why observed social outcomes occur

Household behavior

LSMS surveys use:

Household Questionnaire

Community Questionnaire

Facility Questionnaire

Starting Point

Discussion with policy makers

– What are key policies to measure

– What are other key issues

Starting Point

Discussion with policy makers

Discussion with Statistical Office

– Issues

– Feasibility

Starting Point

Discussion with policy makers

Discussion with Statistical Office

Create Data Users’ Group

Starting Point

Discussion with policy makers

Discussion with Statistical Office

Create Data Users’ Group

Existing Questionnaires

– Comparability

categories

variables

reference periods, age groups

– Panel

how to track households, individuals

Starting Point

Discussion with policy makers

Discussion with Statistical Office

Create Data Users’ Group

Existing Questionnaires

Qualitative research-

Sequential

Additional topics

Starting Point

Discussion with policy makers

Discussion with Statistical Office

Create Data Users’ Group

Existing Questionnaires

Qualitative research

“Increasing Policy Relevance of LSMS”

– Revision of experience

– Identify issues that hhld survey data can address

– Draft modules linked to policy issues

Starting Point

Discussion with policy makers

Discussion with Statistical Office

Create Data Users’ Group

Existing Questionnaires

Qualitative research

“Increasing Policy Relevance of LSMS”

Research

Concepts

Techniques

Designing for Quality

Multi-topic Questionnaire: Complexity

– Missing values

– Internal Consistencies

– Inaccuracies

– Omission of key issues by analysts

Quality Control Mechanisms

Explicit Questions

Explicit Questions

Example:

Language?

What is the highest

level and grade of

education that

[NAME] completed?

Education?

Quality Control Mechanisms

Explicit Questions

Pre-Coded

Pre-Coded Questions

2. Is [NAME]

......................

22. Where did [NAME] go toobtain medical care?

Male............1Female........2

Own

home...................1

Provider's

home...................2

Health

post...................3

Public Health

clinic.................4

Private Health Clinic..5

Public

Hospital...............6

Private

Hospital...............7

Other..................8

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Explicit Skip Patterns

4. Do youcurrentlysmokecigarettes?

5. Have youever smokedcigarettes?

6. At what agedid you startsmoking?

7. In the past30 days did youconsumealcoholicbeverages?

Yes............1 >>Q6No..............2

Yes.............1No...............2 >>Q7

Age in years Yes...............1No.................2 >>Sect. 5

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Quality of data

Respondent burden

Different attitudes

Modules

Roster

Parents of Hhld members

Housing, Utilities

Education

Health

Labor and Other Income

Privatization

Migration

Fertility

Credit

Expenditures /

Consumption

Agriculture

Non-Agr. Businesses

Other Income Hhld

Anthropometrics

Modules Head Individual Best nformed

Roster

Parents of Hhld members

Housing, Utilities

Education

Health

Labor and Other Income

Privatization

Migration

Fertility

Credit

Expenditures /

Consumption

Agriculture

Non-Agr. Businesses

Other Income Hhld

Anthropometrics

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Sensitive Issues Last – Fertility

– Savings and debt

– Income

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Sensitive Issues Last

Packaging

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Sensitive Issues Last

Packaging

Two-round format

Two Round Format

– Breaks up interview

– Allows entry and checking of first half data

– Allows for corrections with respondents

– Reference periods

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Sensitive Issues Last

Packaging

Two-round format

Small Sample

Quality Control Mechanisms

Explicit Questions

Pre-coded

Explicit Skip Patterns

Direct Informants

Sensitive Issues Last

Packaging

Two-round format

Small Sample

Training

Pilot or Field Testing of

Questionnaires

Ensure Questionnaires capable of collecting

required information:

Is the full scope of information needed being

collected?

Covering Credit?

Roster

Parents of Hhld members

Housing, Utilities

Education

Health

Labor and Other Income

Privatization

Migration

Fertility

Credit

Expenditures /

Consumption

Agriculture

Non-Agr. Businesses

Other Income Hhld

Anthropometrics Individual loans:

formal-informal,

unmet demand

Mortgages

Durable

goods

Food

Purchases

Supplier

credit

Supplier

credit

Pilot or Field Testing of

Questionnaires

Ensure Questionnaires capable of collecting required information:

Is the full scope of information needed being collected?

Does every section respond to policy needs?

Is the information collected in different sections internally consistent?

Specific Sections

Is the section inclusive?

Ex. Does the questionnaire include all

prevalent

– activities

– living arrangements

– sources of income

– consumption items

Individual Questions

Clear in all circumstances?

Lead to unambiguous responses?

Alternative interpretations?

All responses anticipated (pre-coded)?

Are skip patterns accurate/complete?

Pilot Test: How to

Involve both field staff and analysts

Test all instruments

In lab, in field in all conceivable situations

Ensure enough responses to each section

to test well

Document Process

Testing Employment Module

Self-employed farmers

Self-employed businesspersons

Employees

Unpaid family workers

Unemployed

Homemakers

Males, females, children

Testing Housing Module

Homeowners

Renters

Squatters

Multi-family Units

Testing Agricultural Module

Subsistence Farmers

Cash-crop Farmers

Landowners

Hhlds raising Livestock

Absentee owners

Revisions

Leave plenty of time to revise

– affects questionnaire

– training

– data entry

Test new revisions

– Take multiple versions to field

DOCUMENTATION CHECKLIST

• QUESTIONNAIRE DESIGN

– Questionnaire

– Pilot test report

– Interviewer manual

– Supervisor manual

– Other manuals