PPA 502 – Program Evaluation

Lecture 5a – Survey research

Introduction

Government administrators and elected officials love to claim that they possess a profound understanding of their public’s needs, desires, and disaffection.

Introduction

Unfortunately, the administrators and officials are learning that storms of controversy provide meager evidence of the workaday values of the everyday people they govern.

Surveys of the public, conducted following the basic precepts of survey design and analysis, are fast becoming the vehicle for genuine connection to the public will.

Introduction

Uses.– Evaluations of government services.– Changing demographics that may signal shifts

in service demand.– Patterns of service utilization.– Problem identification.– Customer service.

Introduction

Surveys have several important qualities.– Anonymity to respondents.– Point of view, characteristics, or use patterns

can be characterized with little confusion.– Good surveys provide input from a

representative cross-section.

Begin Before the Beginning

The best surveys grow from well-conceived and well-articulated reasons for conducting them.

Resist the temptation to hit the ground running.

Be certain of the purposes of the survey.

Begin Before the Beginning

Identify the appropriate audiences. Identify the political and personal will for

doing the survey. Determine whether the questionnaire to be

developed is better as a one-time or periodic survey.

Think about the usefulness of comparative data.

Getting Started

Convene a steering committee with key stakeholders.

Enlist the help of top government officials or administrators.

Designing the Survey

Sampling.– Choose the appropriate sampling frame: about

what population do you wish to generalize?– A sampling plan must give every respondent in

the sampling universe an equal chance of ending up in the sample.

• Simple random sample.

• Stratified sampling.

• Stratified random cluster sampling.

Designing the Survey

Targeting the individual in the household.– If no list exists, you may only have addresses or

phone numbers. If so, use household member with most recent birthday.

Mail, Phone, or In-person Interviews The best ways to conduct surveys vary by

accuracy, speed, and cost.– Most common are mail and phone surveys.

Mail, Phone, or In-person Interviews

Criteria Mail Phone In-Person

Accuracy

Response rate 45-55% 65-75% 75%+

Permits anonymity High Moderate Low

Is free from interviewer bias High Moderate Low

Handles various question types

Long/ complex Low Moderate High

Visual aids High Low High

Ensures question order Low High High

Permits widest coverage

Targets geographic areas High Low High

Avoids education bias Low Moderate High

Gives easy access to target population High High Moderate

Speed of administration Month Week Month

Cost per interview $8-12 $15-20 >$20

Mail, Phone, or In-person Interviews Increasing response rates.

– Multiple mailings (up to three) with stamped, return address envelope.

– Press coverage.– Combination of methods often best: Mail

survey with telephone and in-person followup.

Mail, Phone, or In-person Interviews Selecting sample size.

– The size of sample depends on desired precision of estimates.

– Generally speaking, if opinions are split as much as possible, than 100 residents will have a margin of error of +/- 10% with 95 percent confidence. Four hundred residents the margin is +/- 5%.

– In general, 100 is a good minimum number, especially for subgroups.

Questionnaire Construction

Each question should be judged against the purposes of the survey and the uses to which it will be put.

Steal widely.– National Citizen Survey from International City

Management Association and National Research Center.

Questionnaire Construction

Major principles.– Consistency.

– Clarity.• Vague wording.

• Double-barreled questions.

• Assumed knowledge.

• Overlapping response categories.

– Simplicity.• Specificity.

• Brevity (30 min. Phone, 60 min. In-person, 10 page mail).

• Context sensitivity.

Questionnaire Construction

Major principles.– Security.

• Demographic at end.

• General to specific.

– Fairness.• Option symmetry (balanced responses).

• Option wording and order.– Background info, pros and cons, opinion.

– Randomize pros and cons in a complicated survey.

Conducting the Survey

The survey steering committee.– Double check questionnaire with steering

committee. Frequency of surveys.

– For most multipurpose surveys, no more than once per year.

Pretest.– Test on twenty people at random. Ask

questions about format and clarity.

Conducting the Survey

Training.– Survey assistants must be trained. All must operate

uniformly, asking the questions in the same way, coding in the same way.

– Consistent open-ended coding.

– 10% recontact of survey respondents.

Trying hard and keeping track.– Three contacts by telephone for each number.

– Warning and at least two mailings for mail surveys.

Reporting Results Data analysis.

– For most government surveys, percentages, average responses, simple cross-classifications.

– The most complicated analysis will be to get accurate population estimates – weighting.

Report writing and presentation.– Executive summary.– Bulleted lists.– Document survey methods in appendix.– Augment tables with bar and pie charts.– Powerpoint for in person presentation.

Hiring a Consultant

Previous experience. Ability to communicate findings. Share work with in-house staff. Intuition.