Designing and Using a Survey

February 7, 2011

Objectives

By the end of this meeting, participants should be able to:Explain how surveys accomplish the goals of: measuring attitudes, measuring change over time, making group comparisons, and analyzing the causes of behavior.

Goals of Surveys

• Measure the frequency of attitudes, beliefs and behaviors

• Measure change over time• Examine differences between groups (race,

class, gender, etc.)• Analyze causes of attitudes, beliefs, and

behaviors

Measuring Behavior

• Time can be awkward for respondents• Forward telescoping• Backward telescoping

• Sensitive subjects• Bogus pipeline technique (ethical concerns?)• Random lists• Randomized response technique (some sensitive, some

not)• Importance of others’ beliefs

Predicting Behavior

• Election Predictions• Problems with third parties• Problems with really close elections• Problem with social desirability for turnout• Undecided• Sensitive topics

2008 Presidential Example- NC

Final ResultsObama 2,142,651 49.70%McCain 2,128,474 49.38%Others 39,726 0.92%Total 4,310,851

Source: Office of the Clerk of the US House of Representatives

How did the individual polls do?

FOX News/Rasmussen McCain MofEReuters/Zogby McCain MofEAmerican Research Group Obama MofESurveyUSA McCain MofEMason-Dixon McCain MofEResearch 2000 Obama MofEPolitico/InsiderAdvantage Tie MofE

Recall: Margin of Error

• So for example, FOX/Rasmussen’s final poll for North Carolina showed 50% for McCain and 49% for Obama

• Sample: 1000 likely voters• Margin of error = ±t {p(1−p)/(n−1)} 1/2 {1 − f}1/2

• But since the population being sampled is large we can ignore the final {1 − f}1/2

• ±1.96 {50(50)/(1000-1 )} 1/2

• = ±3.1%• In other words, Rasmussen/FOX was 95% confident that the

final vote for McCain would be between 46.9% and 53.1%. Right on the mark, pretty much.

• Yet, if they had to make a prediction of the winner, they’d guess wrong. (Some polls had MO wrong, too.)

Measuring Attitudes

• One of the most common goals of surveys• Need to be concerned about measuring non-attitudes

(Converse)• Need to consider attitude strength, respondents may not care

equally about all issues (environment, gun control, etc.)• How do they feel about the issue personally?• How knowledgeable are they about this issue?• How certain are they of their opinion?• How much thought they have given to the issue?• Should you offer counterarguments?

Measuring attitudes

d) Attitudes especially weakly held attitudes will change over time (first impression versus considered opinions)

e) Need to watch disagreements over the meaning of words• Example: ideology and Stimson’s work

f) Need to watch for changing frames• Verbal (example: welfare, next page)• Contextual (example: sexual harassment)

Example of Verbal Differences- GSS

Measuring Change over Time

• Measurement of personal attitude change is frequently unreliable

• Comparisons of cross sectional surveys are more common but pose their own issues• Similar interviewing methods?• Similar sample sizes?• Similar question wordings?• Does the change exceed sampling variation?

Measuring Change over Time

c) Frequently can be atheoreticald)Can only measure gross change, not change at

the individual levele)Insta-polls can allow for immediate reactions

to stimuli but suffers worries about representativeness and long term impact

Measuring Change over Time

• Preferred method: panel studies• Repeated studies of the same individuals over time

• Limitations• Long term studies are prohibitively expensive• Atrophy of the original sample (and worries about those

that remain)• Errors in interviewing

Sub-group Comparisons

• Researchers frequently want to compare the attitudes of various sub-groups of the population

• One difficulty is the smaller and less accessible groups may be harder to randomly sample• Double sample or over sampling• Pyramiding or combining multiple surveys

What Leads to Change in Attitudes over Time?

• Surveys generally struggle with this type of question

• Most people’s answers for why their opinions changed are post-hoc rationalizations

• Individuals may not be sure of their reasons for opinion change

• Similar stimulus may affect people differently

• For February 9: Work on research proposals, you will have time for group work next meeting.

• For February 11: Download the program R from http://www.r-project.org/ and bring your laptop to class. You also may want to print-out “Getting Our Feet Wet with R” from the website.

• For February 14:– Research proposals due for each group.– Read WKB Chapter 8– On p. 189 answer either question 1, 2, or 3.– Construct a causal model relevant for YOUR

research question. (Turn-in the C.8 items individually.)