Survey Research. Sampling and Inference

June 9, 2008

Ivan Katchanovski, Ph.D.

POL 242Y-Y

2

Survey Research• Surveys: One of the main methods of data collection

and research in political science and sociology

• Polls: Used in mass media and business

• Advantages• Reliable

• Involves relatively small numbers of respondents

• Relatively cheap and fast– Compare with census

• Disadvantages– Sampling error

– Validity

Example of Polls: Bush Approval

3

Observation and Sampling

• Polls and surveys are based on observations

• Population: universe of subjects the researcher wants to sample– Adult population in Canada

– Voters in Ontario

• Sample: cases or observations drawn from a population

4

Random Sample

• Random selection: Each element has an equal chance of selection independent of any other event in the selection process– Are internet polls random?

– Are telephone polls random?

• Weighting: Giving some cases in a survey dataset more weight than others– Weights examples

5

Parameters and Sample Statistics

• Parameters: Characteristics of a population– Proportion of the vote for political parties in national or local

elections

– Median personal income in Canada

• Sample Statistics: Estimates of population parameters, based on a sample drawn from a population– Exit poll estimate of proportion of the vote for political

parties in national or local elections

– Survey estimate of median personal income in Canada

• Inference about the characteristics (parameters) of a population based on samples

6

Confidence Level

• Surveys and polls: margin of error

• Random sampling error:– Decreases with increase in sample size

• Confidence Level– The estimated probability that a population parameter lies

within a given confidence interval

• Example: we might be 95% confident that between 48 and 52% of all voters favor Candidate A.

7

Election Eve Polls - U.S. Presidential Candidates, 2004

Date Begun

Agency Bush Kerry

10/29 Gallup/CNN/USA 49 51

10/29 NBC/WSJ 51 49

10/29 Harris 52 48

10/29 CBS 51 49

10/30 Fox/OpinDynamics 49 52

10/30 TIPP 51 49

10/31 GWU Battleground 52 48

11/2 Actual Vote 52 488

Types of Survey Questions

• Open-ended questions– Respondent is asked to provide his or her own answer to

the question

• Closed-ended questions– Respondent is asked to select an answer from among a list

provided by the researcher

• Example: different % correct responses in open-ended and closed-ended questions

9

Example: Two Types of Questions in a 2007 Pew Poll of Americans

Open-Ended Question %

Can you tell me the name of the president of Russia?

Yes, Vladimir Putin 36

Yes, other incorrect 3

No, don’t know 61

Total, 100

Closed-Ended Question, %

Who is the President of Russia? Is it…

(A) Vladimir Putin 60

(B) Boris Yeltsin 7

(C) Mikhail Gorbachev 7

(D) or is it someone else 8

Don’t know (Vol.) 18

Total 10010

Contingency Questions and Biased Questions

• Contingency questions: Questions which are contingent on response to previous questions

• Example of a contingency question:Q14.1a. When it comes to political matters, do you ever think of yourself

in terms of Left and Right?

(1) Yes (2) No (9) DK, etc.

(IF YES TO Q.14.1a, ASK Q.14.1b)

14.1b) How would you place your views on this scale, generally speaking – where 1 means “very Left” and 10 means “very Right”?

• Biased questions: Questions that encourage respondents to answer in a particular way

11

Response Rate

• Number of people participating in a survey divided by the number selected in the sample

• Acceptable response rates– 50% - adequate for analysis and reporting

– 60% - good

– 70% - very good

• Lower response rates– Can be used if non-respondents are similar to

respondents

12