Understanding Polling with Claudia Deane

8/11/2019 Understanding Polling with Claudia Deane

1/40

Interpreting and Using Polls in

2014

Claudia Deane

Director, Research Practices


2/40


3/40

Value of polls in reporting

Most high profile use: Election horserace

Much more important at Pew Research Center:

Scientifically represent public attitudes, values,

experiences

Add scope, meaning to anecdotes

Track societys changing attitudes

Giving voice to those not always heard


4/40

Journalists play a critical role in vetting,

interpreting survey research

Just like any source, polls require vetting

But no peer review for many polls

Especially now, varying methodologicalapproaches and quality

Journalists continue to be gatekeepers


5/40

Basic (but challenging) journalistic question

Is pol l rel iable enough to repo rt?

This is getting harder to answer as survey research,

like journalism, is in flux


6/40


7/40

Are they still accurate?

NOTE: Candidate error is the difference between final resultSOURCE: National Council on Public Polls

12

6

5

10

9

3

2

5

2.5 2.72

6.3

4.3

3.2

2.2

4.1

2.2

0.9 0.9 1.6

0

2

4

6

8

10

12

14

1936 1940 1944 1948 1952 1956 1960 1964 1968 1972 1976 1980 1984 1988 1992 1996 2000 2004 2008 2012

Average Candidate ErrorPresidential Elections 1936-2012


8/40

Questions to ask when writing about polls

AAPOR

1. Who paid for the poll and why

2. Who conducted the poll

3. How were interviews conducted

4. Number of interviews/margin of sampling error5. How were people chosen (probability sample?)

6. What population is poll is trying to represent

7. Timing of poll

8. Question wording/order9. Results based on full sample or subset

10. Were the data weighted, and if so, to what


9/40

To evaluate the quality of a poll,

you need two things:

1. A copy of the topline/trenddocumentThe actual

questionnaire, including

responses

2. The method box or surveymethodology, with details on

how poll was conducted

Practically speaking


10/40

1. Who paid for the poll and why2. Who conducted the poll

Questions to ask when writing about

polls


11/40

Usually easy to find

Washington Post-ABC News poll

This poll was conducted for The Washington Post and ABCNews by telephone April 17 to 21, 2013 among a random

national sample of 1,000 adults, including users of bothconventional and cellular phones. The results from the fullsurvey have a margin of sampling error of plus or minus3.5 percentage points. Sampling, data collection andtabulation by SSRS of Media, Pa. Produced for the

Washington Post by Capital Insight.*= less than 0.5 percent


12/40

SOURCE: http://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.html
http://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.htmlhttp://www.washingtonpost.com/blogs/fact-checker/post/a-misleading-obamacare-poll-courtesy-of-the-chamber-of-commerce-and-harris-interactive/2013/07/30/26e5f51c-f94a-11e2-8e84-c56731a202fb_blog.html


13/40

Here, the polling company, Harris Interactive, and the sponsor, the U.S.

Chamber of Commerce, presented the data in a highly misleading way and

then made false claims about the type of poll that had been conducted.

The Chamber has been a fierce opponent of the health-care law, a.k.a.

Obamacare, and we frequently warn readers they should always be skeptical

of polls peddled by partisan organizations. Perhaps it should be no surprise

that this poll was released just as the GOP-led House of Representatives

scheduled a vote to repeal the law.

Given the way the data was presented, Republican lawmakers thought they

had been handed a gift and ended up with eggon their faces.

Kessler, The Washington Post
http://www.harrisinteractive.com/http://www.uschamber.com/http://www.uschamber.com/http://www.politifact.com/truth-o-meter/statements/2013/jul/29/marco-rubio/sen-marco-rubio-says-75-small-businesses-say-theyl/http://www.factcheck.org/2013/07/gop-mistweets-obamacare-survey-results/http://www.factcheck.org/2013/07/gop-mistweets-obamacare-survey-results/http://www.politifact.com/truth-o-meter/statements/2013/jul/29/marco-rubio/sen-marco-rubio-says-75-small-businesses-say-theyl/http://www.uschamber.com/http://www.uschamber.com/http://www.harrisinteractive.com/http://www.harrisinteractive.com/


14/40

Possible Sponsors

Federal, state and local governments

Media organizations

Academic institutions

Non-profit groups or foundations

Special interest groups

Businesses and corporations

Political campaigns, consultants and candidates

Who paid for the poll and why?

Who conducted the poll?


15/40

1. Who paid for the poll and why

2. Who conducted the poll


4. Number of interviews/margin of sampling error


polls


16/40

Also usually easy to find

Washington Post-ABC News poll

This poll was conducted for The Washington Post and ABCNews by telephone April 17 to 21, 2013 among a random

national sample of 1,000 adults, including users of bothconventional and cellular phones. The results from the fullsurvey have a margin of sampling error of plus or minus 3.5percentage points. Sampling, data collection and tabulation bySSRS of Media, Pa. Produced for the Washington Post by

Capital Insight.*= less than 0.5 percent


17/40

0%

2%

4%

6%

8%

10%

12%

14%

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

MarginofSamplingError(+/-)

Sample Size

N=50

MOSE= +/- 14 pts

N=1,000

MOSE= +/- 3 pts

N=2,500MOSE= +/- 2 ptsN=10,000

MOSE= +/- 1 pts

Relationship between sample size and

margin of sampling error


18/40


19/40

Modes of administration

Telephone

RDD = Random Digit Dial of landline and (often) cell

phones

Voter list-based phone surveysIn-person

Mail

The Internet?!


20/40

The Internet is the edge

Challenges

Obvious coverage issues

Less obvious, but bigger issue: How to get a random

sample?Wide variety of methodologies being tried

Internet-based samples are often not

probability samples

Suggestion: Requires extra look to understand

and report on what youre dealing with


21/40




5. How were people chosen (probability sample?)


polls


22/40

Probability or non-probability

Probability sample Non-probability sampleCAN extrapolate from your sample to

a larger population (Most

Americans)

CANNOT extrapolate to a larger

population (yet)

CAN apply a margin of sampling

error

Margin of sampling error cannot be

computed

Ie, a randomly selected sample, a

random sample of the nationalpublic

Ie, Self-selected samples, click-

through polls, most internet panels


23/40

Yes No

Easy: Online click-in polls

SOURCE: From www.nationalreview.com, accessed 1/10/12.
http://www.nationalreview.com/http://www.nationalreview.com/


24/40

Trickier: Hybrid Internet panels

The sample itself is random

But the population from which thesample is drawn is made up of people

who have signed up to be members ofthe panel

Question: In what cases can this representthe full U.S. population?

Answer: Unresolved


25/40


26/40

9/17/2014 26


27/40

September 17, 2014 www.pewresearch.org 27

Is Pew Research ever going to use the kind of online non-probability

panel that the Times and CBS are using?

Scott Keeter: Yes, we willbut the real question is what we will use it for. Our

current standards permit the use of non-probability samples for certain

purposes, such as conducting experiments or doing in-depth interviews. In

addition, we have embarked on a program of research to help us better

understand the conditions under which non-probability samples can provide

scientifically valid data. We also are exploring how to utilize non-survey data

sources, which by their very nature tend to come from samples that are not

random. But until we understand the pros and cons of those methods a lot

better, were going to be very cautious about incorporating them into our

research.

Q


28/40







polls


29/40

What population is poll trying to represent?

All voters?

All conservatives?

All residents of Pakistan?

A thinking question, mainly, requiring 90% common

sense and 10% methodological savvy


30/40

So if you want to represent young people

Blue= cell

Green =

landline

SOURCE: Washington Post-ABC News analysis; data from 2009-2012

Q ti t k h iti b t


31/40






7. Timing of poll

8. Question wording/order


polls


32/40


33/40

Wording DOES Matter: Govt Surveillance

9/17/2014 33

Q ti t k h iti b t


34/40






7. Timing of poll

8. Question wording/order

9. Results based on full sample or subset


polls


35/40

ASK IF HEARD A LOT OR A LITTLE AND INTERNET USER ((INT1=1 OR

INT2=1 OR INT3=1) AND (Q.HB1=1,2)) [N=897]:

Q.HB2 Do you think your own online personal information was put at

risk by the Heartbleed bug, or do you think your information was not

put at risk?

Apr 23-27

2014 All internet users

45 Own information put at risk 29

47 Own information not put at risk 30

8 Dont know/Refused (VOL.) 5

9/17/2014 35

Q ti t k h iti b t


36/40






7. Timing of poll

8. Question wording/order9. Results based on full sample or subset

10. Were the data weighted, and if so, to what


polls


37/40

Were the data weighted, and if so, to what?

CBS News/New York Times

The com bined resul ts have been weigh ted to adjust

for var iat ion in the samp le relat ing to geographic

region , sex, race, Hispanic or ig in, age, education and

number of adu l ts in the household . Respondents in the

land l ine samp le were also weighted to take account o f

the number of te lephone lines into the residence.


38/40

Weighting of poll data

Its necessary in almost all surveys

(Nearly) Everyone does it

It corrects for the problem of not interviewing people in

the sample in correct proportion to their size in thepopulation


39/40

Good practices when writing about public opinion

Report the topline story Then break it down: identify key subgroups, look for

interesting differences

Are the numbers changing over time?

Be cautious with causality Are the results confirmed by other recent polls? If

not, write through it!

Use poll archives to validate conflicting claims about

public opinion Use personal quotes/interviews to animate raw

numbers


40/40

Thank you

[email protected]

Documents

Understanding Polling with Claudia Deane