Poom Nukulkij, Joseph Hadfield

and Stefan SubiasKnowledge Networks, Inc.

Evan LewisProgram on International Policy Attitudes (PIPA)

An Investigation Of Panel Conditioning With Attitudes Toward

U.S. Foreign Policy

Presented at the AAPOR 62nd Annual Conference

May 18, 2007

© 2007 Knowledge Networks, Inc.

Prepared for:

2

An Investigation of Panel Conditioning

Panel conditioning: The possibility that prior survey completion could affect the way participants respond to survey questions.

We investigate the matter using KnowledgePanelSM, the RDD-recruited Internet panel managed by Knowledge Networks (KN). Members are asked to participate in approximately three surveys per month using either a home-based PC connected to the Internet or a web appliance provided by Knowledge Networks.

The survey data presented here are from an October 2006 survey conducted by Knowledge Networks and the Program on International Policy Attitudes (PIPA) at the University of Maryland. The October 2006 survey is one study in a series of PIPA-KN surveys on foreign policy issues.

3

Research Questions

Research question #1:

Is the length of panel tenure predictive of attitudes toward U.S. foreign policy?

Being on a survey panel may result in longer-tenured participants to answer survey questions differently. They might, for instance, report different attitudes, knowledge levels, and opinions as a result of taking surveys on a panel.

If this hypothesis is true, survey responses for longer-tenured panelists will be significantly different from those with shorter tenure after controlling for demographic composition.

4

Research Questions

Research question #2:

Does prior participation on a particular survey topic relate to attitudes on the subject?

We selected foreign policy as a kind of “worst case” for investigating the presence of panel conditioning. For the duration of the conflict in Iraq, KN has conducted a number of foreign policy surveys using KnowledgePanelSM.

We speculated that if panel conditioning effects were measurable on KnowledgePanelSM, it would be in the area of foreign policy surveys.

5

Sample and Methodology

In October 2006, Knowledge Networks conducted a study examining attitudes about U.S. foreign policy on behalf of PIPA.

1,533 nationally representative adult members of KnowledgePanelSM were invited to participate with 1,057 completing the interview.

Among those who completed the survey: The median panel tenure was 28 months. The typical panelist had completed two surveys about foreign policy;

one-third had not completed any prior surveys on this topic.

6

Analysis Strategy

This research attempts to determine whether actual survey responses to the policy attitude items are associated with panel tenure and the number of foreign policy surveys taken previously, while controlling for a number of possible covariates which were selected due to their correlation with political behavior and attitudes.

Logistic regressions were conducted including the following variables of interest in the model:

Hypothesized key drivers: Panel tenure (number of months on the panel) Number of foreign policy surveys completed (range 0 to 12)

7

Analysis Strategy

Covariates: Political party ID (Republican, Independent/Other, Democrat) Age (in years) Gender (Male, Female) Education level (Less than HS, HS diploma, some college, bachelor's

degree or higher) Race/Ethnicity (White, African-American, Hispanic, other racial

descent) Christian religion (Yes/No) Evangelical beliefs (Yes/No)

8

Analysis Strategy

We examined the impact of panel tenure, prior completion of foreign policy surveys, and the covariates for each of the following items in the October 2006 survey:

Presidential job approval ratings Isolationism versus global view Satisfaction with position of the United States in the world Diplomacy versus military use The more important lesson of 9/11 Preferences for amount of U.S. budget devoted to defense spending

9

“Do you approve or disapprove of the wayGeorge W. Bush is handling his job as president?”

Approve Evangelicals, p < .001

Disapprove Democrats, p < .001 Independents/Others, p <.001 Women, p < .01 African-Americans, p < .001

n = 1,057

Nonsignificant Panel tenure, p > 0.62 Number of foreign policy surveys completed, p > 0.80

10

“Should the U.S. have to make sacrifices to help the world?”

The U.S. should not make sacrifices in an effort to help the world as a whole

Sometimes the U.S. should be willing to make some sacrifices if this will help the world as a whole

Older respondents, p < .001 Women, p < .05

n = 800

Nonsignificant Panel tenure, p > 0.80 Number of foreign policy surveys completed, p > 0.21

11

“In the effort to fight terrorism, do you think that in the future, compared with what it has been doing, the Bush administration…”

Should put more emphasis on military methods

Evangelicals, p < . 001

Should put more emphasis on diplomatic and economic methods

Number of foreign policy surveys completed, p < .05

Democrats, p < .001 Independent/Other, p < .01 Higher educated, p < .05

n = 802

Nonsignificant Panel tenure, p > 0.10

12

“On the whole, would you say that you are satisfied or dissatisfied with the position of the United States in the world today?”

Satisfied Christians, p < .01 Evangelicals, p < .01

Dissatisfied Democrats, p < .001 Older respondents, p < .05 African-Americans, p < .01

n = 802

Nonsignificant Panel tenure, p > 0.86 Number of foreign policy surveys completed, p > 0.07

13

“What do you think is the more important lesson of September 11th?”

The U.S. needs to work more closely with other countries to fight terrorism Democrats, p < .01 Younger respondents, p < .05

The U.S. needs to act on its own more to fight terrorism

n = 796

Nonsignificant Panel tenure, p > 0.31 Number of foreign policy surveys completed, p > 0.86

14

How should this part of the budget should be distributed? [Defense spending not including extra costs for Iraq and Afghanistan]

(More money) Number of foreign policy

surveys completed, p < .05 Higher educated respondents, p

< .05 Evangelicals, p < .01

(Less money) Democrats, p < .001 Women, p < .001 African-American respondents, p

< .01

n = 776

Nonsignificant Panel tenure, p > 0.49

15

Discussion & Conclusions

We found no evidence of panel conditioning resulting from panel tenure. Of the six attitude questions, for only one was there a possible weak, nonsignificant association between length of panel tenure and survey responses.

16

Discussion & Conclusions

There is some evidence that completion of prior surveys on the topic of foreign policy can have an effect on how participants answer questions on this topic. On three of the six measures, there was some evidence (p = .10 or less) that participants who had taken more surveys about foreign policy responded differently.

Those who completed 1–2 prior foreign policy surveys (71%) were more likely to indicate that the Bush administration should place more emphasis on diplomatic and economic methods compared to those who completed 0 or 3+ prior surveys (63% and 64%, respectively).

Those who completed 3 or more prior foreign policy surveys (64%) were less satisfied with the U.S. position in the world than those who completed 0 or 1–2 prior surveys (68% and 73%, respectively).

Those who had completed 3 or more prior foreign policy surveys ($475) indicated that they would allocate more money to defense spending than those who completed 0 or 1–2 prior surveys ($432 and $434, respectively).

17

Directions for Future Research

It is possible that topics with more personal relevance for respondents could produce different results.

For example, an older respondent who completes many surveys on Medicare and Social Security may learn more about the issues that directly affect him or her by completing such surveys, and may change his or her opinions over time.

Thus, future research should focus on issues of varying personal relevance for respondents to determine whether conditioning effects exist, and if so, whether there is a consistent relationship between relevance and conditioning.

18

Directions for Future Research

As mentioned earlier, the results described in this study apply only to KnowledgePanelSM.

Future research examining panel conditioning on panels using different recruitment methodologies and comprised of different samples is warranted.

As discussed earlier, KnowledgePanelSM members complete surveys on a wide range of topics, which is not the case for some survey panels. Research focusing on other panel types should be conducted to determine whether panel conditioning effects exist on such panels.

19

For Additional Information

Please visit the Knowledge Networks Government & Academic Research website for reprints of the full paper:

http://www.knowledgenetworks.com/ganp/2007aapor.html

Or email the first author, Poom Nukulkij at:

pnukulkij@knowledgenetworks.com