How to interpret results of public opinion surveys? How to respond?EU Accession Communications Network
Ohrid – 21-22 September 2006
Leendert de Voogd
2
How to interpret results of public opinion surveys? How to respond?
CONTENT
Back to basics
How to interpret the results?
Some hints
Case studies
What strategy?
Back to basics
4
A survey, what for?It is not necessary to interview everyone or we simply cannot.
There could be many reasons for conducting a survey. Some examples:
To identify attitudes, needs, concerns and/or expectations about specific topics
To measure opinions (agreement or opposition) to certain ideas
To test specific messages and communication plans
A survey will never:
Replace a census
Replace an election
It is a picture at a given time, in a given context, of the diversity of
opinions
5
Types of research
Two types of research: quantitative and qualitative
Quantitative: the objective is to ‘quantify’ opinions related to a specific issue. This can be done for different targets (general public, specific target groups, opinion leaders…)
Qualitative: to explore in depth motivations, opinions, attitudes and behaviours of individuals facing a given problematic. This is can be done through focus groups or in-depth interviews.
Both types are/could be complementary
6
Data collection modes
PAPI, CAPI, CATI,…CAWI
CATI stands for Computer Assisted Telephone Interviewing
Face-to-face, telephone or Internet?
FACE TO FACE
(PAPI/CAPI)
TELEPHONE
(CATI)
INTERNET
(CAWI)
For long questionnaire
Material to show
For difficult questions
For fast results
Targets difficult to reach
For difficult areas to get in
Self administration
Specific targets
Cheaper
7
Representativeness of a sampleFirst step: definition of the universe to represent
Second step: definition of a sample
The first quality of a sample is to be representative of the surveyed universe. This means without BIAS.- Sampling methods mostly used: random or quota sampling
The law of large numbers: The law says that with a large enough sample many odd coincidences are likely to happen.
The weight of the universe to survey has almost no incidence on the size of the
sample
8
Margin of error
Survey results remain approximations, the accuracy of which, everything being equal, rests upon the sample size and upon the observed percentage. Depending on sample sizes, the real percentages vary within precise confidence limits*. The formula:
±2√(p*q)/n
For 1000 interviews
Observed % 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%
Confidence limits
± 1.9 points ± 2.5 points ± 2.7 points ± 3.0 points ± 3.1 points
For 10 000 interviews
Observed % 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%
Confidence limits
± 0.6 point ± 0.8 point ± 0.9 point ± 1 point ± 1 point
* at 95% level of confidence
9
Margin of error (cont’d)
Survey results
(n=1000)Hyp.: Whole population
Yes 55% 51.85 < X < 58.15
No 45% 41.85 < Y < 48.15
How to interpret results?
11
Interpretation of the resultsInterpretation of the results is the step following the tabulation of the responses.
Rely on own basic knowledge:
Theoretical knowledge in sociology, economy, psychology and political science
Marketing models
Analyse the influence of social factors:
Gender: could refer to own perceptions of role differences between men & women
Age: could indicate an evolution in time but could also reveal generation’s effect
Education: cultural indicator but also social status indicator
Profession: social status indicator but also economical indicator
Constructed variables: opinion leaders (e.g. people who talk politics on a regular basis at home and who try to convince other people)
12
Interpretation of the results (cont’d)While interpreting results we must keep extra attention to the following:
At an individual level:
Age or ageing effect : public opinion vary with age, whatever the generation considered is
Generation effect : public opinion is typical from one generation; the youngest, while ageing, do not have the same response pattern as the oldest
Period effect: public opinion evolve in time, whatever the age.
At a collective level:
Structural changes models: major social or economic evolutions, technical innovations…
Political or economical cycles
13
Interpretation of the results (cont’d)
Behaviours Social and
demographic variables
Attitudes
Some hints
15
Context of the study
In an information society, public opinion is reacting almost immediately to particular events. This has a clear impact on survey results.
Some examples:
Bombings of Sarajevo’s markets obliged us to conduct new fieldwork in some countries
Shifts in American public opinion after 9/11: support for Bush increased
Better image of NGO’s after Tsunami in South-East Asia
Level of information on the Euro after the changeover took place (+20 points between November 2001 and January 2002)
16
Basis of respondents
Before drawing any conclusion on one particular result, one must look at the basis of respondents (i.e. how many people answer the question?)
A general rule that could be followed:
A minimum of 50 respondents per variable considered seems necessary for a sample of 1000 interviews
A minimum of 100 respondents per region considered seems necessary to speak of some regional trends
A minimum of 1000 respondents per variable
considered seems necessary for a sample
of 25000 interviews (e.g. in the EU)
17
Going beyond the obvious
Cross tabulations could be very helpful but sometimes are not really necessary.
Vertical %
(DK not shown)
Total
EU 25
Has benefited from EU
membership
Has not benefited from EU
membership
EU membership: a good thing
56% 82% 24%
EU membership: a bad thing
13% 2% 32%
EU membership: neither a good nor a bad thing 28% 14% 42%
18
Going beyond the obvious (cont’d)
E.G.: Level of information with the support to further enlargement
Vertical %
Total
EU 25
Feel well informed about
enlargement
Feel badly informed about
enlargement
FOR further enlargement
45% 53% 41%
AGAINST further enlargement
42% 41% 43%
DK 13% 6% 16%
2 hyp.: either good information produces support to enlargement; or people who already support enlargement are more interested in this topic and tend to seek more information.
19
Going beyond the obvious (cont’d)
Be careful to redundancy. The same persons are present in other sub-groups.
Examples given:
- Managers have similar answers to those who finished their education after the age of 20. Of course… they are often the same persons.
- Less educated persons have similar patterns of responses than those aged 65+.
It does not bring added value
20
Significant differencesA difference between two percentages has to be significant if one wants to comment it.
21
Correlation does not imply causality
There is a positive correlation between two variables if they are moving in the same direction. This does not mean that one variable is the cause (or the consequence) of the other.
Example given:
- People have a good image of the EU and feel happy.
On the contrary :
- People acknowledge to have received better information on one specific issue and objectively know more things about the topic
Assumptions can still be made
22
ScalesCommentators tend to forget what is behind a sub-total. It is essential to have a closer look at the distribution of responses.
65% is not equal to 65%
20
45
28
25
0
10
20
30
40
50
A new enlargement of the EU
Totally agree Tend to agree Tend to disagree Totally disagree DK
45
20
2
28
5
0
10
20
30
40
50
A new enlargement of the EU
Totally agree Tend to agree Tend to disagree Totally disagree DK
65
30
5
0
20
40
60
A new enlargement of the EU
Total agree Total disagree DK
23
Multi-answers vs. single answer possible
The same written percentage can have a different weight depending on how many answers the respondent could give.
When analysing multi-answers questions, it is possible to process the data on the basis of responses and not respondents.
50
40
30
20
5
0
10
20
30
40
50
3 answers were possible
A B C D DK
50
30
105 5
0
10
20
30
40
50
Only 1 answer was possible
A B C D DK
Case studies
25
EU Constitution
Surveys prior to referenda in France and the Netherlands showed that:
Reasons for supporting the ‘NO’ were clearly more concrete than those mentioned by supporters of the ‘YES’.
Other public opinion indicators (like public opinion mood, support for government in place…) were not helping the ‘YES’
Historical indicators should have been taken into account:
- Citizens in the former Member States were not consulted for the various wavesof enlargement. Especially not for the last one that took place in 2004.
- Very close win of the ‘YES’ in France in 1992
Clear signs of fragility
26
Threats to peace in the world
One of our survey was clearly misinterpreted by the media.
Original question: ‘For each of the following countries, tell me if in your opinion, it represents or not a threat to peace in the world’
Interpretation made: ‘EU poll: Israel 'biggest threat' to world peace’
‘Over half of Europeans think Israel poses the "biggest threat to world peace," according to a controversial poll commissioned by the European Commission.’ Source: WorldNetDaily.com
YES NO DK
Israel 59% 37% 4%
Iran 53% 41% 6%
North Korea 53% 40% 7%
What strategy?
28
Changing communication plans
Surveys can identified new themes, new issues, new messages and new targets to work on.
Results showed recently that European citizens made a difference between information sources legitimate to talk about a certain topic and sources they are using to get informed.
Potential actions:
Mobilize these legitimate
sources and make them intervene in
traditional media.
Focus on some particular publics,
geographical areas.
Use new messages, simplify them
29
Going forward
EU Constitution:
Recent surveys about the future of Europe showed a significant support for a EU Constitution (even in France and the Netherlands).
Results also clearly indicate that amongst supporters of a Constitution, there is a need to renegotiate and adapt the content of the current draft to include more political and social aspects.
Further enlargements:
EU citizens call for more information and communication about benefits and challenges of the process (especially in economical and social terms)
National Governments and media have a crucial role to play
grazie mille!
thank you!vielen Dank!
tack!merci!
bedankt!muchas gracias!
kiitos!