Qualitative methodology in Behavioural Studies for
EU Policy-Making
Workshop-reportJune 2016
European Commission
Joint Research Centre
Institute for Prospective Technological Studies
Contact information
Rene van Bavel
Address: JRC-IPTS, Edificio Expo Calle Inca Garcilaso 3, 41092 Seville, Spain
E-mail: [email protected]
Legal Notice
The opinion expressed in this report does not imply a policy position of the European Commission.
Neither the European Commission nor any person acting on behalf of the Commission is responsible
for the use which might be made of this report
1
QUALITATIVE METHODOLOGY IN BEHAVIOURAL STUDIES FOR EU POLICY-MAKING
Main points from the workshop
1. The twin objectives of the workshop were to (a) get to know each other and address
issues relating to the Framework Contract for the Provision of Behavioural Studies
and (b) discuss the role of qualitative methodologies for EU policy-making. These
minutes present a structured summary of the main points relating to the latter
objective.
2. These points must be set against the backdrop of research for policy support in the
EU. In other words, what are the relevant issues to consider given that EU policy-
makers expect robust research based on diverse perspectives (which decreases
biased perceptions, provides a reality check, and promotes buy-in from participating
stakeholders)? [SP]1
Why conduct qualitative research?
3. The research industry is experiencing profound change, driven by technology and
digital innovations. Two of the major trends in the research consultancy world are
the rise of big data and the accompanying rise in the importance of strategic
qualitative research. These two trends are highly complementary – we need
qualitative depth to turn ‘big data’ into ‘smart data’. [MF]
4. Good qualitative research has a role at all stages of the research process, given the
importance of context and its major impact on behaviour. The context needs to be
investigated, but it is difficult to assess direct with quantitative methods. To
properly understand the role of context requires a mix of qualitative techniques
such as observation, ethnography (in its various forms), cognitive interviewing, even
desk research. [MF]
5. Qualitative methods can be used to guide quantitative research, especially among
researchers whose main area of competence is quantitative. They see the process as
a pyramid, with qualitative at the bottom and methods like experiments at the top.
[TM] Qualitative methods help mirror the real environment, stripping out the
irrelevant stuff. This makes them particularly relevant for EU behavioural studies,
which are quite bespoke. [CD]
1 Initials in square brackets refer to those participants who made, contributed to, or supported the point (broadly
speaking). See list of participants at the end of the document.
2
6. However, there is also a role for qualitative methods after quantitative methods. It
can help understand patterns in the data that are unexpected or ambiguous. In one
study on social media, for example, odd quantitative findings emerged about people
clicking on a particular link on a webpage. Qualitative research subsequently
uncovered that the patterns of behaviour that had been observed were in fact due to
Facebook's algorithms, not necessarily to people's preferences. [GV] Qualitative
research can also be used to help explain the why – the reasons behind peoples’
views and opinions as expressed in quantitative surveys. It can also be used as a risk
hedge (e.g. "we didn't pick something up in the experiment, but we found in
subsequent qualitative research that…"). [CD]
7. Qualitative methods are also excellent as a communications strategy, and can help
engage the audience and bring issues to life. For example, a quote from qualitative
research can capture the policy-maker's attention in a way that statistics cannot.
These verbatims can be used as a complement to cold numbers or complex models
and help make sense of them. [TM]
8. Qualitative methods may be useful when focusing on specific populations that would
otherwise be hard to target in a study, such as marginalised sectors of the
population, or when researching particularly sensitive or controversial topics. With
quantitative surveys you might get politically correct or socially desirable responses
but won’t find out what people really think or how they really behave… with
qualitative you’ve at least got a chance to get under the surface. [CD]
9. They can also help recognise who the participant identifies with and pinpoint their
motivations (using e.g. collage techniques), leading to more effective policy
interventions. Take the example of campaign against drunk driving: a teenager calls
her dad who is at home and asks him to pick her up because she missed last bus. But
the father has had a few beers at home. The message reads: “Would you like to be
dead with your girl or would you rather have your daughter mad at you because you
weren’t able to give her a ride?” Such a message is likely to hit home more than one
picturing young, reckless drivers. [CRS]
10. 'Immature' topics of research may indeed be good candidates for qualitative
research. For example, one study on illegal downloads was quite explorative in
nature; the contractor was not looking for anything particular. They started with
qualitative research, with one focus group in each EU Member State. They did this
first because there was not a lot of available research to guide the study. This
allowed them to understand the drivers and barriers of the investigated behaviour,
i.e., what motivates and what could discourage young people to access illegal content
online and buy counterfeit products online. [AD]
3
Which qualitative methods are used in behavioural studies?
11. Qualitative research has been very much affected by technological changes (e.g.,
mobile technology, social media, online communities, wearable devices and new
touchpoints) which open new options for researchers and participants to interact.
Qualitative data is no longer limited to focus groups, interviews, document analysis
and participant observation, although these traditional methods are still important.
[MF]
12. One of the methods used is to have a "shop-along", where researchers follow
consumers through a supermarket and listen to their thoughts aloud. Thinking aloud
is different to self-reporting (addressed further on) because participants are not put
on the spot. [TM] They use it to characterise a phenomenon as much as possible, to
"cover all the bases" or to "seek the variability", not to reach conclusions. It is about
being open to surprises. [GV]
13. Journey maps are also qualitative alternatives. In a study on illegal downloading,
participants drew a tree of their decision-making process when searching for
information on the internet. Journey maps can vary quite significantly depending on
the product (e.g. films vs. music) and on participants' own preferences on whether
or not to use illegal content. They can also vary from country to country depending
on the culture regarding illegal downloads. [AD]
14. The Zaltman metaphor elicitation technique2 can be used to tap into what goes on in
people's minds, but which they might be unaware of. [CRS]
15. The question emerged: is qualitative vs. quantitative a useful distinction? Some
workshop participants felt it was outdated, thanks to the changes brought about by
the digital revolution [GV]. Take the example of eye-tracking, which allows exploring
what parts of a letter of website people pay attention to, or mystery shopping. Are
these qualitative methods?
16. For example, insights from decision-making are used to perfect online algorithms
(the kind used by Google of AirBNB). Qualitative research has helped develop these
hypotheses, which then feed into the system to produce large amounts of data,
which can be explored quantitatively. This analysis might suggest that these
algorithms be improved, resulting in a kind of feedback loop between qualitative
and quantitative methodology. [GV]
2 Zaltman, G. (2001). U.S. Patent No. 6,315,569. Washington, DC: U.S. Patent and Trademark Office.
4
Criticism of qualitative methods
17. Probably the biggest problem of qualitative methods, in policy-makers' eyes, is their
inability to answer questions such as: to what extent are the results generalizable?
What is the prevalence of the phenomenon being observed? A political party facing
an election, for example, would be foolish to rely solely on focus groups as
representative of the whole population. They might offer a good overview of the
breadth of opinions among the electorate, but their results should not be generalized
to the general public. [RvB]
18. From the point of view of inferential statistics, the problem of generalisation of
qualitative methodology boils down to the problem of representativeness: samples
sizes are too small. Even if purposeful sampling is used, there is some risk: how to
decide who the right participant is? [DdR]
Rebuttal – measuring attitudes and opinions and estimating their prevalence in a
given population is not the aim of qualitative methodology. Quantitative surveys
are probably best suited for that, as they allow giving an equivalent weight to each
participant. [FD]
19. Another problem of qualitative methodologies is that data analysis heavily relies on
the researcher's subjective interpretation.
Rebuttal – there is also subjectivity in quantitative behavioural research (e.g., in
designing experiments and data analysis), as well as in the way survey results are
analysed and interpreted. [BF, NRP]
20. Qualitative research is criticised for not being replicable. Each observation of a social
reality is unique to that place, point in time, and observer.
Rebuttal – replicability is not only an issue for qualitative research. It is also a big
issue for experimental work [GV]. A recent study has showed that only 1/3 of
quantitative psychological experiments had been replicated3. This being said,
triangulation can contribute to addressing this issue (see point 32).
21. Another problem with focus groups is reaching consensus or finding shared ideas.
For one, participants who speak the most are usually people with the highest
education level. Researchers may incorrectly consider their opinion as the
consensus. [FD, CD] And what is the threshold to determine that an opinion is
generally shared? How do you deal with different opinions? [DdR] The moderator
3 Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science,
349(6251), aac4716.
5
may spend some time building consensus, but this does not entirely solve the
problem. [AD]
Rebuttal - in focus groups, the idea is to go for maximum variation, not seek for a
consensus. Two focus groups dealing with simple issues may lead to similar results.
But the objective is the opposite: to obtain as diverse results as possible. So you try
to recruit different people for different focus groups. The more obscure and
sensitive the issue, the more diverse the results will be. [MF, GV]
22. Often, qualitative methodologies rely on self-reporting, which has its disadvantages.
This applies equally to quantitative surveys, whereby respondents are also generally
asked to self-report. When asked, people will respond practically anything4. They
will even talk about things they don't know, because it is embarrassing not to have
an answer. People confabulate: they can come up with a lot of reasons to rationalize
their opinions and behaviours, at times not based on real experience. For example,
children may talk about condoms when they haven't even had any sexual
intercourse. So this is a concern – but it is a concern equally for quantitative surveys
as for qualitative research. According to this view, the focus of empirical research
should be on behaviour, not introspection. [BF, TM]
23. When self-reporting is required, qualitative methods may be good for thoughtful,
reflective choice, but not so good for habitual, automatic, non-conscious choice. They
may also not be so good for reporting sensitive issues or issues where social
desirability biases are at play, as people will not express themselves freely. And
what is considered 'sensitive' varies from country to country (e.g. food is a sensitive
issue for Finns). [BF, CRS]
Rebuttal – not all qualitative methods rely on self-reporting. Moreover, sometimes
it’s the self-reporting itself that is of interest, as it discloses people's narratives and
representations (i.e. whatever they have in their head is considered important,
regardless of its relation to actual behaviour). [GV, RvB]
24. Recommended methods for complementing or overcoming the shortcomings of self-
reporting
- "Concept mapping procedure": compared to focus groups, it allows for less social
desirability, more structure and larger sample size. It was developed originally in
sociology for policy issues. People have opinions on certain issues; the goal is to
explore these and their understanding. How? By eliciting concepts through clear
questions. The concepts (or terms or nodes) are then pooled and sorted
4 Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental
processes. Psychological Review, 84(3), 231.
6
according to content. Participants subsequently rate their importance, establish
links between them and discuss the results5. [BF]
- Another technique is to ask people about their experience, instead of relying on
their possible fabrications of experience. In other words, instead of asking
general or hypothetical questions, ask people to recall a particular event or
situation in which they were involved. [BF]
- "Thought listing technique": people are exposed to a message or an event and are
asked to list the thoughts that go through their head. It is a fully unstructured
method. Participants get three minutes to write down what they want. They then
check the difference in salience of one thought between the treatment and the
control group. This could be used as a manipulation check or as a proof of
principle. This technique had been criticised heavily because words were not
equally distributed. [GV] However, it is about relative frequency, i.e. compared
between one treatment and the other. [BF]
- "Projective techniques" could be useful, as they would not rely on people having
to voice what they were thinking. It allows participants a way out. Some
examples include collage, sentence completion, personification and laddering in
means-chains theory6. [BF, FD, MCS]
Some contemporary problems in the use of qualitative methods for policy-making
25. Commodification of qualitative research (i.e. sold by the bulk). Contracting
authorities are often concerned with the number of focus groups and the number of
participants in a focus group. This shouldn't be given too much importance. The
number of participants has little relevance, as long as some minimum standards are
met. Similarly, there is no ideal length of a focus group: they can last half an hour or
half a day. There is no way of saying a priori how much time was needed. The quality
of focus groups is more important. Quality is determined not by the number, size or
duration of the groups, but more so by factors such as how participants will be
selected and recruited, the design of the discussion guide, and the way in which the
qualitative information is analysed. Award criteria for tenders based on the number
of focus groups and participants show a lack of understanding of what qualitative
research tries to achieve. [MF]
5 See, for example: Yin, Y., Vanides, J., Ruiz‐Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison
of two concept‐mapping techniques: Implications for scoring, interpretation, and use. Journal of Research in Science Teaching, 42(2), 166-184. 6 Saadatfard, E. (2014). Means-End Chain Methods Comparison: An Application to Consumer Goods in Iran. International Journal of Economics, Commerce and Management United Kingdom Vol. II, Issue 7, 2014. Available at SSRN: http://ssrn.com/abstract=2481777
7
26. Time constraints. Ideally, qualitative and quantitative methods should be used
sequentially, with qualitative methods used to fine-tune quantitative experiments.
But time constraints were a problem. There is often not enough time for qualitative
results to feed into the design of quantitative research leg. This means they are
often run in parallel because of high time constraints put on contractors. More time
is required, especially at the very beginning of a project – need a proper inception
phase. In behavioural studies for EU policy-making, it would be advisable to allow
contractors more time for these tasks and to have EU officials available during these
phases to quickly respond to the contractors' doubts. Time is thus the biggest
constraint in using qualitative research effectively for these studies. [CD]
27. Timing: qualitative research is often used too late in the process. Because of this, its
impact on policy-making is low, and the opportunity to get full leverage of
qualitative findings is missed. It should be incorporated from the outset to allow for
the depth and richness of qualitative findings to have maximum power to inform
policy development. [MF]
28. Budgetary pressures. These lead to poorer outcomes, because qualitative methods
and analysis involve an iterative process that cannot be done quickly. Budgetary
pressures imply less importance being given to analysis and interpretation, because
they translate to work being done by junior researchers. The real value comes when
translating data (which is now easy to collect) into meaning (which costs money).
[MF]
29. Increased tendency for reportage: Reportage is the tendency in qualitative reporting
to simply feedback what participants say – numerous direct quotes, linked together
with a bit of descriptive text, with minimal analysis, interpretation or attempt to link
the information to the ‘bigger picture’. This makes it harder for policy-makers to
understand and to use for policy, since it is of little value. [MF]
30. There was support to the idea of providing training in behavioural sciences to EU
officials across the European Commission. Designing experiments takes a lot of time
and implies a lot of interactions between contractors and EU officials. It is a very
intensive process during which policy-officers are asked to make decisions, but they
are often hesitation as many of them are not very knowledgeable about behavioural
sciences and the JRC cannot be behind every project. Training could help policy-
makers gaining more confidence to make these decisions. [CD]
8
Good practices for qualitative behavioural studies for EU policy-making7 31. When considering qualitative research…
Be receptive to research proposals that include qualitative methodology.
Move beyond the sterile quantitative-qualitative dichotomy. Simply ask why a
specific method is being proposed. The value added of the qualitative research be
clear (e.g. novel methods shouldn't be chosen just because they are novel)
Remember that the purpose of qualitative methodology is to identify issues, to
see what is out there. Don't expect quantification of what you find in a focus
group (i.e. "most participants thought…"). That is not the purpose of a focus
group.
Do not always expect qualitative research to be followed up with a quantitative
study. Qualitative research has its own place. It is not just a pilot. Depending on
the question it can be a standalone study. Alternatively, qualitative research can
follow the quantitative phase, rather than precede it.
If qualitative work is conducted prior to quantitative research, ask for the
context to be mapped out well. Make sure people have been listened to. Do
expect the qualitative findings to create the environment where the quantitative
research can lead to good results.
32. To ensure quality…
Give emphasis to how the research will be designed and analysed, instead of
buying by the bulk. Ask the bidder to justify why they chose the method they
chose, and how they link to the idea being studied and the objective of the study.
Ask for details in the design of the qualitative methodology used. Ask: "let us
know all the quality standards you intend to apply to guarantee a good result".
Ask for success stories in order to appreciate the value of qualitative research for
policy-makers.
Ask for details on how researchers expect to analyse findings and how they will
translate them into clear policy recommendations.
Consider the benefits of triangulation (of data or of method). Triangulation is
good for the robustness of a study. It allows you to rule out odd effects which
might appear occasionally, but which should not be given importance. Finding
7 This part of the discussion was too dynamic and fast-paced to accurately record who made what claim.
9
the same result in different countries and using different methodologies is a sign
of robustness.
Do ask about inter-coder reliability for increased quality in qualitative studies.
An inter-coder reliability test assesses the degree to which two or more
independent researchers agree on the how to code the same body of content.
This is often not brought into qualitative studies, and could be introduced in the
terms of reference of a study as a guarantee of quality.
33. About interviews and focus groups…
Keep interviews loosely structured – don’t design interview guides as a
restrictive questionnaire, for this will impose words and categories on
participants which they might not have used on their own.
Ask for a good discussion guide, arrived at in a good, well-thought out inception
phase. Allow for that inception phase.
Ensure the right participants for focus groups are selected, setting clearly what
are the selection and exclusion criteria. Focus groups that are well-targeted will
yield more informative results.
Consider the role of the moderator of different focus groups in a study.
Qualitative research recognizes the impact of a researcher's presence on the
object being studied (in this case the focus group interaction). If the results of
similar focus groups are going to be compared, the profile of the moderators in
these groups should be comparable too.
Acknowledgments
This document captures the main points of the discussion at the workshop "Qualitative
Methodology in Behavioural Studies for EU Policy-Making" (Seville, 27 July 2016), and
includes insights from all participants. Drafted by RvB with input from FD, JSL and SRA.
Thanks to MF for comments on an earlier draft.
Participants
Name Surname Institution Initials
Jean-Jacques AMITY CHAFEA J-JA
Femke DE KEULENAER Ipsos FdK
10
Denise DE RIDDER University of Utrecht DdR
Franςois DESSART JRC, European Commission FD
Charlotte DUKE London Economics CD
Allison DUNNE GfK AD
Bob FENNIS University of Groningen BF
Mark FRANCAS TNS MF
Ioannis MAGHIROS JRC, European Commission IM
Tim MITCHELL Decision Technology TM
Silvia PELLA DG JUST, European Commission SP
Sara RAFAEL ALMEIDA JRC, European Commission SRA
M. Carmen RODRIGUEZ SANTOS University of Leon CRS
Nuria RODRIGUEZ-PRIEGO JRC, European Commission NRP
Joana SOUSA LORENҪO JRC, European Commission JSL
René VAN BAVEL JRC, European Commission RvB
Giuseppe VELTRI University of Leicester GV
Agenda
09:15 Arrival at JRC-IPTS and registration
Session 1: Welcome and introduction
09:30 Welcome and tour de table
Ioannis Maghiros, JRC-IPTS
10:15 Objectives of the meeting & setting the scene
Rene van Bavel – Nuria Rodríguez, JRC-IPTS
10:30 Foresight and Behavioural Insights Unit's current and future work
Joana Sousa Lourenco – François Dessart – Sara Rafael Almeida, JRC.DDG.02
11:00 Discussion
11:15 Coffee Break
Session 2: Presentations by research teams: key considerations based on past
11
experience
11:45 BEACONS (Denise de Ridder – Bob Fennis)
12:05 Decision Technology (Tim Mitchell)
12:25 GfK Belgium (Allison Dunne)
12:45 University of Leon (M. Carmen R. Santos)
13:05 London Economics (Charlotte Duke – Femke De Keulenaer)
13:25 London School of Economics (Giuseppe Veltri – Mark Francas)
13:45 Lunch
Session 3: Application to EU policy
14:45 Qualitative methods for EU policy-making: the view from Brussels
Silvia Pella, DG JUST
15:15 Discussion: What should be the role of qualitative methodology in
behavioural studies for EU policy?
a) How can qualitative research reliably contribute to behavioural
studies?
b) How to overcome concerns about lack of representativeness?
c) When to use qualitative methods, when not?
d) What are best practices to ensure successful outcomes?
e) Strengths and weaknesses of interviews, focus groups, participant
observations and analysis of documents
f) Generally, how can qualitative methodology inform the policy-making
process?
g) How can policy-makers be made more aware of the value of qualitative
methodology?
16:15 Coffee break
16:45 Discussion: What should be the role of qualitative methodology in behavioural studies for EU policy? (continuation)
17:15 Closing remarks; any other business
Rene van Bavel, JRC-IPTS
17:30 End of the workshop