Issues and challenges around integrating qualitative and quantitative data
Angela Harden, Professor of Community and Family Health
InCQuuiRES, 8th September, 2011
• A centre for research, innovation and community engagement to promote health and wellbeing.
• 15 members of staff from a range of health and social science disciplines.
• Includes collaborative programme of research with Newham University Hospital NHS Trust (NUHT) focused on women, children and young people
INSTITUTE FOR HEALTH AND HUMAN DEVELOPMENT
Overview of current SR projectsTitle Funder Main collaborating
institutions
The effects of schools and school-environment interventions on health
NIHR LSHTM, UEL, IOE
Can specific approaches to community engagement help to reduce inequalities in health?
NIHR IOE, UEL, LSE
A meta-narrative review of conceptualisations and meanings of ‘community’ within and across research traditions
AHRC UEL
The acceptability, feasibility and transferability of interventions to improve early uptake of antenatal care
NIHR UEL, IOE
Acknowledgements
• Colleagues at the EPPI-Centre, Social Science Research Unit, Institute of Education
• Co-convenors of the Cochrane Qualitative Research Methods Group
Outline
• Drivers and challenges of integration
• Review of approaches and methods
• Comparing methods
• Future challenges and issues
Drivers for integration• Greater recognition of the value of qualitative research in evidence-
based policy• ‘Empty’ reviews• Public perspectives and experiences • Systematic reviews of complex interventions• Issues of process and implementation• Extension of evidence-based health care to other areas of public
policy• Growing tradition for integration in primary research (mixed
methods research)• Dedicated research funding for methodological research• Establishment of dedicated methods groups (e.g. Cochrane
Qualitative Methods Research Group)
Travelling east from Westminster, every two tube stops represent over one year of life expectancy lost –Data revised to 2004-08
Westminster
Waterloo
Southwark
London Bridge
BermondseyCanada
Water
CanaryWharf
NorthGreenwich
Canning Town
London Underground Jubilee Line
Differences in Male Life Expectancy within a small area in London
Electoral wards just a few miles apart geographically have lifeexpectancy spans varying by years. For instance, there are eight stops between Westminster and Canning Townon the Jubilee Line – so as one travels east, every two stops, onaverage, mark over a year of shortened lifespan. 1
River Thames
1 Source: Analysis by London Health Observatory of ONS and GLA data for 2004-08. Diagram produced by Department of Health
Male Life Expectancy73.6 (CI 71.9-75.2)
Male LifeExpectancy78.5 (CI 75.5-81.6)
Mega-events and urban regeneration
• Examples of complex interventions with multiple anticipated outcomes
• Regeneration - a set of activities to reverse economic, social and physical decline
• Housing stock, service improvements, physical environment, creation of socially mixed population
McCartney et al. BMJ 2010; 340:c2369
McCarthy et al. (2010) conclude…….
“There is little evidence that major multi-sport events held between 1978 and 2008 delivered health or socio-economic benefits for the population of the host
country…..”
“Future events such as the 2012 Olympic Games and Paralympic Games, or the 2014 Commonwealth Games, cannot be expected to automatically provide benefits.
Until decision makers include robust, long term evaluations as part of their design and implementation of
events, it is unclear how the costs of major multi-sport events can be justified in terms of benefits to the host
population.”
Unanswered questions
• What happens during implementation?
• How do the various actors involved experience mega-sporting events and the associated regeneration?
• What are the pathways to impact?
• Do variations in the above relate to outcomes?
Challenges of integration
• New and evolving field• Scale of task• Paradigm wars• Lack of training and expertise• Few worked examples• Infrastructure in development
Approaches and methods (1)• Narrative summary• Thematic analysis
• Grounded theory• Meta-ethnography• Meta-study• Miles and Huberman's data
analysis techniques• Content analysis• Case survey• Qualitative comparative
analysis • Bayesian meta-analysis
2004
Approaches and methods ( 2 )
Meta-narrative mapping
Critical Interpretive
synthesis
Realist synthesis
Mixed methodssynthesis
Narrative synthesis
Integration of qualitative
and quantitative
research
Bayesian meta-
analysis
Origins of approaches and methodsMethod Developed by/Exemplars Context and purpose
Narrative synthesis Popay et al. (2005) Cochrane remit – to examine issues of process, implementation and experience
Mixed methods synthesis
Thomas et al. (2004)Harden and Thomas (2005)
Informing policy to promote children’s health
Bayesian synthesis Roberts et al. (2002) Factors that affect the uptake of childhood immunisation
Critical interpretive synthesis
Dixon-Woods et al. (2006) Access to healthcare for vulnerable groups
Meta-narrative Greenhalgh et al. (2005) To review research on diffusion of innovation to inform healthcare policy
Realist synthesis Pawson (2006) To develop and test theories of change underpinning complex policy interventions
Comparing approaches and methodsMethod Idealist –
realist continuum
Deconstruct body of literature?
Mixed methods lens Other characteristics
Narrative synthesis
Realist No Complementary strengths stance
Little iteration in methods
Synthetic product aims to directly address policy
Mixed methods synthesis
Realist No Complementary strengths stance andDialectical stance
Bayesian synthesis
Realist No Alternative or single paradigm stance
Critical interpretive synthesis
Idealist Yes Alternative or single paradigm stance Iterative approach
key
Synthetic product requires interpretation
Meta-narrative Idealist Yes Dialectical stance
Realist synthesis Idealist No Alternative or single paradigm stance
In focus: Mixed methods synthesis
• Aim is to generate and test theory from diverse body of literature
• Exhaustive search, review questions, inclusion criteria and quality assessment largely specified a priori
• Each review typically has three syntheses: 1. Statistical meta-analysis2. Thematic synthesis3. Cross-study synthesis
Recent examples
Teenage pregnancy and social disadvantage (Harden et al., 2009)
Knowledge and information needs of young people with epilepsy (Lewis
et al., 2010)Behavioural interventions for weight
management in pregnancy (Campbell et al., 2011)
The effects of schools and school environment interventions (Bonell
et al., 2011)
Mixed methods systematic reviews
Working definition
Combining the findings of ‘qualitative’ and ‘quantitative’ studies within a single systematic
review, in order to address the same, overlapping or complementary review
questions(Harden and Thomas, 2010)
Three ways in which reviews are mixed….
1. The types of studies included and hence the type of findings to be synthesised (i.e. ‘qualitative/ textual and quantitative/numerical)
2. The types of synthesis method used (e.g. statistical meta-analysis and qualitative synthesis)
3. The mode of analysis: theory testing AND theory building
Review questions•What is known about the barriers to, and facilitators of, healthy eating amongst children?
•Do interventions promote healthy eating amongst children?
•What are children’s perspectives on healthy eating?
•What are the implications of the above for intervention development?
REVIEW PROCESS
Searching, screening and mapping
Synthesis 1: Trials (n=33)1. Quality assessment
2. Data extraction3. Statistical meta-analysis
Synthesis 2: Qualitative studies (n=8)
1. Quality assessment2. Data extraction
3. Thematic synthesis
Synthesis 3: Trials and qualitative studies
Focus narrowed to ‘fruit &veg’
Children’s views Trials
Recommendation for interventions
Good quality
Other
Do not promote fruit and vegetables in the same way
0 0
Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one
5 5
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health
5 6
Synthesis 3: Across studies
Increase (standardised portions per day) in vegetable intake across trials
0
0.1
0.2
0.3
0.4
0.5
0.6
War
dle
Liquo
ri
Henry
Ander
son
Reyno
lds Auld
Auld (b
)
Baran
owsk
i
Perry
Study
Po
rtio
ns
Little or no emphasis on health messages
Synthesis 3: Across studies
In focus: Bayesian synthesis
• Aim is to test theory• Findings from qualitative and
quantitative research are ‘fused’• Only three worked examples to
date• Variation in weight given to the
qualitative evidence• Synthesis product is a set of
weighted factors associated with/predicting the phenomenon under review
Recent examples
Factors affecting the uptake of childhood immunisation
(Roberts et al., 2002)
Factors that influence adherence to HIV
medication regimes (Vollis et al., 2009)
(Crandell et al., 2011)
Factors that influence adherence to HIV medication (Crandell et al., 2011)
In focus: Critical Interpretive Synthesis
• Aim is to generate theory from large and diverse body of literature
• Literature itself is an object of scrutiny (critical)
• Comprehensive search to identify sampling frame
• Purposive and theoretical sampling
• Analysis leads to generation of synthetic constructs and a synthesising argument
Recent examples
Access to health care for vulnerable groups (Dixon-
Woods et al. 2006)
Use of morphine to treat cancer related pain (Flemming, 2009)
Nurses response to suicide and suicidal patients (Talseth and
Gilje, 2011
Access to health care for vulnerable groups (Dixon-Woods et al. 2006)
• Based on 119 papers
• Interpretive qualitative analysis of diverse types of studies
• Conceptual and methodological problems with measures of health service utilisation
• Synthesising argument organised around a set of central concepts (e.g. navigation, adjudications) with the synthetic construct of ‘candidacy’ at the core
Access to health care for vulnerable groups (Dixon-Woods et al. 2006)
Core construct – candidacy
“Candidacy describes the ways in which people's eligibility for
medical attention and intervention is jointly negotiated between
individuals and health services…….candidacy is a dynamic and contingent
process, constantly being defined and redefined through
interactions between individuals and professionals,
including how "cases" are constructed.”
Central concepts
• Identification of candidacy• Navigation
• Permeability of services• Appearances at health services
• Adjudications• Offers and resistance
• Operating conditions and local production of candidacy
In focus: Meta-narrative review
• Aim is to make sense of and understand diverse bodies of literature and their findings
• Literature itself is an object of scrutiny (critical)
• Searching is iterative, ‘snowballing’ a key technique
• Analysis leads to production of a set of meta-narratives (‘storylines of research’)
Recent examples
Spread and sustainability of innovations in health service
delivery and organisation (Greenhalgh et al., 2005)
Understanding the use of electronic patient records in health care
organisations (Greenhalgh et al., 2009)
See also the:Realist and meta-narrative evidence
synthesis evolving standards project (RAMASES) (Greenhalgh et
al., 2011)
A meta-narrative approach
• The influence of Kuhn’s ‘paradigms’ (1962) and the makings of the first meta-narrative
• The essential technique is interpretive synthesis exploring distinct research traditions, each with its own meta-narrative
• Methods of ‘unpacking’ the meta-narrative: exploratory methods; expert consultations; snowballing; database searching
Stages of a meta-narrative review (from Greenhalgh et al., 2009)
Meta-narratives identified in the electronic patient records review (from Greenhalgh et al., 2009)
Comparing approaches and methodsMethod Idealist –
realist continuum
Deconstruct body of literature?
Mixed methods lens Other characteristics
Mixed methods synthesis
Realist No Complementary strengths stance andDialectical stance Little iteration in
methods
Synthetic product aims to directly address policy
Bayesian synthesis
Realist No Alternative or single paradigm stance
Critical interpretive synthesis
Idealist Yes Alternative or single paradigm stance Iterative approach
key
Synthetic product requires interpretation
Meta-narrative Idealist Yes Dialectical stance
Future challenges• More worked examples are key• Focus on methods and tools for the actual
integration– Enhancing transparency– Establishing rigour
• Further conceptual work to illuminate points of difference, strengths and weaknesses, fit for purpose
• Learning from, and contributing to, the mixed methods literature for primary research.
Thank [email protected]
Forthcoming book
Gough D, Oliver S, Thomas J (Eds) (forthcoming, Feb 2012) An Introduction to Systematic Reviews. London: Sage