Systematic reviews as a source of useful evidencethe experience of the EPPI-centre

TOWARD AN EVIDENCE-BASED DEVELOPMENT POLICYLaunch of the UK office of 3ie at LIDC11 October 2010, London

Sandy Oliver

2

Outline

The EPPI-Centre story of systematic reviews• Addressing knowledge needs• Clinical model to social influence model • Diversity in methodology

The broader history of systematic reviews for decision-making • Innovation and challenge• Achievements and culture change

Capacity building for development policy• Barriers and drivers for innovation

3

Part 1: The EPPI-Centre story

Systematic reviews/ evidence syntheses

• Answer important questions by looking at prior research• Bring together and ‘pool’ the findings of primary research• Not just a report of the findings of the individual studies in

a review• Synthesis of the findings of all the included studies in

order to answer the review question

Reviews should (just like primary research):

• be pieces of research - following principled methods and a research question, with a protocol and reflection on own strengths and limitations

• take steps to reduce hidden ‘bias’ and ‘error’• be accountable, and in some sense replicable and

updateable

7

Support and tools for review groups: (60+ groups, c130 reviews),

Education, criminology, employment,speech and language, social care,

International development

Conducting reviews since 1993 In health promotion, education,

transport, social care, work and pensions

On-line libraries of research evidence

Short courses and Masters level courses in research synthesis

Formal links with Cochrane and Campbell Collaborations

Methodological work, e.g. Methods for Research Synthesis Project

ESRC National Centre for Research Methods

8

Addressing knowledge needs

Early reviews disappointing…Few randomised controlled trials evaluating health promotion in the community, mostly American

Randomised controlled trials evaluating clinic-based health promotion did not provide sufficient evidence to convince practitioners and service planners

9

Knowledge needs assessment

Service planners wanted reviews reporting...• clear justifiable methods

AND • questions framed with professionals and service users• detailed descriptions of interventions, resources and

training• whether interventions worked, AND how and why

11

Peer delivered health promotion

What is the effectiveness?• > half trials showed at least one positive effect on behaviour• ? specific characteristics of an effective model• peer leaders delivering didactic information does not change behaviour What is the appropriateness?• negative views about peer education • not all schools could cope with adult/ young people partnerships• young people at greater risk of adverse health behaviours difficult to reach• young men more reluctant to take on the role of peer educator

Harden A, Oakley A, Oliver S (2001) Peer-delivered health promotion for young people: A systematic review of different study designs. Health Education Journal 60: 339-353.

12

Smoking cessation in pregnancyPractitioner response to systematic reviews

1995Statistical meta-analysesNo mention of• Potential harms• Theory underpinning

interventions• Emotional and social outcomes• Social context of women

smoking• Information for implementationTotally inadequate justification for intervening in women’s lives

13

Smoking cessation in pregnancystakeholder influence of evidence

1995Statistical meta-analysesNo mention of• Potential harms• Theory underpinning

interventions• Emotional and social outcomes• Social context of women

smoking• Information for implementationTotally inadequate justification for intervening in women’s lives

2005Women’s views soughtHealth promotion practitioners’ views soughtReview addresses• Potential harms• Theory underpinning

interventions• Emotional and social outcomes• Social context of women

smoking• Information for implementationCited in policy support documents internationally

Rapid reviewsto meet policy timelines

• To inform the work of the Prime Minister's Commission on the Future of Nursing and Midwifery

The socioeconomic value of nursing and midwifery: a rapid systematic review of reviews

• To inform NICE Guidance on Quitting smoking in pregnancy and following childbirth

An equity analysis of a Cochrane systematic review (72 RCTs) of smoking cessation in pregnancy.

14

15

Clinical model to social influence model

16

Social determinants of health

• Age, gender and constitutional factors• Individual lifestyle factors• Social and community networks• Living and working conditions

Agricultural and food production, education, work environment, unemployment, water and sanitation, health care services, housing

• General socio-economic, cultural and environmental conditions

17

Social determinants of health

Age, gender and constitutional factors Individual lifestyle factors Social and community networks Living and working conditions

Agricultural and food production, education, work environment, unemployment, water and sanitation, health care services, housing

General socio-economic, cultural and environmental conditions

18

Young people and physical activityEnhancing: Individuals’ knowledge

classroom teaching Social and community support:

engaging parents in supporting/ encouraging children's activity multi-component, multi-site interventions using a combination of

school-based physical education and home-based activities.   Living and working conditions:

education and provision of equipment for monitoring TV or video-game use

Brunton G, Thomas J, Harden A, Rees R, Kavanagh J, Oliver S, Shepherd J, Oakley A (2005) Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Health Education Journal 64: 323-338.

19

Socio-economic position: PROGRESS

PROGRESS

Place of residence

Race/ethnicity

Occupation

Gender

Religion

Education

Socio-economic status

Social capital

PROGRESS-Plus

PROGRESS +• Characteristics subject to prejudice, discrimination and stigma

E.g. age, disability, sexual orientation

• Vulnerable people

E.g. older people returning home from hospital or respite care

• Excluded people• E.g. children not attending school, ‘looked after’ children or runaways

Reviews with an ‘Equity lens’• No studies presented impact data

related to participants’ gender, age, religion, education or social capital.

• Pooling the findings in a statistical meta-analysis suggested that interventions might be less effective for people who are more socio-economically disadvantaged

• Kavanagh J, Oliver S, Lorenc T, Caird J, Tucker H, Harden A, Greaves A, Thomas J, Oakley A (2009) School-based cognitive-behavioural interventions: A systematic review of effects and inequalities.  Health Sociology Review, 18: 61-78.

22

Diversity in methodology

A systematic review…

For testing a hypothesis:

Cognitive behavioural interventions in school for young people’s mental health

A systematic review…

For building theory Food in the

school

Chosen foods

Provided foods

Food in the home

Influences on foods eaten

Food preferences

Non-influencing factors

Health benefits

Knowledge behaviour gap

Roles and responsibilities

Healthy eating concepts (understanding)

‘Good’ and ‘bad’ foods

Health consequences

Limited choices

Eating to socialize

Contradictions

Breaking rules

Food rules

Understandings of healthy eating

Synthesis methods

Meta-ethnography

Thematic analysis

Realist review

Textual narrative review

Framework analysis

Tabular methods

Ecological triangulation

Meta-analysis

Meta-regression

Synthesis methods

Meta-ethnography

Thematic analysis

Realist review

Textual narrative review

Framework analysis

Tabular methods

Ecological triangulation

Meta-analysis

Meta-regression

Theory building

Theory testing

Deriving Deriving descriptivedescriptive themesthemes

Visualisation of descriptive Visualisation of descriptive themesthemes

28

Part 2: The broader history of systematic reviews for

decision-making

29

Innovation and challenge

30

Innovation and challenge

In Medicine• Challenged consultants’ authority within professionIn Nursing and Maternity Care• Challenged value of ‘tender loving care’• Challenged professional knowledge over patientsIn health promotion• Challenged community development and activismIn education• Challenged ‘craft skills’ and ‘experiential knowledge’ of teachers• Challenged qualitative research tradition in higher education

31

Achievements and culture change

Changes in policy and practice

• Evidence-informed practitioners• Evidence-informed shared decision-making• Evidence-informed activism• Evidence-informed guidelines across clinical health and public health

Changes in systematic reviews

• Inspired the Cochrane and Campbell Collaborations• Wealth of evidence readily accessible, mainstreamed within guidance• Improved the keywording of trials on MEDLINE• Greater range of useful reviews

– Reviews to generate, explore and test theories– Reviews of complex interventions– Reviews of impact, implementation, sustainability, theory, context and

equity

Changes in primary research

• Raised the quality of impact evaluations• Raised the accessibility of publications reporting trials• Inspired a growth in qualitative nursing research• Inspired a striving for relevance: in outcomes and questions

35

Capacity building for development policy

36

Barriers to capacity building• Infrastructure: Research and knowledge resources and conventions (e.g. availability of databases, and keywording of available databases)• Information and communication technology: access to databases, and internet connectivity• Language barriers: languages of publication, language skills of review teams• Methodological innovation is required for synthesising evidence in any new area • Novice reviewers learning review skills and simultaneously facing all the challenges above

37

What drives innovation?• Synthesising different types of data (quantitative, qualitative and mixed data)• Researching new settings or populations with different social and political contexts. • Requires:

– interdisciplinary working to prepare systematic reviews– engaging new potential users of systematic reviews– transparency and clarity to enable debate about methods and how they

are applied– multidisciplinary teams with expertise both in synthesis methods and in

development studies– a degree of risk taking (trial and error)

Anticipate…

• Challenges to policy makers, practitioners and researchers• Mutual learning• Raised standards for doing and using research• New methods for research and working together

• “If you are poor you actually need more evidence than if you are rich”Dr Hassan Mshinda, Ifakara Centre, Tanzania

39

www.ioe.ac.uk/ssru/http://eppi.ioe.ac.uk

s.oliver@ioe.ac.uk

Thank you

EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/