Systematic reviews and Meta-analyses

Alison Brettle, Research Fellow (Information)

Salford Centre for Nursing, Midwifery and Collaborative Research

University of Salford

Aims

To discuss the role of the systematic reviews and meta-analyses and cover issues involved in their critical appraisal and interpretation

Systematic Review

A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.

What is a systematic review?

An overview of primary research studies conducted according to explicit and reproducible methodology

A rigorous method of summarising research evidence

Shows what we know and don’t know about a topic area

Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

Systematic reviews aim

To find all relevant research studies (published and unpublished)

To assess each study on basis of defined criteria

Synthesise the findings in an unbiased way

Present a balanced and impartial summary of the findings taking any flaws into consideration

Advantages of systematic reviews

Summarise evidence, keep people up to date without reading all published research literature

Allow large amounts of data to be assimilated (eg by busy clinicians, policy makers etc)

A clearer picture by collating results of research Reduce bias – removes reviewers personal

opinions, preferences and specialist knowledge Explicit methods - allow the reader to assess how

review has been compiled More reliable conclusions because of methods

used

Systematic review models

Medical/Health care Cochrane Collaboration, NHS Centre for

Reviews and Dissemination Usually includes “high quality” research

evidence – RCTs Often includes meta-analysis (mathematical

synthesis of results of 2+ studies that addressed same hypothesis in same way)

Social care/Social Sciences SCIE, EPPI Centre, Campbell Collaboration Often include wider range of studies including

qualitative Often narrative synthesis of evidence

Systematic review process

Define/focus the question Develop a protocol Search the literature (possibly 2 stages scoping

and actual searches) Refine the inclusion/exclusion criteria Assess the studies (data extraction tools, 2

independent reviewers) Combine the results of the studies to produce

conclusion– can be a qualitative or quantitative (meta-analysis)

Place findings in context – quality and heterogeniety of studies, applicability of findings

Methodology for a systematic review of randomised controlled trials1

Greenhalgh, T, BMJ 1997;315:672-675

What type of study design?

How effective is paracetamol at reducing pain?

Does smoking increase the risk of oral cancer?

STRONG Experimental studies/ clinical trialsRandomised controlled trialsNon-randomised controlled trials

Observational studies

CohortsCase-controlsCross-sectional surveysCase seriesCase reports

WEAK Expert opinion, consensus

Experimental studies

Randomised controlled trial

Non-randomised controlled clinical trial

Evaluating the effectiveness of an intervention

Observational studies

Cohort

Case-control

Cross-sectional survey

Measuring the incidence of a disease; looking at the causes of disease; determining prognosis

Looking at the causes of disease; identification of risk factors; suitable for examining rare diseases

Measuring the prevalence of a disease; examining the association

What is a meta-analysis?

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis

The process of using statistical methods to combine the results of different studies.

The aim is to integrate the findings, pool the data, and identify the overall trend of results

(Dictionary of Epidemiology, 1995)

Systematic Reviews

Understanding the jargon and the blobs!

The likelihood of something happening

V

The likelihood of something not happening

Odds Ratio, Relative RiskMeasures of risk

Odds Ratio Graph (Blobbogram)

2 more than 1

0.5less than 1

1

Line of no significance

LEFTESS

MORIGHTE

Odds Ratio

2 more than 1

0.5less than 1

1

Best estimate

Confidence Interval(wobble factor)

2 more than 1

0.5less than 1

1

Odds Ratio (Blobbogram)

Confidence Interval

Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).

Confidence Intervals(Wobble factor)

Confidence Interval (CI)

= the wobble factor, how sure are we about the results?

- the shorter the CI the more certain we are about the results

- if it crosses the line of 1 (no treatment effect) the intervention might not be doing any good and could be doing harm

Heterogeneity

Clinical heterogeneity – differences in trial characteristics

Statistical heterogeneity - the variability in the reported effect sizes between studies

how similar are the results?

are the differences among the results of the trials greater than could be expected by chance alone?

the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific outcome

Number needed to treat (NNT)

The p-value in a nutshellHow often you would see a similar result by chance, when

actually there was no effect by the drug or treatment.

p=0.001 Very unlikely 1 in 1000

p=0.05 Fairly unlikely 1 in 20

p=0.5 Fairly likely 1 in 2

p=0.75 Very likely 3 in 4

Impossible Certain Absolutely0 1

Critical appraisal

Is the study valid? Trustworthy

What are the results? Is it useful in practice?

Relevant? Generalisable?

Evaluating quality of systematic reviews

Is there a clearly defined question? Thorough and comprehensive search Was methodological quality assessed and

studies weighted accordingly? (Were studies reliable and valid?)

How sensitive are the results to the way the review was done – ie if you changed the inclusion criteria how would this affect results?

Interpretation of numerical results

Further reading

Greenhalgh T (1997) How to read a paper: papers that summarize other papers (systematic reviews and meta-analyses), BMJ, 315:672-675

Useful resources

Cochrane Collaboration http://www.cochrane.org/ http://www.cochrane.org/docs/irmg.htm

Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/

Finding studies for systematic reviews http://www.york.ac.uk/inst/crd/revs.htm

EPPI-Centre – Stages of a review http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89

SCIE - The conduct of systematic research reviews for SCIE knowledge reviews http://www.scie.org.uk/publications/details.asp?

pubID=111