Systematic reviews and Meta-analyses
Alison Brettle, Research Fellow (Information)
Salford Centre for Nursing, Midwifery and Collaborative Research
University of Salford
Aims
To discuss the role and process of systematic reviews and meta-analyses
Systematic Review
A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.
What is a systematic review?
An overview of primary research studies conducted according to explicit and reproducible methodology
A rigorous method of summarising research evidence
Shows what we know and don’t know about a topic area
Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence
Systematic reviews aim
To find all relevant research studies (published and unpublished)
To assess each study on basis of defined criteria
Synthesise the findings in an unbiased way
Present a balanced and impartial summary of the findings taking any flaws into consideration
Advantages Disadvantages
Advantages of systematic reviews
Summarise evidence, keep people up to date without reading all published research literature
Allow large amounts of data to be assimilated (eg by busy clinicians, policy makers etc)
A clearer picture by collating results of research Break down barriers of getting research into
practice Reduce bias – removes reviewers personal
opinions, preferences and specialist knowledge Explicit methods - allow the reader to assess how
review has been compiled More reliable conclusions because of methods
used
Disadvantages of systematic reviews
Inconclusive conclusions Applicability to practice? Time to undertake Quality?
CRD Guidance
Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decision-makers. When appropriate, combining the results of several studies gives a more reliable and precise estimate of an intervention’s effectiveness than one study alone.5, 6, 7, 8 Systematic reviews adhere to a strict scientific design based on explicit, pre-specified and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. As well as setting out what we know about a particular intervention, systematic reviews can also demonstrate where knowledge is lacking.4, 9 This can then be used to guide future research.10
http://www.york.ac.uk/inst/crd/
Systematic review models
Medical/Health care Cochrane Collaboration, NHS Centre for
Reviews and Dissemination Usually includes “high quality” research
evidence – RCTs Often includes meta-analysis (mathematical
synthesis of results of 2+ studies that addressed same hypothesis in same way)
Social care/Social Sciences SCIE, EPPI Centre, Campbell Collaboration Often include wider range of studies including
qualitative Often narrative synthesis of evidence
Systematic review process
Define/focus the question Develop a protocol Search the literature (possibly 2 stages scoping
and actual searches) Refine the inclusion/exclusion criteria Assess the studies (data extraction tools, 2
independent reviewers) Combine the results of the studies to produce
conclusion– can be a qualitative or quantitative (meta-analysis)
Place findings in context – quality and heterogeniety of studies, applicability of findings
What type of study design?
How effective is paracetamol at reducing pain?
Does smoking increase the risk of oral cancer?
STRONG Experimental studies/ clinical trialsRandomised controlled trialsNon-randomised controlled trials
Observational studies
CohortsCase-controlsCross-sectional surveysCase seriesCase reports
WEAK Expert opinion, consensus
Experimental studies
Randomised controlled trial
Non-randomised controlled clinical trial
Evaluating the effectiveness of an intervention
Observational studies
Cohort
Case-control
Cross-sectional survey
Measuring the incidence of a disease; looking at the causes of disease; determining prognosis
Looking at the causes of disease; identification of risk factors; suitable for examining rare diseases
Measuring the prevalence of a disease; examining the association
What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis
The process of using statistical methods to combine the results of different studies.
The aim is to integrate the findings, pool the data, and identify the overall trend of results (Dictionary of Epidemiology, 1995)
Focus is the direction and magnitude of effects across studies
A method of aggregating research results – statistical technique for amalgamating, summarising and reviewing previous research
What is meta-analysis?
Using meta-analysis, a wide variety of questions can be investigated, as long as a reasonable body of primary research studies exist.
Selected parts of the reported results of primary studies are entered into a database, and this "meta-data" is "meta-analysed", in similar ways to working with other data - descriptively and then inferentially to test certain hypotheses.
Meta analysis can be used as a guide to answer the question 'does what we are doing make a difference to X?', even if 'X' has been measured using different instruments across a range of different people. Meta-analysis provides a systematic overview of quantitative research which has examined a particular question.
When can you do meta-analysis?
Research must be Empirical rather than theoretical Have quantitative results Examine the same constructs and
relationships Have findings that can be put in a
comparable statistical form (e.g. effects sizes or odds ratios)
Are comparable given the question in hand
(adapted from DB Wilson, 1999)
Effect size
The effect size makes meta-analysis possible – it standardises findings across studies so they can be directly compared
Any standardised index can be an effect size (for example, odds ratio, relative risk) as long as it is comparable across studies, represents magnitude and direction of relationship of interest, is independent of sample size
What to include? Must have clear inclusion and exclusion criteria Published studies – publication bias? Important to identify all studies that meet
eligibility criteria
Strengths and weaknesses
Represents findings in a systematic way Can find relationships across studies Clarifies interpretation of studies Can handle large numbers of studies
But Doesn’t always capture more qualitative
distinctions between studies Comparability can require judgement Inclusion of less robust studies Selection bias (reporting of negative findings)
Meta-analysis
Understanding the jargon and the blobs!
The likelihood of something happening
V
The likelihood of something not happening
Odds Ratio, Relative RiskMeasures of risk
Relative risk
A ratio of the probability of the event occuring in the treatment (exposed) group versus a control group (non exposed)
RR = Probability of event in treatment group Probability of event in control group
For example, if the probability of developing an infection in treatment group was 20% and among control 1%, then the relative risk would be 20 and would favour the control group.
Similarly if the probability of developing an infection in treatment group was 4% and among control 10%, then the relative risk would be 0.4 and would favour the treatment group.
A RR of 1.0 = no difference between groups
Odds ratio
The odds of the event in the intervention group divided by the odds of the event in the control group
OR of 1.0 = No difference between groups
OR<1.0 means event is less likely in the intervention group
Odds Ratio Graph (Blobbogram)
2 more than 1
0.5less than 1
1
Line of no significance
LEFTESS
MORIGHTE
Odds Ratio
2 more than 1
0.5less than 1
1
Best estimate
Confidence Interval(wobble factor)
2 more than 1
0.5less than 1
1
Odds Ratio (Blobbogram)
Confidence Interval
Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).
Confidence Intervals(Wobble factor)
Confidence Interval (CI)
= the wobble factor, how sure are we about the results?
- the shorter the CI the more certain we are about the results
the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific outcome
Number needed to treat (NNT)
The p-value in a nutshellHow often you would see a similar result by chance, when
actually there was no effect by the drug or treatment.
p=0.001 Very unlikely 1 in 1000
p=0.05 Fairly unlikely 1 in 20
p=0.5 Fairly likely 1 in 2
p=0.75 Very likely 3 in 4
Impossible Certain Absolutely0 1
Practical Example
Marik PE & Zaloga GP. 2004. Meta-analysis of parenteral nutrition versus enteral nutrition in patients with acute pancreatitis. BMJ; 328:1407
Objective To compare the safety and clinical outcomes of enteral and parenteral nutrition in patients with acute pancreatitis.
Problem in a ‘nutshell’
Parenteral nutrition: intravenous feeding bypassing ‘eating and digestion’; typically through an infusion pump; complications: bacterial infection
Enteral nutrition: feeding through a feeding tube to the gut Evidence gut is optimal route yet parenteral nutrition remains
widespread In acute pancreatitis parenteral nutrition standard care but
evidence suggest enteral is feasible In acute pancreatitis most sever complication is pancreatic
infection with mortality of up to 80% Studies report parenteral nutrition increases infection rates in
critically ill patients and when compared enteral nutrition is associated with improved immune function and decreased infections
Studies under-powered; differences not always statistically significant; magnitude a treatment effect unknown
From the abstract
Data sources Medline, Embase, Cochrane controlled trials register, and citation review of relevant primary and review articles.
Study selection Randomised controlled studies that compared enteral nutrition with parenteral nutrition in patients with acute pancreatitis. From 117 articles screened, six were identified as randomised controlled trials and were included for data extraction.
Data extraction Six studies with 263 participants were analysed. Descriptive and outcome data were extracted. Main outcome measures were infections, complications other than infections, operative interventions, length of hospital stay, and mortality. The meta-analysis was performed with the random effects model*.
*Random effects models are used when observations are not taken from a simple random sampling to take account of a clustering or multilevel sampling
Relative risks and continuous data outcomes are presented with 95% confidence intervals and chi square tests for heterogeneity
Relative risk is a ratio of the probability of the event occurring in the exposed group versus a non-exposed group.
RR = Probability of event in enteral group Probability of event in parenteral group
For example, if the probability of developing an infection in enteral group was 20% and among parenteral 1%, then the relative risk would be 20 and would favour the parenteral group.
Similarly if the probability of developing an infection in enteral group was 4% and among parenteral 10%, then the relative risk would be 0.4 and would favour the enteral group.
Testing heterogeneity between studies: χ2 test with p≤ 0.05 indicating significant heterogeneity
Fig 1 Process of study selection of randomised controlled trials (TPN=total parenteral nutrition; ENT=enteral nutrition; PN=parenteral nutrition)
Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.
Table 1 Demographic data of studies included in meta-analysis. Figures are for enteral nutrition/total parenteral nutrition, and scores are given as means (SDs)
No of patients Ranson criteria
Glasgow Score
APACHE II Siting of nasojejunal tube
McClave, 1997
16/16 1.3 (0.35) / 1.3 (0.35)
Enoscopic
Windsor, 1998
16/18 2/2 8 / 9.5 Fluoroscopic
Kalfarentzos, 1997
18/20 4.2 (0.9) / 4.6 (1.1)
12.7 (2.6)/ 11.8 (1.9)
Fluoroscopic
Abou-Assi, 2002
26/27 3.1 (0.5) / 2.5 (0.4)
Fluoroscopic/ endoscopic
Olah, 2002 41/48 2.6 (1.2) / 2.4 (1.6)
Fluoroscopic
Gupta, 2003 8/9 8 / 10 Blind
Table 2 Outcome data of studies included in meta-analysis (figures are for enteral nutrition/total parental nutrition)
No of patients
Septic complications
Other complications
Surgical complications
LoS Mortality
McClave, 1997
16/16 2/2 9.7/11.9
0/0
Windsor, 1998
16/18 0/3 0/5 1/5 12.5/15
0/2
Kalfarentzos, 1997
18/20 5/10 3/5 2/4 40/39 1/2
Abou-Assi, 2002
26/27 1/9 13/17 1/2 14.2 /18.4
6/8
Olah, 2002
41/48 5/13 3/4 5/11 16.8 /23.6
2/4
Gupta, 2003
8/9 0/2 0/6 7/10 0/0
Table 3 Jadad quality score of trials included in meta-analysis
Year Randomisation method
Blinding Withdrawals/drop outs accounted for
Jadad score
McClure, 1997
1997 Not stated None Yes 2
Windsor, 1998
1998 Odd/even hospital number
None Yes 1
Kalfarentzos, 1997
1997 Sealed number envelopes
None Yes 3
Abou-Assi, 2002
2002 Not stated None Yes 2
Olah, 2002 2002 Birth date None Yes 1
Gupta, 2003 2003 Sealed number envelopes
None Yes 3
Results
Infections: Relative risk RR = 0.45, (CI 0.26-0.78), p=0.004 Test for heterogeneity between studies p=0.59 Complications: RR = 0.61 (0.31 – 1.22), p=0.16 Surgical interventions: RR = 0.48 (0.23 - 0.99), p=0.05 χ2 = 0.62, p=0.89 Length of hospital stay: Mean reduction 2.9 days (CI 1.6 – 4.3) χ2 = 16.5, p=0.0056 Mortality RR = 0.66 (0.32 – 1.37), p=0.3
Risk of infection, complications other than infection, surgical intervention, and mortality; results from meta-analyses of randomised trials comparing enteral with parenteral nutrition in pancreatitis
Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.
Random effects model of relative risk (95% confidence interval) of infections associated with enteral feeding compared with parenteral nutrition
Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.
Limitations
Poor quality studies No blinding – may overestimate intervention
effect Different inclusion/exclusion criteria (wide range
of disease severity) Small sample numbers leading to wide confidence
intervals Heterogeneity of studies Possibility of publication bias
Conclusion Evidence does not support use of parenteral
nutrition in patients with acute pancreatitis
Further reading
Greenhalgh T (1997) How to read a paper: papers that summarize other papers (systematic reviews and meta-analyses), BMJ, 315:672-675
Sheldon T (2000) Statistics for evidence based nursing, Evidence Based Nursing, 3; 4-6
Sheldon T (2000) Estimating treatment effects: real or the result of chance? Evidence Based Nursing, 3; 36-39
Further reading
Systematic reviews relevant to your area of practice Do they exist? Are they really applicable? To what extent to they marry up with
your practice?
Useful resources
Cochrane Collaboration http://www.cochrane.org/ http://www.cochrane.org/docs/irmg.htm
Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/
Finding studies for systematic reviews http://www.york.ac.uk/inst/crd/revs.htm
EPPI-Centre – Stages of a review http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89
SCIE - The conduct of systematic research reviews for SCIE knowledge reviews http://www.scie.org.uk/publications/details.asp?
pubID=111