Systematic reviews and Meta-analyses

Alison Brettle, Research Fellow (Information)

Salford Centre for Nursing, Midwifery and Collaborative Research

University of Salford

Aims

To discuss the role and process of systematic reviews and meta-analyses

Systematic Review

A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.

What is a systematic review?

An overview of primary research studies conducted according to explicit and reproducible methodology

A rigorous method of summarising research evidence

Shows what we know and don’t know about a topic area

Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

Systematic reviews aim

To find all relevant research studies (published and unpublished)

To assess each study on basis of defined criteria

Synthesise the findings in an unbiased way

Present a balanced and impartial summary of the findings taking any flaws into consideration

Advantages Disadvantages

Advantages of systematic reviews

Summarise evidence, keep people up to date without reading all published research literature

Allow large amounts of data to be assimilated (eg by busy clinicians, policy makers etc)

A clearer picture by collating results of research Break down barriers of getting research into

practice Reduce bias – removes reviewers personal

opinions, preferences and specialist knowledge Explicit methods - allow the reader to assess how

review has been compiled More reliable conclusions because of methods

used

Disadvantages of systematic reviews

Inconclusive conclusions Applicability to practice? Time to undertake Quality?

CRD Guidance

Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decision-makers. When appropriate, combining the results of several studies gives a more reliable and precise estimate of an intervention’s effectiveness than one study alone.5, 6, 7, 8 Systematic reviews adhere to a strict scientific design based on explicit, pre-specified and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. As well as setting out what we know about a particular intervention, systematic reviews can also demonstrate where knowledge is lacking.4, 9 This can then be used to guide future research.10

http://www.york.ac.uk/inst/crd/

Systematic review models

Medical/Health care Cochrane Collaboration, NHS Centre for

Reviews and Dissemination Usually includes “high quality” research

evidence – RCTs Often includes meta-analysis (mathematical

synthesis of results of 2+ studies that addressed same hypothesis in same way)

Social care/Social Sciences SCIE, EPPI Centre, Campbell Collaboration Often include wider range of studies including

qualitative Often narrative synthesis of evidence

Systematic review process

Define/focus the question Develop a protocol Search the literature (possibly 2 stages scoping

and actual searches) Refine the inclusion/exclusion criteria Assess the studies (data extraction tools, 2

independent reviewers) Combine the results of the studies to produce

conclusion– can be a qualitative or quantitative (meta-analysis)

Place findings in context – quality and heterogeniety of studies, applicability of findings

What type of study design?

How effective is paracetamol at reducing pain?

Does smoking increase the risk of oral cancer?

STRONG Experimental studies/ clinical trialsRandomised controlled trialsNon-randomised controlled trials

Observational studies

CohortsCase-controlsCross-sectional surveysCase seriesCase reports

WEAK Expert opinion, consensus

Experimental studies

Randomised controlled trial

Non-randomised controlled clinical trial

Evaluating the effectiveness of an intervention

Observational studies

Cohort

Case-control

Cross-sectional survey

Measuring the incidence of a disease; looking at the causes of disease; determining prognosis

Looking at the causes of disease; identification of risk factors; suitable for examining rare diseases

Measuring the prevalence of a disease; examining the association

What is a meta-analysis?

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis

The process of using statistical methods to combine the results of different studies.

The aim is to integrate the findings, pool the data, and identify the overall trend of results (Dictionary of Epidemiology, 1995)

Focus is the direction and magnitude of effects across studies

A method of aggregating research results – statistical technique for amalgamating, summarising and reviewing previous research

What is meta-analysis?

Using meta-analysis, a wide variety of questions can be investigated, as long as a reasonable body of primary research studies exist.  

Selected parts of the reported results of primary studies are entered into a database, and this "meta-data" is "meta-analysed", in similar ways to working with other data - descriptively and then inferentially to test certain hypotheses.

Meta analysis can be used as a guide to answer the question 'does what we are doing make a difference to X?', even if 'X' has been measured using different instruments across a range of different people.  Meta-analysis provides a systematic overview of quantitative research which has examined a particular question.

When can you do meta-analysis?

Research must be Empirical rather than theoretical Have quantitative results Examine the same constructs and

relationships Have findings that can be put in a

comparable statistical form (e.g. effects sizes or odds ratios)

Are comparable given the question in hand

(adapted from DB Wilson, 1999)

Effect size

The effect size makes meta-analysis possible – it standardises findings across studies so they can be directly compared

Any standardised index can be an effect size (for example, odds ratio, relative risk) as long as it is comparable across studies, represents magnitude and direction of relationship of interest, is independent of sample size

What to include? Must have clear inclusion and exclusion criteria Published studies – publication bias? Important to identify all studies that meet

eligibility criteria

Strengths and weaknesses

Represents findings in a systematic way Can find relationships across studies Clarifies interpretation of studies Can handle large numbers of studies

But Doesn’t always capture more qualitative

distinctions between studies Comparability can require judgement Inclusion of less robust studies Selection bias (reporting of negative findings)

Meta-analysis

Understanding the jargon and the blobs!

The likelihood of something happening

V

The likelihood of something not happening

Odds Ratio, Relative RiskMeasures of risk

Relative risk

A ratio of the probability of the event occuring in the treatment (exposed) group versus a control group (non exposed)

RR = Probability of event in treatment group Probability of event in control group

For example, if the probability of developing an infection in treatment group was 20% and among control 1%, then the relative risk would be 20 and would favour the control group.

Similarly if the probability of developing an infection in treatment group was 4% and among control 10%, then the relative risk would be 0.4 and would favour the treatment group.

A RR of 1.0 = no difference between groups

Odds ratio

The odds of the event in the intervention group divided by the odds of the event in the control group

OR of 1.0 = No difference between groups

OR<1.0 means event is less likely in the intervention group

Odds Ratio Graph (Blobbogram)

2 more than 1

0.5less than 1

1

Line of no significance

LEFTESS

MORIGHTE

Odds Ratio

2 more than 1

0.5less than 1

1

Best estimate

Confidence Interval(wobble factor)

2 more than 1

0.5less than 1

1

Odds Ratio (Blobbogram)

Confidence Interval

Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).

Confidence Intervals(Wobble factor)

Confidence Interval (CI)

= the wobble factor, how sure are we about the results?

- the shorter the CI the more certain we are about the results

the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific outcome

Number needed to treat (NNT)

The p-value in a nutshellHow often you would see a similar result by chance, when

actually there was no effect by the drug or treatment.

p=0.001 Very unlikely 1 in 1000

p=0.05 Fairly unlikely 1 in 20

p=0.5 Fairly likely 1 in 2

p=0.75 Very likely 3 in 4

Impossible Certain Absolutely0 1

Practical Example

Marik PE & Zaloga GP. 2004. Meta-analysis of parenteral nutrition versus enteral nutrition in patients with acute pancreatitis. BMJ; 328:1407 

Objective To compare the safety and clinical outcomes of enteral and parenteral nutrition in patients with acute pancreatitis.

Problem in a ‘nutshell’

Parenteral nutrition: intravenous feeding bypassing ‘eating and digestion’; typically through an infusion pump; complications: bacterial infection

Enteral nutrition: feeding through a feeding tube to the gut Evidence gut is optimal route yet parenteral nutrition remains

widespread In acute pancreatitis parenteral nutrition standard care but

evidence suggest enteral is feasible In acute pancreatitis most sever complication is pancreatic

infection with mortality of up to 80% Studies report parenteral nutrition increases infection rates in

critically ill patients and when compared enteral nutrition is associated with improved immune function and decreased infections

Studies under-powered; differences not always statistically significant; magnitude a treatment effect unknown

From the abstract

Data sources Medline, Embase, Cochrane controlled trials register, and citation review of relevant primary and review articles.

Study selection Randomised controlled studies that compared enteral nutrition with parenteral nutrition in patients with acute pancreatitis. From 117 articles screened, six were identified as randomised controlled trials and were included for data extraction.

Data extraction Six studies with 263 participants were analysed. Descriptive and outcome data were extracted. Main outcome measures were infections, complications other than infections, operative interventions, length of hospital stay, and mortality. The meta-analysis was performed with the random effects model*.

*Random effects models are used when observations are not taken from a simple random sampling to take account of a clustering or multilevel sampling

Relative risks and continuous data outcomes are presented with 95% confidence intervals and chi square tests for heterogeneity

Relative risk is a ratio of the probability of the event occurring in the exposed group versus a non-exposed group.

RR = Probability of event in enteral group Probability of event in parenteral group

For example, if the probability of developing an infection in enteral group was 20% and among parenteral 1%, then the relative risk would be 20 and would favour the parenteral group.

Similarly if the probability of developing an infection in enteral group was 4% and among parenteral 10%, then the relative risk would be 0.4 and would favour the enteral group.

Testing heterogeneity between studies: χ2 test with p≤ 0.05 indicating significant heterogeneity

Fig 1 Process of study selection of randomised controlled trials (TPN=total parenteral nutrition; ENT=enteral nutrition; PN=parenteral nutrition)

Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Table 1 Demographic data of studies included in meta-analysis. Figures are for enteral nutrition/total parenteral nutrition, and scores are given as means (SDs)

No of patients Ranson criteria

Glasgow Score

APACHE II Siting of nasojejunal tube

McClave, 1997

16/16 1.3 (0.35) / 1.3 (0.35)

Enoscopic

Windsor, 1998

16/18 2/2 8 / 9.5 Fluoroscopic

Kalfarentzos, 1997

18/20 4.2 (0.9) / 4.6 (1.1)

12.7 (2.6)/ 11.8 (1.9)

Fluoroscopic

Abou-Assi, 2002

26/27 3.1 (0.5) / 2.5 (0.4)

Fluoroscopic/ endoscopic

Olah, 2002 41/48 2.6 (1.2) / 2.4 (1.6)

Fluoroscopic

Gupta, 2003 8/9 8 / 10 Blind

Table 2 Outcome data of studies included in meta-analysis (figures are for enteral nutrition/total parental nutrition)

No of patients

Septic complications

Other complications

Surgical complications

LoS Mortality

McClave, 1997

16/16 2/2 9.7/11.9

0/0

Windsor, 1998

16/18 0/3 0/5 1/5 12.5/15

0/2

Kalfarentzos, 1997

18/20 5/10 3/5 2/4 40/39 1/2

Abou-Assi, 2002

26/27 1/9 13/17 1/2 14.2 /18.4

6/8

Olah, 2002

41/48 5/13 3/4 5/11 16.8 /23.6

2/4

Gupta, 2003

8/9 0/2 0/6 7/10 0/0

Table 3 Jadad quality score of trials included in meta-analysis

Year Randomisation method

Blinding Withdrawals/drop outs accounted for

Jadad score

McClure, 1997

1997 Not stated None Yes 2

Windsor, 1998

1998 Odd/even hospital number

None Yes 1

Kalfarentzos, 1997

1997 Sealed number envelopes

None Yes 3

Abou-Assi, 2002

2002 Not stated None Yes 2

Olah, 2002 2002 Birth date None Yes 1

Gupta, 2003 2003 Sealed number envelopes

None Yes 3

Results

Infections: Relative risk RR = 0.45, (CI 0.26-0.78), p=0.004 Test for heterogeneity between studies p=0.59 Complications: RR = 0.61 (0.31 – 1.22), p=0.16 Surgical interventions: RR = 0.48 (0.23 - 0.99), p=0.05 χ2 = 0.62, p=0.89 Length of hospital stay: Mean reduction 2.9 days (CI 1.6 – 4.3) χ2 = 16.5, p=0.0056 Mortality RR = 0.66 (0.32 – 1.37), p=0.3

Risk of infection, complications other than infection, surgical intervention, and mortality; results from meta-analyses of randomised trials comparing enteral with parenteral nutrition in pancreatitis

Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Random effects model of relative risk (95% confidence interval) of infections associated with enteral feeding compared with parenteral nutrition

Marik, P. E et al. BMJ 2004;328:1407 Copyright ©2004 BMJ Publishing Group Ltd.

Limitations

Poor quality studies No blinding – may overestimate intervention

effect Different inclusion/exclusion criteria (wide range

of disease severity) Small sample numbers leading to wide confidence

intervals Heterogeneity of studies Possibility of publication bias

Conclusion Evidence does not support use of parenteral

nutrition in patients with acute pancreatitis

Further reading

Greenhalgh T (1997) How to read a paper: papers that summarize other papers (systematic reviews and meta-analyses), BMJ, 315:672-675

Sheldon T (2000) Statistics for evidence based nursing, Evidence Based Nursing, 3; 4-6

Sheldon T (2000) Estimating treatment effects: real or the result of chance? Evidence Based Nursing, 3; 36-39

Further reading

Systematic reviews relevant to your area of practice Do they exist? Are they really applicable? To what extent to they marry up with

your practice?

Useful resources

Cochrane Collaboration http://www.cochrane.org/ http://www.cochrane.org/docs/irmg.htm

Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/

Finding studies for systematic reviews http://www.york.ac.uk/inst/crd/revs.htm

EPPI-Centre – Stages of a review http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89

SCIE - The conduct of systematic research reviews for SCIE knowledge reviews http://www.scie.org.uk/publications/details.asp?

pubID=111