Akbar Soltani. MD,Tehran University of Medical Sciences (TUMS)

Shariati Hospitalwww.soltaniebm.com

How to Read Systematic Review : An Approach for the Clinicians

(part-1)

Systematic Reviews:Objectives:

• Definition of systematic review and meta-analysis?

• Principles of methodology of systematic review

• Discuss differences between systematic review and narrative review

• Consider clinical implications of a Systematic review

Scenario

• At the end of a long week in the office, you sink back into your chair, reflecting on some of the more memorable patients you cared for and counseled.

• Through gentle history taking, you discovered that urinary incontinence is the underlying cause of an elderly patient's increasing social isolation.

Scenario (cont’d)

• You discontinued procainamide therapy in a 72-year-old man who had asymptomatic PVC after MI.

• To prevent bleeding from esophageal varices, you started ß-blocker therapy in a woman with long-standing cryptogenic cirrhosis and portal hypertension.

• You presented the risk factors for major and minor bleeding to a 39-year-old woman who was considering warfarin therapy because of recently diagnosed AF and valvular heart disease.

Scenario (cont’d)

• You stumbled while debating the pros and cons of breast cancer screening with a healthy 48-year-old woman .

• You questioned the merits of a personalized walking program suggested to you by a motivated 66-year-old man with severe claudication.

• Explaining that you wanted to review the best current evidence on these issues, you resolved to address your uncertainties before these patients made their next office visits, in a week's time.

Scenario (cont’d)

• Sighing deeply, you acknowledge that you have little time to read.

• You subscribe to three journals, which you browse months after they arrived either when your journal pile becomes severely high or when your guilt is sufficiently motivational.

• You sometimes find the conclusions of individual articles conflicting or confusing.

Scenario (cont’d)

• You know that some of the decisions and suggestions you made this week, specifically your decisions about stopping procainamide therapy and starting ß-blocker therapy and your advice about bleeding risks from anticoagulant therapy, were based on the best current research evidence .

Scenario (cont’d)

• On the other hand, your patients' inquiries about breast cancer screening and exercise treatment for claudication highlight your need for a concise, current, rigorous synthesis of the best available evidence on each of these topics: in brief, a systematic review.

Definition:

• Science … is the organized systematic activity that gathers knowledge about the world and condenses the knowledge into testable laws and principles. (Edward

O.Wilson; American Scientist, Jan 1998, pg.6.)

Medical literature

Primary (analytic) studies

Experimental

RCTNonrandomizedControlled trial

Observational

cohort case-control cross sectional descriptive, surveys case reports

Secondary (integrative) studies

Systematic reviewNonsystematic reviewMeta-analysisPractice guidelineDecision analysisEconomic analysisEditorial, commentary

Medical literature

Primary (analytic) studies

Experimental

RCTNonrandomizedControlled trial

Observational

cohort case-control cross sectional descriptive, surveys case reports

Secondary (integrative) studies

Systematic reviewNonsystematic reviewMeta-analysisPractice guidelineDecision analysisEconomic analysisEditorial, commentary

Definitions

• Authors sometimes use the terms overview, systematic review, and meta-analysis interchangeably.

• systematic review :any summary that attempts to address a focused clinical question using methods designed to reduce the likelihood of bias

• meta-analysis: describes reviews that use quantitative methods to summarize the results.

Summarizing the evidence is needed whenever there is:

• Substantial uncertainty: contradicting results of studies or effects which vary too much among different types of subjects

In a small randomized double-blind trial of a new treatment for acute myocardial infarction, the mortality in the treated group was half that in the control group, but the difference was not significant. We can conclude that:

a) The treatment is useless

b) there is no point in continuing to develop the treatment

c) the reduction in mortality is so great that we should introduce the treatment immediately

d) we should keep adding cases to the trial until the Normal test for comparison of two proportions is significant

e) we should carry out a new trial of much greater size

Paul Glasziou- University of Oxford

The 17 studies : Forest Plot/Blobbogram

A. Which is the smallest study?

B. Which is the largest study?

C. How many are statistically significant?

D. Which studies are “large enough”?

Paul Glasziou- University of Oxford

The 17 studies: of streptokinase for MI

Paul Glasziou- University of Oxford

Systematic review Or Overview

Comprehensively

• locates

• evaluates

• synthesizes

all (?) the available literature on a given topic

using a strict scientific design which

must itself be reported in the review

A ‘systematic review’, therefore, aims to be:

• Systematic (e.g. in its identification of literature)

• Explicit (e.g. in its statement of objectives, materials and methods)

• Reproducible (e.g. in its methodology and conclusions

Stages of a systematic review

• Planning the review– i.e. identifying the need for a review, and documenting the methodology

• Conducting the review – i.e. finding, selecting, appraising, extracting and synthesizing primary research studies

• Reporting and dissemination – i.e. writing up and disseminating the results of the review

Well formulated question

Comprehensive and pre-defined data search

Unbiased pre-defined selection and extraction process

Critical appraisal of data

Synthesis of data

The Process of Conducting a Systematic Review

Question components : PICO

• What types of Participants?

• What types of

Interventions?

• What types of Comparison?

• What types of Outcomes?

Well formulated question

– study design (e.g., RCT’s?, DBPC?, Cohort & CCS?)– setting (emergency department, outpatient, inpatient)– age (adults only, > 60 only, etc)– year of publication or conduct (esp. if technology or

typical dosing changes)– similarity of exposure or treatment (e.g., drug class, or

dosage)– similarity of outcomes (case definitions)– minimum sample size or follow-up– languages?– pre-1966?

Comprehensive and pre-defined data search

Comprehensive and pre-defined data search

Is finding all published studies enough?

• Negative studies less likely to be published than ‘Positive’

• How does this happen?• Follow-up of 737 studies at Johns Hopkins

(Dickersin, JAMA, 1992)– Positive SUBMITTED more than negative

(2.5 times)

Bias in Systematic Reviews

• Publication bias is the selective publication of manuscripts based on the magnitude, direction, or statistical significance of the study results.

Data sources for a systematic review

• Electronic databases– MEDLINE and EMBASE

– The Cochrane Central Register of Controlled Trials (CENTRAL)

• Hand searching • “Grey literature” ( thesis, Internal reports,

pharmaceutical industry files)

• Checking reference lists • Unpublished sources known to experts in the

specialty seek by personal communication)

• Raw data (from published trials)

Which are biased? Which OK?

1. All studies published in BMJ, Lancet, JAMA or NEJM

2. All publicly funded studies

3. All studies with more than 100 patients

4. All studies conducted in the Northern Hemisphere

5. All studies registered studies

Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined

Published Registered

No. studies 16 13

Survival ratio 1.16 1.05

95% CI 1.06-1.27 0.98-1.12

P-Value 0.02 0.25

Simes, J. Clin Oncol, 86, p1529

Unbiased pre-defined selection and extraction process

Critical appraisal of data

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial

and randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to

treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar

at the start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?

Critical appraisal of data

1. Was there an independent, blind comparison with a reference standard?

2. Did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice?

3. Did the results of the test being evaluated influence the decision to perform the reference standard?

4. Were the methods for performing the test described in sufficient detail to permit replication?

Critical appraisal of data

• when an overview incorporates a specific statistical strategy for assembling the results of several studies into a single estimate

• Systematic reviews do not have to have a meta-analysis

• There are times when it is not appropriate or possible.

Synthesis of data

Systematic Review vs Meta-Analysis

Fig 2: Improvements up to two weeks after steroid injection in knee

Graphical and tabular summary of studies?

Synthesis of data

Synthesis of data

Well formulated question

Comprehensive and pre-defined data search

Unbiased pre-defined selection and extraction process

Critical appraisal of data

Synthesis of data

The Process of Conducting a Systematic Review

Feature Narrative Review-Textbook Systematic Review

Question often broad in scope a focused clinical question

Sources and search

Not usually specifiedpotentially biased

Comprehensive sourcesand explicit search strategy

SelectionNot usually specified

potentially biased

Criterion-based selectionuniformly applied

Appraisal Variable Rigorous critical appraisal

SynthesisOften a qualitative

summaryQuantitative summary

Inferences Sometimes evidence-based

Usually evidence-based

Pros and cons of systematic reviews

• Advantages– Larger numbers & power– Robustness across

PICOs

• Disadvantages– May conclude small

biases are real effects

Where to find systematic reviews

• The Cochrane Library

• DARE (Database of Abstracts of Reviews of Effectiveness)

• The NHS Health Technology Assessment (HTA) Database

The Cochrane Collaboration

Preparing, maintaining and disseminatingsystematic reviews of the effects of health care

Thank You !