Session 5: Assessing Multiple Studies Peter Tarczy-Hornoch MD Head and Professor, Division of BHI Professor, Division of Neonatology Adjunct Professor,

Session 5: Assessing Multiple Studies

Peter Tarczy-Hornoch MDHead and Professor, Division of BHIProfessor, Division of Neonatology

Adjunct Professor, Computer Science and Engineering

faculty.washington.edu/pth

November 3, 2008

Assessing Multiple Studies

Context for Assessing Multiple Studies Systematic Review Statistics PPICONSS Revisited Information Overload Revisited Makeups “Test” Exam

Steps to Finding & Assessing Information

1. Translate your clinical situation into a formal framework for a searchable question (Session 1)

2. Choose source(s) to search (Session 2)

3. Search your source(s) (Session 2)

4. Assess the resulting articles (documents) Therapy documents (Session 3) Diagnosis documents (Session 4) Systematic reviews/comparing documents (Session 5)

5. Decide if you have enough information to make a decision, repeat 1-4 as needed (ICM, clinical rotations, internship, residency) (Session 6)

Multiple Studies/Systematic Reviews For a given topic there are multiple articles

Clinical Query: “diagnosis” “appendicitis” => 453 Clinical Query: “therapy” “breast cancer” =>5154

To compare a small number of studies use PPICONSS approach and statistics from MIDM

Systematic reviews do the work for you and statistically merge/summarize a collection of related articles if there are enough articles on a given topic AND if there is someone who did the work (recently)



Systematic Review: Forest Plot Likely no statistically significant difference

Likely statistically significant differenceRed line is “no difference”

Red line is “no difference”

Diamond is pooled/aggregate result & CI

Copyright ©2000 BMJ Publishing Group Ltd.

Sutton, A J et al. BMJ 2000;320:1574-1577

• Plot of effect size vs. precision(or sample size or standard error)

• No publication bias => funnel • Smaller trials more numerous• Smaller trials more variable (random variation)

• Publication bias => no funnel • Smaller studies less often published• Non-significant studies less often published

• Figures show funnel plot generated from 35 simulated studies (top) and same data with five missing studies showing a typical manifestation of publication bias (bottom)

Systematic Review Funnel Plot (Looking for Publication Bias)

Treatment/Diagnosis Statistics Systematic reviews generally use similar statistics

for treatment and diagnosis as in Session 4 & 5 One new treatment statistic is the Odds Ratio (OR)

OR = Ratio of odds of having outcome in treatment group to odds of having outcome in control group

Null value (no difference between groups) is 1.0 Axis label shows which side favors treatment vs control

Contrast to likelihood ratio treatment statistic LR = Likelihood test result occurs in patient with

disease compared to likelihood of same result in patient without disease

Example: Search “Community acquired pneumonia treatment” Healthlinks => Cochrane => reiew

Antibiotics for community acquired pneumonia in adult outpatients. Lise M Bjerre, Theo JM Verheij, Michael M Kochen Year: 2004

Short-course versus long-course antibiotic therapy for non-severe community-acquired pneumonia in children aged 2 months to 59 months. Batool A Haider, Muhammad Ammad Saeed, Zulfiqar A Bhutta Year: 2008

Example: Results Objectives: To summarize the evidence currently available from randomized

controlled trials (RCTs) concerning the efficacy of alternative antibiotic treatments for CAP in ambulatory patients above 12 years of age.

Search strategy: We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library, 2003, issue 2) which contains the Cochrane Acute Respiratory Infections Groups Specialized Register; MEDLINE (January 1966 to September week 3, 2003), and EMBASE (January 1974 to March 2003).

Selection criteria: We included all randomized controlled trials (RCTs) in which one or more antibiotics were tested for the treatment of CAP in ambulatory adolescent or adult patients. Studies testing one or more antibiotic and reporting the diagnostic criteria used in selecting patients as well as the clinical outcomes achieved were included. No language restrictions were applied.

Data collection and analysis: Data were extracted and study reports assessed by two independent reviewers (LMB and TJMV). Authors of studies were contacted as needed to resolve any ambiguities in the study reports. The data were analyzed using the Cochrane Collaboration's RevMan 4.2.2 Software. Differences between reviewers were resolved by discussion and consensus.

Example: Results Main results: Three randomized controlled trials involving a total of 622 patients

aged 12 years and older diagnosed with community acquired pneumonia were included. The quality of the studies and of the reporting was variable. A variety of clinical, radiological and bacteriological diagnostic criteria and outcomes were reported. Overall there was no significant difference in the efficacy of the various antibiotics under study.

Authors' conclusions: Currently available evidence from RCTs is insufficient to make evidence-based recommendations for the choice of antibiotic to be used for the treatment of community acquired pneumonia in ambulatory patients. Pooling of study data was limited by the very low number of studies. Individual study results do not reveal significant differences in efficacy between various antibiotics and antibiotic groups. Multi-drug comparisons using similar administration schedules are needed to provide the evidence necessary for practice recommendations.



Refresher: Finding answer(s) (PPICOS) Translate clinical question to searchable question (PPICOS)

P: Problem What is the question of interest? E.g. “How to diagnose Alzheimers?”

P: Patient Demographics (e.g. gender/age range), condition, disease E.g. “Elderly male with report by informant of memory loss”

I: Intervention Diagnosis/treatment, which one is of primary interest/preferred a priori E.g. “Diagnosis using a genetic marker”

C: Comparison Alternative diagnosis(es)/treatment(s) (of secondary interest) E.g. “Diagnosis using different kinds of neuropsychiatric testing ”

O: Outcome Diagnostic accuracy, complication, death, cost, etc. E.g. “Highest LR+

S: Study Design/Type What type of study/document are you ideally looking for E.g. “Systematic Review”

Refresher: Assessing one documentSearch Result Comparison

Problem at hand Problem studied Are they really the same?

Patient characteristics Population characteristics Is patient similar enough to population studied?

Intervention most relevant to patient/provider

Intervention studied (primary one)

Are they the same?

Comparison – other alternatives considered

Comparison – alternatives studied

Are alternatives studied those of interest to you?

Outcomes – those important to pat/prov

Outcomes – those looked at by study

Are outcomes studied those of interest to you?

Number of subjects Does study have enough subjects to trust results?

Study design hoped for Statistics – study design and statistical results

Is study design good? What do results mean?

Sponsor – who paid for study

Is there potential bias?

Comparing Two Documents: PPICONSSSearch Result #1 Result #2 Comparison

Problem at hand Problem studied Problem studied Which result is closest to problem at hand?

Patient characteristics

Population characteristics Population characteristics

Which result population most similar my patient?

Intervention most relevant to patient/provider



Which result intervention is most similar to mine?

Comparison – other alternatives considered



Which result has alternatives most like what I’m considering?

Outcomes – those important to pat/prov



Which result has more important outcomes?

Number of subjects Number of subjects Which result has more subjects?

Study design hoped for

Statistics – study design and statistical results

Statistics – study design and statistical results

Is one results study design better than the others? Statistics better?



Does either study has sponsorship bias?

Study types: from weaker to stronger Case Series: 1 series of patients with outcome of interest, no control group, +/-

protocol, weakest design (“anecdote”) E.g. last 10 toddlers with asthma and their histories

Cohort Trial: 2 groups (cohorts), protocol is “exposure of interest” vs. not, follow FORWARD for outcome of interest, can’t control for UNKNOWN factors

E.g. 10 infants born to mothers who smoked vs. 10 infants born to mothers who did not and seeing (looking FORWARD) whether or not they develop asthma

Case Control Trial: 2 groups, protocol is outcome of interest (“cases”) vs. those without (“controls”), look BACK for “exposure of interest”, can only match cases/controls for KNOWN factors (e.g. age/gender)

E.g. 10 toddlers with asthma and 10 similar toddlers without asthma looking BACK to see if their mothers smoked

Randomized Controlled Trial Protocol is treatment (intervention) randomly assigned, avoids bias by ensuring

KNOWN and UNKNOWN factors (confounders) that determine outcome are evenly distributed between treatment and control groups (or treatment 1 vs treatment 2), often called “gold standard” study design. Double blinding is when neither investigator nor study subjects know who got what intervention.

E.g. 100 toddlers with asthma and randomly giving 50 one treatment and 50 another treatment and seeing if one group does better/worse

Systematic Review/Meta-analysis Summary of medical literature using explicit protocol to search the literature

and critically appraise and combine studies E.g. 20 studies looking at asthma treatments in toddlers

It’s not just statistics & evidence Patient factors

Personal values (e.g. testing for Huntington Disease) Beliefs & experiences related to diagnosis/treatment (e.g. models

of disease mechanisms across cultures & forms of healthcare) Cost & convenience

Provider factors Personal values (e.g. abortion for fatal fetal malformation) Knowledge base & experience (e.g. intern vs. expert, generalist

vs. specialist) Local policies & procedures

External factors Insurance coverage Availability of therapy/diagnosis (e.g. formularies) Marketing (to patients and providers)



McMaster PLUS Project A tiny proportion of research is “ready for application”

60,000 articles/year in 120 journals monitored => info overload! ~3,500 articles/year meet critical appraisal and content criteria

(94% noise reduction - NR) Only a tiny fraction of the “ready for application” research

is “relevant” to the practice of a given practice/physician 4500 MDs in 56 practice categories, rate 3,500 for relevance

Only a tiny proportion of the “relevant” research for a given practitioner is “interesting” in the sense of being something new, important, and actionable 4,500 MDs rate each article for interest/importance/impact ~10-35 articles/yr for one practice/clinician type (99.95% NR) ~0-50 articles/yr for authors of EBM textbooks Information overload?

Adapted from slides from Brian Haynes, AMIA 2007

BMJ Updates Alerting

Can get alerts from one or more of 56 clinical areas e-mailed to you

Can choose relevance, importance, etc.

Adapted from slides from Brian Haynes, AMIA 2007

http://bmjupdates.mcmaster.ca/

Pragmatism Can’t read all the medical literature nor read all the

results of a search for a common problem Even for “narrow, specific search” Clinical Queries get a lot

of information and it keeps growing“appendicitis” => 453 (vs. 426 last year)“breast cancer” =>5154 (vs. 4662 last year)

What do you do? Use the top of the pyramid Use alerting services Use skills from MIDM class to screen abstracts and articles

(may not be as good as McMasters PLUS project but can help you focus your reading of full articles on the best of the literature)

Small Group Wednesday Nov 5th Small group leads to give examples of recent

clinical situations where they had to evaluate one or more documents and/or a systematic review related to making a diagnosis or deciding on a treatment

Group to review and discuss assignment with multiple short examples which will focus on:

Interpretation of a forest plot in a systematic review OR, NNT, and RR in the context of a systematic review Comparing two documents related to the same diagnosis or

therapy

Any other MIDM related questions from students welcomed



Makeups For any small group sessions you missed a makeup assignment is

due by e-mail to me by 5P on Mon 11/10 (if you have not already done so)

E-mail assignment to [email protected] Makeup for miss of Small Group on 10/22, 10/27, 10/29, 11/3, 11/6

Review missed Slides http://courses.washington.edu/midm/schedule.htm Download missed Assignment to Prepare for Small Group Answer the question(s) for each of the examples E-mail me your answers

Makeup for miss of Small Group 10/21 Review 10/21 Slides http://courses.washington.edu/midm/schedule.htm E-mail me the following What you had hoped to get from small group sessions Two clinical situations/scenarios to which you could see applying what you

learned in MIDM

mailto:[email protected]

http://courses.washington.edu/midm/schedule.htm

http://courses.washington.edu/midm/schedule.htm



Results of taking the “test” exam See e-mail “[MIDM] Important - testing your

Questionmark login id…” for more details First: get your login id/password from MyGrade Second: test your login id/password and your

ability to save and retrieve you exam: https://primula.dme.washington.edu/q4/perception.dll

2 question “Test” exam up until 5P Monday 11/3 So far 12 students of 104 (~12%) needed help Remember to test and if you have problems to e-

mail Deven Hamilton [email protected]

https://primula.dme.washington.edu/q4/perception.dll

mailto:[email protected]

QUESTIONS? Context for Assessing Multiple Studies Systematic Review Statistics PPICONSS Revisited Information Overload Revisited Makeups “Test” Exam Small Group