Systematic reviews meta-analysis and critical reading of

medical literatureEvidence-based medicine

Phyllis W Speiser MDChief Div Ped Endo CCMC

Professor of PediatricsHofstra-NSLIJ School of Medicine

The need to be evidence-based

1 Wide variations in practice1 Continued use of ineffective treatments 2 Excess use of inappropriate treatments 3 Poor uptake of effective practice

2 Increasing consumerism 1 Unvetted Internet information2 Direct-to-consumer pharma advertising

3 Increasing demand on $ resources1 Need to demonstrate efficacy

4 Exponential growth in research1 Need to compare amp evaluate many

studiesrsquo variable quality amp conflicting results

Information overload

bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)

bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations

Types of clinical trials

bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults

bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects

bull Phase IIIndash Large multicenter efficacy amp longer-term safety

bull Phase IVndash Post-marketingndash Extended safety profile

- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated

Pitfalls of small clinical trials

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

The need to be evidence-based

1 Wide variations in practice1 Continued use of ineffective treatments 2 Excess use of inappropriate treatments 3 Poor uptake of effective practice

2 Increasing consumerism 1 Unvetted Internet information2 Direct-to-consumer pharma advertising

3 Increasing demand on $ resources1 Need to demonstrate efficacy

4 Exponential growth in research1 Need to compare amp evaluate many

studiesrsquo variable quality amp conflicting results

Information overload

bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)

bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations

Types of clinical trials

bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults

bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects

bull Phase IIIndash Large multicenter efficacy amp longer-term safety

bull Phase IVndash Post-marketingndash Extended safety profile

- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated

Pitfalls of small clinical trials

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Information overload

bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)

bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations

Types of clinical trials

bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults

bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects

bull Phase IIIndash Large multicenter efficacy amp longer-term safety

bull Phase IVndash Post-marketingndash Extended safety profile

- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated

Pitfalls of small clinical trials

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Types of clinical trials

bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults

bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects

bull Phase IIIndash Large multicenter efficacy amp longer-term safety

bull Phase IVndash Post-marketingndash Extended safety profile

- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated

Pitfalls of small clinical trials

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated

Pitfalls of small clinical trials

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Why bother with evidence-based medicine (EBM)

bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype

bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions

bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Types of EBM studies

bull Diagnosisndash Selecting appropriate diagnostic tests

bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit

bull Prognosisndash Outcomes amp complications

bull Associations Causesndash Identify etiologies eg infectious

environmental iatrogenic

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

What makes a review ldquosystematicrdquo

bull Basis for EBM recommendations

bull Based on a clearly formulated question

bull Identifies relevant studies with pre-set criteria

bull Appraises quality of studies

bull Summarizes evidence by use of explicit methodology

bull Recommendations are based on evidence gathered

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Assessing quality

bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias

bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias

bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical

phenotypendash Low inter-assay and intra-assay variability

bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis

bull Appropriate statistical methods used

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Quality validity of studies Design

bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control

ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo

Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Quality validity of studies Intervention

ndash Nature of interventionbull Placebo-controlled v best current treatment v

uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose

ndash Verification of methodsbull Same or different assays inter- amp intra- assay

variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Quality validity of studies Data

ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls

patient diaries pill counts etc)

ndash Statistical analysisbull Appropriateness of statistical methods

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

What is a meta-analysis

Optional part of a systematic review

Systematic reviews

Meta-analyses

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Meta-analysis Are the studies consistent

bull Are variations in results between studies consistent with chance

bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Hierarchy of evidence for treatment decisions

Meta-analysis of RCTsSystematic review of RCTs

Individual RCT

Observational studiespatient-important outcomes

Basic researchtest tube animal human physiology

Clinical experience

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Pitfalls of meta-analysis

bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search

bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition

bull Length of follow-up

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Pitfalls of meta-analysis cont

bull Methods of meta-analysisndash Sensitivity analysis for robustness

bull Fixed vs random effectsbull Outlier exclusions

ndash Stratification of subject populations

bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot

ldquoHeterogeneityrdquo

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

GRADE system Knowledge translation

bull Transparent process of moving from evidence to recommendations

bull Developed by representative group of international guideline developers

bull Separates quality of evidence amp strength of recommendations

bull Stresses importance of outcomes of alternative management strategies

bull Explicit acknowledgment of values and preferences

bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Grading evidence

bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect

bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

bull Very low qualitymdash Any estimate of effect is very uncertain

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Strength of Recommendations

Factor High Rank Low Rank

Quality evidence

RCT Case series

Balance of risk amp benefit

Low toxicity amp High efficacy

High toxicity amp

High efficacy

Values amp preferences

Life-saving or QOL-enhancing

No major advance

Cost Inexpensive Costly

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Evidence-based clinical decisions Are antibiotics indicated in pediatric

otitis media

bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media

bull Should antibiotics be prescribedndash Benefitsndash Risks

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Glasziou Cochrane systematic review 2003

bull Systematic review of RCTs

bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp

combine comparable data

bull Comprehensive search

bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Odds ratios allow comparisons of different studies in meta-analysis

TestTx + TestTx -

Disease A TRUE POS B FALSE NEG

Unaffected C FALSE POS D TRUE NEG

Odds ratio for treatment efficacy =

AD BC

or TP x TN FN x FP

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Confidence intervals Definition

Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times

For a 95 confidence interval if the experiment were

repeated many times 95 of the intervals would contain the true treatment effect

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important

01 1 10

Burke

van Buchem (a)

Pooled Estimate

Favours Antibiotics Favours Placebo

Appelman

Damoiseaux

Halsted

Kaleida

van Buchem (b)

Mygind

Thalin

Odds Ratio (95 CI)

N = 121

N = 225

N = 240

N = 89

N = 980

N = 149

N = 316

N = 84

N = 83

N = 2287

086 (034 222)

065 (034 122)

055 (032 094)

108 (039 297)

050 (029 085)

045 (022 090)

057 (029 110)

043 (014 127)

057 (021 156)

057 (045 073)

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

lt00001001003009029p-value

027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI

0404040404Proportion responding in control arm

408642Responders in control arm

1002015105Number in control arm

0808080808Proportion responding in treatment arm

80161284Responders in treatment arm

1002015105Number in treatment arm

Effect of population size Confidence interval and significance improve with larger or pooled samples

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Moral

Any observed difference between two groups no matter how small can be made ldquostatistically

significantrdquo - at any level of significance - by taking a sufficiently large sample

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Balanced decisionsbull Are the differences are both clinically amp statistically significant

bull Is the benefit greater than the risk

bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics

bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15

bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis

Razvi S et al J Clin Endocrinol Metab 2008932998-3007

Forest plot of IHD prevalence in SCH and euthyroid controls Age differences

lt65 yo

gt65 yo

Evidence-based clinical decisions Subclinical hypothyroidism

bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism

EBM Beta blockers post MI- needed to

achieve desired outcome

bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8

bull A meta-analysis of RCTs of beta-blockers after MI

suggests a 25 risk reduction

bull Must treat 50 such pts to prolong a life

bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted

21

5

101

1 2

8

7

8

12

4

3

1

1

2

8

7

2

1

1

1

2

8

1

5

15

6

No

t M

en

tio

ne

d

Ro

uti

ne

Ex

pe

rim

en

tal

Ra

reN

ev

er

Sp

ec

ific

M

M

M

M

M

M

TextbookReviewRecommendations

Odds Ratio (Log Scale)

05 10 20

Favors Treatment Favors Control

RCTs Pts

1 23

2 65

3 149

4 316

7 1793

10 254411 265115 331117 392922 5452

Plt01

23 5767

27 612530 634633 657143 21 05954 22 051

67 47 53165 47 185

70 48 154

Plt001

Plt00001

CumulativeYear

1960

1965

1970

1980

1985

1990

Thrombolytic therapy in MI Power in NImportance of current textbooks

Growth of Cochrane Reviews and Protocols

1995 2003

reviews

protocols

2500 completed mid-2005

2010 gt4000

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Evidence-based clinical decisions Subclinical hypothyroidism

bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism

EBM Beta blockers post MI- needed to

achieve desired outcome

bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8

bull A meta-analysis of RCTs of beta-blockers after MI

suggests a 25 risk reduction

bull Must treat 50 such pts to prolong a life

bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted

21

5

101

1 2

8

7

8

12

4

3

1

1

2

8

7

2

1

1

1

2

8

1

5

15

6

No

t M

en

tio

ne

d

Ro

uti

ne

Ex

pe

rim

en

tal

Ra

reN

ev

er

Sp

ec

ific

M

M

M

M

M

M

TextbookReviewRecommendations

Odds Ratio (Log Scale)

05 10 20

Favors Treatment Favors Control

RCTs Pts

1 23

2 65

3 149

4 316

7 1793

10 254411 265115 331117 392922 5452

Plt01

23 5767

27 612530 634633 657143 21 05954 22 051

67 47 53165 47 185

70 48 154

Plt001

Plt00001

CumulativeYear

1960

1965

1970

1980

1985

1990

Thrombolytic therapy in MI Power in NImportance of current textbooks

Growth of Cochrane Reviews and Protocols

1995 2003

reviews

protocols

2500 completed mid-2005

2010 gt4000

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

EBM Beta blockers post MI- needed to

achieve desired outcome

bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8

bull A meta-analysis of RCTs of beta-blockers after MI

suggests a 25 risk reduction

bull Must treat 50 such pts to prolong a life

bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted

21

5

101

1 2

8

7

8

12

4

3

1

1

2

8

7

2

1

1

1

2

8

1

5

15

6

No

t M

en

tio

ne

d

Ro

uti

ne

Ex

pe

rim

en

tal

Ra

reN

ev

er

Sp

ec

ific

M

M

M

M

M

M

TextbookReviewRecommendations

Odds Ratio (Log Scale)

05 10 20

Favors Treatment Favors Control

RCTs Pts

1 23

2 65

3 149

4 316

7 1793

10 254411 265115 331117 392922 5452

Plt01

23 5767

27 612530 634633 657143 21 05954 22 051

67 47 53165 47 185

70 48 154

Plt001

Plt00001

CumulativeYear

1960

1965

1970

1980

1985

1990

Thrombolytic therapy in MI Power in NImportance of current textbooks

Growth of Cochrane Reviews and Protocols

1995 2003

reviews

protocols

2500 completed mid-2005

2010 gt4000

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

21

5

101

1 2

8

7

8

12

4

3

1

1

2

8

7

2

1

1

1

2

8

1

5

15

6

No

t M

en

tio

ne

d

Ro

uti

ne

Ex

pe

rim

en

tal

Ra

reN

ev

er

Sp

ec

ific

M

M

M

M

M

M

TextbookReviewRecommendations

Odds Ratio (Log Scale)

05 10 20

Favors Treatment Favors Control

RCTs Pts

1 23

2 65

3 149

4 316

7 1793

10 254411 265115 331117 392922 5452

Plt01

23 5767

27 612530 634633 657143 21 05954 22 051

67 47 53165 47 185

70 48 154

Plt001

Plt00001

CumulativeYear

1960

1965

1970

1980

1985

1990

Thrombolytic therapy in MI Power in NImportance of current textbooks

Growth of Cochrane Reviews and Protocols

1995 2003

reviews

protocols

2500 completed mid-2005

2010 gt4000

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Growth of Cochrane Reviews and Protocols

1995 2003

reviews

protocols

2500 completed mid-2005

2010 gt4000

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Real world clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Knowledge Translation Researchhellip

hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Knowledge Translation Research

KT Type 1 KT Type 2

Based on Hulley et al Designing Clinical Research 2007 p 23

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Barrier Solutions

bull too little research addressing ldquoreal worldrdquo problems

bull large simple randomized trials

bull ldquohead to headrdquo comparisons

Step 1 Generating Research Evidence

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Possible comparators

placebo usual carecurrent best care

Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

gt57000 EBM articles 2009

Systematic reviews

Original journal articles

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

By the year 2020 90 of clinical decisions will be supported by accurate timely and

up-to-date clinical information and will reflect the best available evidence

IOM Roundtable on Evidence-Based Medicine

This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if

EBM+KT=090

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Clinical problem

bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Whatrsquos the question

bull Is Tamiflu more effective than fluids rest and anti-pyretics

bull Does Tamiflu reduce flu severity

bull How much does Tamiflu cost

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

How to find answers

bull Search Tamiflu AND efficacy

bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]

bull Search Tamiflu AND severity of illness

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

ResultsSAE + SAE- Totals

Rx 17 228 245

Pbo 19 110 129

Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78

Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53

Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication

JAMA 2000283(8)1016-1024

Informed decision

bull MD discusses data with patientndash Benefitsndash Risksndash Cost

Other resources

1 Victor Montori MD

Mayo Clinic KER unit

2 McMaster Inst Institute of Medicine

3 Online tutorial

httpwwwhslunceduServicesTutorialsEBMwelcomehtm

4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439

Other resources

1 Victor Montori MD

Mayo Clinic KER unit

2 McMaster Inst Institute of Medicine

3 Online tutorial

httpwwwhslunceduServicesTutorialsEBMwelcomehtm

4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439