Upload
grupograde
View
998
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
Grading the quality of evidence Grading the quality of evidence and the strength of and the strength of recommendationsrecommendations
San SebastiánSan Sebastián12 - 14 May 200812 - 14 May 2008
Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann
Workshop agendaWorkshop agendaMonday, 12 May 2008Monday, 12 May 2008
08.15 – 13.30 h - Plenary08.15 – 13.30 h - Plenary Introduction to GRADEIntroduction to GRADE Grading the quality of evidence for each Grading the quality of evidence for each
important outcomeimportant outcome– Deciding which outcomes are importantDeciding which outcomes are important– Factors that can lower the quality of evidenceFactors that can lower the quality of evidence– Factors that can raise the quality of evidenceFactors that can raise the quality of evidence
Judgements about the overall quality of Judgements about the overall quality of evidenceevidence
15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion
IntroductionIntroduction
AndyAndy
Why bother about grading?Why bother about grading?
People draw conclusions about thePeople draw conclusions about the– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations
Systematic and explicit approaches can helpSystematic and explicit approaches can help– protect against errorsprotect against errors– resolve disagreementsresolve disagreements– facilitate critical appraisalfacilitate critical appraisal– communicate informationcommunicate information
However, there is wide variation in currently However, there is wide variation in currently used approachesused approaches
Many grading systemsMany grading systems
EvidenceEvidenceRecommendationRecommendation
BB Class IClass I C+ C+ 11 IVIV CC
OrganizationOrganization AHAAHA ACCPACCP SIGNSIGN
Recommendation for use of oral anticoagulation in Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral patients with atrial fibrillation and rheumatic mitral valve diseasevalve disease
Australian NMRCAustralian NMRC Oxford Center for Evidence-based MedicineOxford Center for Evidence-based Medicine Scottish Intercollegiate Guidelines (SIGN)Scottish Intercollegiate Guidelines (SIGN) US Preventative Services Task ForceUS Preventative Services Task Force Professional organizationsProfessional organizations
– AHA/ACC, ACCP, AAP, Endocrine society, etc....AHA/ACC, ACCP, AAP, Endocrine society, etc....
Lots of confusion and dismayLots of confusion and dismay
A common international grading system?A common international grading system?
International groupInternational group– ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC, ACCP, AHRQ, Australian NMRC, BMJ Clinical Evidence, CC,
CDC, CC, NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, CDC, CC, NICE, Oxford CEBM, SIGN, UpToDate, USPSTF, WHOWHO
> 60 contributors> 60 contributors– methodologists, guideline developers, systematic methodologists, guideline developers, systematic
reviewers, researchers, clinicians, editorsreviewers, researchers, clinicians, editors ~ 20 meetings over last seven years~ 20 meetings over last seven years
– ~10 – 40 participants~10 – 40 participants
www.gradeworkinggroup.org
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research
and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and
Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research
and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and
Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Agency for Health Care
Research and Quality (AHRQ)Research and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs Canadian Agency for Drugs
and Technology in Healthand Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre Norwegian Knowledge Centre for the Health Servicesfor the Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research
and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and
Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research
and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and
Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Developing Journal of Infection in Developing Countries - InternationalCountries - International
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care Society of Critical Care
MedicineMedicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
American College of PhysiciansAmerican College of Physicians Agencia sanitaria regionale, Agencia sanitaria regionale,
Bologna, ItaliaBologna, Italia Agency for Health Care Research Agency for Health Care Research
and Quality (AHRQ)and Quality (AHRQ) Allergic Rhinitis in Asthma Allergic Rhinitis in Asthma
GuidelinesGuidelines American College of Chest American College of Chest
PhysiciansPhysicians American Endocrine SocietyAmerican Endocrine Society American Thoracic SocietyAmerican Thoracic Society BMJ Clinical EvidenceBMJ Clinical Evidence British Medical JournalBritish Medical Journal Canadian Agency for Drugs and Canadian Agency for Drugs and
Technology in Health Technology in Health Cochrane CollaborationCochrane Collaboration EBM Guidelines FinlandEBM Guidelines Finland European Respiratory SocietyEuropean Respiratory Society European Society of Thoracic European Society of Thoracic
SurgeonsSurgeons Evidence-based Nursing Sudtirol, Evidence-based Nursing Sudtirol,
Alta Adiga, ItalyAlta Adiga, Italy
Journal of Infection in Journal of Infection in Developing Countries - Developing Countries - InternationalInternational
Kidney disease: Improving global Kidney disease: Improving global outcomeoutcome
National Institute for Clinical National Institute for Clinical Excellence (NICE)Excellence (NICE)
Norwegian Knowledge Centre for Norwegian Knowledge Centre for the Health Servicesthe Health Services
Ontario MOH Medical Advisory Ontario MOH Medical Advisory SecretariatSecretariat
Polish Institute for EBMPolish Institute for EBM Society of Critical Care MedicineSociety of Critical Care Medicine Society of Pediatric EndocrinologySociety of Pediatric Endocrinology Society of Vascular SurgerySociety of Vascular Surgery Surviving sepsis campaignSurviving sepsis campaign The German Agency for Quality in The German Agency for Quality in
Medicine Medicine University of Pennsylvania Health University of Pennsylvania Health
System Center for EB Practice System Center for EB Practice UpToDateUpToDate World Health OrganizationWorld Health Organization
GRADE UptakeGRADE Uptake
What is quality of What is quality of evidence andevidence and
how should it be how should it be graded?graded?
Judging the quality of evidence Judging the quality of evidence requires considering the requires considering the
contextcontextIn the context of making In the context of making
recommendationsrecommendations The quality of evidence reflects the The quality of evidence reflects the
extent to which our confidence in an extent to which our confidence in an estimate of the effect is adequate to estimate of the effect is adequate to support a particular recommendation. support a particular recommendation.
In the context of a systematic reviewIn the context of a systematic review The quality of evidence reflects the The quality of evidence reflects the
extent to which we are confident that extent to which we are confident that an estimate of effect is correct. an estimate of effect is correct.
Although the degree of confidence is Although the degree of confidence is a continuum, we suggest using four a continuum, we suggest using four
categoriescategories
HighHighModerateModerateLowLowVery lowVery low
Categories of qualityCategories of quality
HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.
ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.
LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.
Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.
Study design is importantStudy design is important
Early systems of grading the quality of Early systems of grading the quality of evidence focused almost exclusively evidence focused almost exclusively on study designon study design
Randomised trials provide, in general, Randomised trials provide, in general, far stronger evidence than far stronger evidence than observational studies.observational studies. – Randomised trials start out at HighRandomised trials start out at High– Observational studies start out at Low Observational studies start out at Low
However, other factors may decrease However, other factors may decrease or increase the quality of evidenceor increase the quality of evidence
What can lower the What can lower the quality of evidence?quality of evidence?
Study limitationsStudy limitations Inconsistency of results Inconsistency of results Indirectness of evidence Indirectness of evidence Imprecise results Imprecise results Publication bias Publication bias
What can raise the quality What can raise the quality of evidence?of evidence?
Large magnitude of effect Large magnitude of effect A dose response relationshipA dose response relationshipAll plausible confoundingAll plausible confounding
Quality assessment criteriaQuality assessment criteria
Strength of recommendationStrength of recommendationThe degree of confidence that the The degree of confidence that the
desirable effects of adherence to a desirable effects of adherence to a recommendation outweigh the recommendation outweigh the undesirable effects. undesirable effects.
Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings
Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts
Categories of recommendationsCategories of recommendations
Although the degree of confidence is a Although the degree of confidence is a continuum, we suggest using two continuum, we suggest using two categories: strong and weak.categories: strong and weak.
Strong recommendation:Strong recommendation: the panel the panel is confident that the desirable effects is confident that the desirable effects of adherence to a recommendation of adherence to a recommendation outweigh the undesirable effects.outweigh the undesirable effects.
Weak recommendation:Weak recommendation: the panel the panel concludes that the desirable effects concludes that the desirable effects of adherence to a recommendation of adherence to a recommendation probably outweigh the undesirable probably outweigh the undesirable effects, but is not confident. effects, but is not confident.
Recommend
Suggest
Determinants of strength of recommendationDeterminants of strength of recommendation
Factors Impact on the strength of a recommendation
Balance between desirable and undesirable effects
Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted.
Quality of the evidence
Higher the quality of evidence, more likely a strong recommendation warranted.
Values and preferences
More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted.
Costs (resource use)
Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
Workshop agendaWorkshop agendaMonday, 12 May 2008Monday, 12 May 2008
Quality of evidenceQuality of evidence
08.15 – 13.30 h - Plenary08.15 – 13.30 h - Plenary Introduction to GRADEIntroduction to GRADE Grading the quality of evidence for each Grading the quality of evidence for each
important outcomeimportant outcome– Deciding which outcomes are importantDeciding which outcomes are important– Factors that can lower the quality of evidenceFactors that can lower the quality of evidence– Factors that can raise the quality of evidenceFactors that can raise the quality of evidence
Judgements about the overall quality of Judgements about the overall quality of evidenceevidence
15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion
Workshop agendaWorkshop agendaTuesday, 13 May 2008Tuesday, 13 May 2008
Strength of recommendationStrength of recommendation
08.30 – 13.30 h - Plenary08.30 – 13.30 h - Plenary Going from evidence to recommendationsGoing from evidence to recommendations
– Summarising and presenting findingsSummarising and presenting findings– Factors that determine the strength of a recommendationFactors that determine the strength of a recommendation
Balance of benefits and harmsBalance of benefits and harms (Quality of evidence)(Quality of evidence) Values and preferencesValues and preferences Costs (resource use)Costs (resource use)
– Judgements about the strength of recommendationsJudgements about the strength of recommendations– Wording of recommendationsWording of recommendations
15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups Summary of findings and the strength of Summary of findings and the strength of
recommendation for HPV vaccinationrecommendation for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion
Workshop agendaWorkshop agendaWednesday, 14 May 2008Wednesday, 14 May 2008
Special challengesSpecial challenges
08.30 – 11.30 h – Plenary + small group 08.30 – 11.30 h – Plenary + small group workwork
Economic evidence (resource use)Economic evidence (resource use)
13.00 – 17.00 h - Plenary + small group 13.00 – 17.00 h - Plenary + small group workwork
Diagnostic testsDiagnostic tests
General questions about GRADE, the General questions about GRADE, the GRADE Working Group or the GRADE Working Group or the
agenda for this workshop?agenda for this workshop?
Relative importance of Relative importance of outcomesoutcomes
HolgerHolger
The quality of evidence needs to be The quality of evidence needs to be considered for each important considered for each important
outcomeoutcome
The quality of evidence may be different for The quality of evidence may be different for different outcomesdifferent outcomes
Decision makers (and review authors) need to Decision makers (and review authors) need to consider the relative importance of outcomesconsider the relative importance of outcomes
Flatulence 2
Hierarchy of outcomes according to their Hierarchy of outcomes according to their importance to assess the effect of phosphate importance to assess the effect of phosphate lowering drugs in patients with renal failure and lowering drugs in patients with renal failure and hyperphosphatemiahyperphosphatemia
Importance of endpoints
Critical for decision making
Important, but not critical for decision making
Of lowimportance
5
Pain due to soft tissue 6 calcification / function
Fractures 7
Myocardial infarction 8
Mortality 9
3
4
1
A different clinical A different clinical questionquestion
Population: In patients with chronic atrial fibrillation and no prior history
of stroke
Intervention: does oral anticoagulation (comparison) compared with no therapy
Outcome: reduce the risk for embolic stroke, hemorrhage
and death?
Population: In patients with chronic atrial fibrillation and no prior history
of stroke
Intervention: does oral anticoagulation (comparison) compared with no therapy
Outcome: reduce the risk for embolic stroke, hemorrhage
and death?Other outcomes:
Inconvenience, quality of life
All disabling or fatal stroke (isch. and hemorrh.)
Major hemorrhage (non IC) All cause mortality Health related quality of life
*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,
Issue 3.
Outcomes/endpointsOutcomes/endpoints
How important is the endpoint for decision making?Judgment about the relative importance for each endpoint on a scale from 9 (most important) to 1 (least important): • 7 – 9: the endpoint is critical for decision making.• 4 – 6: the endpoint is important but not critical for decision making.• 1 – 3: the endpoint is not important.
Outcomes/endpointsOutcomes/endpoints
2
Importance of endpoints
Critical for decision making
Important, but not critical for decision making
Of lowimportance
5
6
7
8
9
3
4
1
2
Importance of endpoints
Critical for decision making
Important, but not critical for decision making
Of lowimportance
5
6
7
8
9
3
4
1
All disabling or fatal stroke (isch. and hemorrh.)
Major hemorrhage (non IC) All cause mortality Health related quality of life
*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,
Issue 3.
Outcomes/endpointsOutcomes/endpoints
9
8
9
7
2
Importance of endpoints
Critical for decision making
Important, but not critical for decision making
Of lowimportance
5
6
7
8
9
3
4
1
Questions about the relative Questions about the relative importance of outcomes?importance of outcomes?
Study limitationsStudy limitations
VictorVictor
Study limitations (Risk of bias)Study limitations (Risk of bias)for Randomized trialsfor Randomized trials
Lack of allocation concealmentLack of allocation concealment Lack of blindingLack of blinding Incomplete accounting of patients and Incomplete accounting of patients and
outcome eventsoutcome events Selective outcome reportingSelective outcome reporting Other limitationsOther limitations
– e.g. stopping early for benefit; use of e.g. stopping early for benefit; use of unvalidated patient-reported outcomes; carry-unvalidated patient-reported outcomes; carry-over effects in cross-over trials; recruitment bias over effects in cross-over trials; recruitment bias in cluster-randomized trialsin cluster-randomized trials
Testosterone for BMD in menTestosterone for BMD in men
Assessment of the risk of biasAssessment of the risk of bias
Cochrane Handbook for Systematic Reviews of Interventions, Draft chapter of version 5 - Assessing risk of bias in included studies http://www.cochrane.org/resources/handbook/handbook5drafts.htm
Summarizing study limitations for randomized trialsSummarizing study limitations for randomized trials
Questions about study Questions about study limitations?limitations?
Inconsistency of resultsInconsistency of results
HolgerHolger
Inconsistency of resultsInconsistency of results
Look for explanationLook for explanation– patients, intervention, comparator, patients, intervention, comparator,
outcome, methodsoutcome, methods
JudgmentJudgment– variation in size of effectvariation in size of effect– overlap in confidence intervalsoverlap in confidence intervals– statistical significance of heterogeneitystatistical significance of heterogeneity– II22
Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.
Heparin or vitamin K antagonists for Heparin or vitamin K antagonists for survival in patients with cancer survival in patients with cancer
Non-steroidal drug use and risk of Non-steroidal drug use and risk of pancreatic cancerpancreatic cancer
Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure
categories.Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.
Example: Thromboprophylaxis Example: Thromboprophylaxis and bleeding in the hospitaland bleeding in the hospital
Dentali et al. Ann Int Med, 2007
Questions about inconsistency Questions about inconsistency of results?of results?
Indirectness of evidenceIndirectness of evidence
VictorVictor
Indirectness of evidenceIndirectness of evidence
Question of Question of interestinterest
Relative Relative effectiveness of effectiveness of alendronate and alendronate and risedronate in risedronate in osteoporosisosteoporosis
Source of indirectnessSource of indirectness
Indirect comparisonIndirect comparison: : Randomized trials have compared Randomized trials have compared alendronate to placebo and alendronate to placebo and risedronate to placebo, but trials risedronate to placebo, but trials comparing alendronate to comparing alendronate to risedronate are unavailable.risedronate are unavailable.
Indirectness of evidenceIndirectness of evidence
Question of Question of interestinterest
Rosiglitazone for Rosiglitazone for prevention of prevention of diabetic diabetic complications in complications in patients at high patients at high risk of developing risk of developing diabetesdiabetes
Source of indirectnessSource of indirectness
Differences inDifferences in Outcome Outcome: A : A randomized trial shows delay in randomized trial shows delay in the development of biochemical the development of biochemical diabetes with rosiglitazone, but diabetes with rosiglitazone, but was underpowered to address was underpowered to address diabetic complicationsdiabetic complications
Indirectness of evidenceIndirectness of evidenceQuestion of interestQuestion of interest
Oseltamivir for Oseltamivir for prophylaxis of avian prophylaxis of avian flu caused by flu caused by influenza A(H5N1) influenza A(H5N1) virus virus
Sigmoidoscopic Sigmoidoscopic screening for screening for prevention of colon prevention of colon cancer mortalitycancer mortality
Choice of medication Choice of medication for schizophreniafor schizophrenia
Source of indirectnessSource of indirectness
Differences inDifferences in Population Population: randomized : randomized trials of oseltamivir are available for trials of oseltamivir are available for seasonal influenza, but not for avian fluseasonal influenza, but not for avian flu
Differences inDifferences in Intervention Intervention: : randomized trials of fecal occult blood randomized trials of fecal occult blood screening provide indirect evidence screening provide indirect evidence bearing on the potential effectiveness of bearing on the potential effectiveness of sigmoidoscopysigmoidoscopy
Differences inDifferences in Comparison Comparison: A series of : A series of trials comparing newer generation trials comparing newer generation neuroleptic agents to fixed doses of 20 neuroleptic agents to fixed doses of 20 mg. of haloperidol provide indirect mg. of haloperidol provide indirect evidence of how the newer agents would evidence of how the newer agents would compare to the lower, flexible doses of compare to the lower, flexible doses of haloperidol clinicians typically usehaloperidol clinicians typically use
Indirectness of evidenceIndirectness of evidence
Guidelines for the treatment of osteogenesis Guidelines for the treatment of osteogenesis imperfecta in adultsimperfecta in adults
In patients with osteoporosis, In patients with osteoporosis, bisphosphonates, either oral or IV, increase bisphosphonates, either oral or IV, increase BMD and reduce the risk of fragility fracture BMD and reduce the risk of fragility fracture by 50%by 50%
RCT oral bisphosphonates showed improved RCT oral bisphosphonates showed improved BMD in patients with OIBMD in patients with OI
RCT IV bisphosphonates showed improved RCT IV bisphosphonates showed improved cortical thickness in patients with OIcortical thickness in patients with OI
Questions about indirectness of Questions about indirectness of evidence?evidence?
Imprecise resultsImprecise results
VictorVictor
How to decide if the evidence How to decide if the evidence about a particular outcomes is about a particular outcomes is
impreciseimprecise 1. If the evidence were to come from a 1. If the evidence were to come from a
single RCT, how big should this RCT single RCT, how big should this RCT need to be to estimate the treatment need to be to estimate the treatment effect?effect?– Optimal information sizeOptimal information size
Control group event rate
Tota
l nu
mb
er
of
eve
nts
re
qu
ire
d
0.0 0.2 0.4 0.6 0.8 1.0
02
00
40
06
00
RRR=30%
RRR=25%
RRR=20%
300 events
How to decide if the evidence How to decide if the evidence about a particular outcomes is about a particular outcomes is
impreciseimprecise– Optimal information sizeOptimal information size– 300 events300 events
Stroke – periop beta blockersStroke – periop beta blockers
1 5 10 50 1000.5 0.1
Study Year Overall Event Rate Relative Risk (95% CI)
Wallace 1998 5 / 200 3.06 (0.49 to 19.02)
Pobble 2005 1 / 103 2.63 (0.11 to 62.97)
DIPOM 2006 2 / 921 4.97 (0.24 to 103.19)
MaVS 2006 6 / 496 1.83 (0.39 to 8.50)
Zaugg 2007 1 / 119 2.97 (0.12 to 72.19)
POISE 2007 60 / 8351 2.13 (1.25 to 3.64)
Fixed Effects Estimate 2.22 (1.39 to 3.56)
p=0.99 for heterogeneity, I²=0%
Offer all effective treatments?Offer all effective treatments? atrial fib at risk of strokeatrial fib at risk of stroke
warfarin increases serious gi bleedingwarfarin increases serious gi bleeding– 3% per year 3% per year
1,000 patients 1 less stroke1,000 patients 1 less stroke– 30 more bleeds for each stroke prevented30 more bleeds for each stroke prevented
1,000 patients 100 less strokes1,000 patients 100 less strokes– 3 strokes prevented for each bleed3 strokes prevented for each bleed
where is your threshold?where is your threshold?– how many strokes in 100 with 3% bleeding?how many strokes in 100 with 3% bleeding?
01.0%
01.0%
01.0%
00.5%1.0%
00.5%1.0%
01.0%
01.0%
How to decide if the evidence How to decide if the evidence about a particular outcomes is about a particular outcomes is
impreciseimprecise– Optimal information sizeOptimal information size– 300 events300 events– Evaluate the boundaries of the Evaluate the boundaries of the
CICI (if low event rates – CI for risk diff)(if low event rates – CI for risk diff)
Rate down for imprecision?Rate down for imprecision?
Rate down for imprecision for Rate down for imprecision for guidelines?guidelines?
Bleeding in patients with cancer Bleeding in patients with cancer receiving heparing for prolonging receiving heparing for prolonging
survivalsurvival
Rate down for imprecision for Rate down for imprecision for guidelines?guidelines?
How to decide if the evidence is How to decide if the evidence is imprecise for a imprecise for a
recommendationrecommendation– 300 events300 events– Evaluate the boundaries of the Evaluate the boundaries of the
CI in light of other important CI in light of other important outcomesoutcomes
Questions about imprecise Questions about imprecise results?results?
Publication biasPublication bias
VictorVictor
In 102 Danish protocols|122 reports in 2003In 102 Danish protocols|122 reports in 2003
50% of outcomes were incompletely 50% of outcomes were incompletely reportedreported
Odds of full reporting: Odds of full reporting:
2:1 2:1 if the outcome was if the outcome was
significantsignificant
JAMA. 2004;291:2457-2465.
Furukawa TA et al. JAMA 2007; 297: 468-9
1
10
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
<20% 20-39% 40-59% 60-80% >80%
Continuous outcomes
Dichotomous outcomes
% eligible trials contributing data to meta-analysis
Eff
ect
size
Od
ds
rati
o
Perioperative beta-Perioperative beta-blockerblocker
PositivePositivePoldermans (NEJM, Poldermans (NEJM,
n=112), RRR 80% n=112), RRR 80%
NegativeNegativeDIPOM (BMJ, n=921), DIPOM (BMJ, n=921),
RRI: 3%RRI: 3%POBBLE (J Vasc Surgery, POBBLE (J Vasc Surgery,
n=103), RRI = 6% n=103), RRI = 6% MAVS (Am Heart J, MAVS (Am Heart J,
n=496), RRR = 15%n=496), RRR = 15%
Statins for DM2Positive
CARDS (Lancet, 2004), RRR = 37%
NegativeASPEN (Diabetes Care, 2006), RRR =
10%
Publication biasPublication bias
Reporting of outcomesReporting of outcomes– Selective outcome reportingSelective outcome reporting
Reporting of studiesReporting of studies– Publication biasPublication bias
number of small studiesnumber of small studies industry sponsoredindustry sponsored
High likelihood could lower qualityHigh likelihood could lower quality
Questions about publication Questions about publication bias?bias?
Factors that can raise the quality Factors that can raise the quality of evidenceof evidence
AndyAndy
What can raise the quality What can raise the quality of evidence?of evidence?
Large magnitude of effect Large magnitude of effect A dose response relationshipA dose response relationshipAll plausible confoundingAll plausible confounding
Large magnitude of effectLarge magnitude of effect
Rules of thumb based on simulation studiesRules of thumb based on simulation studies– RR < 0.5RR < 0.5– RR < 0.2RR < 0.2
More compelling if: More compelling if: – consistency of effectconsistency of effect
almost everyone used to do badly, now does wellalmost everyone used to do badly, now does well– quick actionquick action
reversal of trajectoryreversal of trajectory– strong indirect evidencestrong indirect evidence
ExamplesExamples– insulin for diabetic ketoacidosisinsulin for diabetic ketoacidosis
– hip replacement for severe osteoarthritiship replacement for severe osteoarthritis
What are examples of interventions What are examples of interventions you can think of for which you might you can think of for which you might
upgrade the quality of evidence upgrade the quality of evidence because of a large magnitude of because of a large magnitude of
effect?effect?
Why would you or would you not Why would you or would you not upgrade the quality of evidence?upgrade the quality of evidence?
When do strong associations When do strong associations provide compelling evidence of provide compelling evidence of
effect (causation)?effect (causation)? Some interventions have such dramatic
effects that biases can be ruled out without randomised trials
Dramatic effects can be defined by the size of the treatment effect (signal) relative to the expected prognosis (noise)
Real treatment effects are likely if the signal to noise ratio is large (above 10)
Large ratios may be due to – the high proportion of patients improved
(consistency of effect) or– the rapidity of improvement (quick action)
And are more compelling if they are “plausible” (strong indirect evidence)
Dose response relationshipDose response relationship Risk of bleeding in patients who have Risk of bleeding in patients who have
supratherapeutic anticoagulation levels supratherapeutic anticoagulation levels – Dose response gradient between higher Dose response gradient between higher
levels of the international normalized ratio levels of the international normalized ratio (INR) and the increased risk of bleeding (INR) and the increased risk of bleeding
Risk of death in patients presenting Risk of death in patients presenting with sepsis and hypotensionwith sepsis and hypotension
What are examples of interventions What are examples of interventions you can think of for which you might you can think of for which you might
upgrade the quality of evidence upgrade the quality of evidence because of a dose response because of a dose response
relationship?relationship?
All plausible confounding All plausible confounding would result in an underestimate of would result in an underestimate of
the treatment effectthe treatment effect Higher death rates in private for-profit Higher death rates in private for-profit
versus private not-for-profit hospitalsversus private not-for-profit hospitals– patients in the not-for-profit hospitals patients in the not-for-profit hospitals
likely sicker than those in the for-profit likely sicker than those in the for-profit hospitalshospitals
– for-profit hospitals are likely to admit a for-profit hospitals are likely to admit a larger proportion of well-insured patients larger proportion of well-insured patients than not-for-profit hospitals (and thus have than not-for-profit hospitals (and thus have more resources with a spill over effect)more resources with a spill over effect)
All plausible biases All plausible biases would result in an overestimate of would result in an overestimate of
effecteffect Hypoglycaemic drug phenformin Hypoglycaemic drug phenformin
causes lactic acidosiscauses lactic acidosis The related agent metformin is under The related agent metformin is under
suspicion for the same toxicity. suspicion for the same toxicity. Large observational studies have failed Large observational studies have failed
to demonstrate an associationto demonstrate an association– Clinicians would be more alert to lactic Clinicians would be more alert to lactic
acidosis in the presence of the agentacidosis in the presence of the agent
What are examples of interventions What are examples of interventions you can think of for which you might you can think of for which you might
upgrade the quality of evidence upgrade the quality of evidence
when an effect (association) was when an effect (association) was found - because all plausible found - because all plausible
confounders would result in an confounders would result in an underestimate of effect orunderestimate of effect or
when an effect (association) was not when an effect (association) was not found - because all plausible biases found - because all plausible biases would result in an overestimate of would result in an overestimate of
effect?effect?
Questions about factors that can Questions about factors that can raise the quality of evidence?raise the quality of evidence?
Judgements about the overall Judgements about the overall quality of evidencequality of evidence
AndyAndy
Quality assessment criteriaQuality assessment criteria
Judgements about the overall Judgements about the overall quality of evidencequality of evidence
Most systems not explicitMost systems not explicit
Options:Options:– BenefitsBenefits– Primary outcomePrimary outcome– Highest Highest – LowestLowest
Based on lowest of all the Based on lowest of all the criticalcritical outcomesoutcomes
Beyond the scope of a systematic reviewBeyond the scope of a systematic review
Flavonoids for HemorrhoidsFlavonoids for Hemorrhoids
venotonic agentsvenotonic agents– mechanism unclear, increase venous returnmechanism unclear, increase venous return
popularitypopularity– 90 venotonics commercialized in France90 venotonics commercialized in France– none in Sweden and Norwaynone in Sweden and Norway– France 70% of world marketFrance 70% of world market
possibilitiespossibilities– French misguidedFrench misguided– Swedes and Norwegians missing outSwedes and Norwegians missing out
Systematic ReviewSystematic Review
14 trials, 1432 patients14 trials, 1432 patients
Key outcomeKey outcome– risk not improving/persistent symptomsrisk not improving/persistent symptoms– 11 studies, 1002 patients, 375 events11 studies, 1002 patients, 375 events
– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57
Minimal side effectsMinimal side effects
Is France right?Is France right?
What is the quality of evidence?What is the quality of evidence?
What can lower the quality of What can lower the quality of evidence?evidence?
Study limitationsStudy limitations– lack of detail re randomisation, allocation lack of detail re randomisation, allocation
concealment and blindingconcealment and blinding– questionnaires not validatedquestionnaires not validated
Rate down quality for study limitations?Rate down quality for study limitations?
Indirectness – no problem?Indirectness – no problem?
Inconsistency - need to look at the resultsInconsistency - need to look at the results
Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement
Study RR (random) Weight RR (random)or sub-category log[RR] (SE) 95% CI % 95% CI
01 Up to seven daysChauvenet -0.8916 (0.2376) 12.67 0.41 [0.26, 0.65] Cospite -2.2073 (0.6117) 5.51 0.11 [0.03, 0.36] Thanapongsathorn -0.4308 (0.2985) 11.18 0.65 [0.36, 1.17]
Subtotal (95% CI) 29.36 0.37 [0.18, 0.77]Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1%Test for overall effect: Z = 2.67 (P = 0.008)
02 Up to four w eeksAnnoni F -1.6094 (0.7073) 4.50 0.20 [0.05, 0.80] Clyne MB -0.9943 (0.3983) 8.94 0.37 [0.17, 0.81] Pirard J -1.1712 (0.3086) 10.94 0.31 [0.17, 0.57] Thanapongsathorn -1.1087 (1.1098) 2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan -0.8916 (0.3691) 9.56 0.41 [0.20, 0.85] Wijayanegara -0.5978 (0.1375) 14.97 0.55 [0.42, 0.72]
Subtotal (95% CI) 61.54 0.48 [0.32, 0.72]Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7%Test for overall effect: Z = 3.57 (P = 0.0004)
03 Further than four w eeksGodeberg -1.7719 (0.3906) 9.10 0.17 [0.08, 0.37]
Subtotal (95% CI) 9.10 0.17 [0.08, 0.37]Test for heterogeneity: not applicableTest for overall effect: Z = 4.54 (P < 0.00001)
Total (95% CI) 100.00 0.40 [0.29, 0.57]Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1%Test for overall effect: Z = 5.14 (P < 0.00001)
0.001 0.01 0.1 1 10 100 1000
Favours treatment Favours control
Publication bias?Publication bias?
Size of studiesSize of studies– 40 to 234 patients, most around 10040 to 234 patients, most around 100
All industry sponsoredAll industry sponsored
Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement
0.001 0.01 0.1 1 10 100 1000
0.0
0.4
0.8
1.2
1.6
RR (fixed)
What is the overall quality of What is the overall quality of evidence across criteria?evidence across criteria?
HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.
ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.
LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.
Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.
Threshold for downgradingThreshold for downgrading
highestlowestOKrate down quality
Overall quality across criteriaOverall quality across criteria
highestlowesthighmoderate low very low
Is France right?Is France right?
RecommendRecommend– for or against?for or against?
Strength of recommendationStrength of recommendation– strong or weakstrong or weak
Questions about judgements of Questions about judgements of the overall quality of evidence?the overall quality of evidence?
Workshop agendaWorkshop agendaMonday, 12 May 2008Monday, 12 May 2008
08.15 – 13.30 h - Plenary08.15 – 13.30 h - Plenary Introduction to GRADEIntroduction to GRADE Grading the quality of evidence for each Grading the quality of evidence for each
important outcomeimportant outcome– Deciding which outcomes are importantDeciding which outcomes are important– Factors that can lower the quality of evidenceFactors that can lower the quality of evidence– Factors that can raise the quality of evidenceFactors that can raise the quality of evidence
Judgements about the overall quality of evidenceJudgements about the overall quality of evidence15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups The quality of evidence for HPV vaccinationThe quality of evidence for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion
Should routine HPV vaccination Should routine HPV vaccination be recommended for be recommended for
12 year old girls in Spain (with 12 year old girls in Spain (with continued screening)?continued screening)?
Introduction to small group workIntroduction to small group work
Decide which outcomes are important
Assess the quality of evidence
Questions about the small group Questions about the small group work?work?
Grading the quality of evidence Grading the quality of evidence and the strength of and the strength of recommendationsrecommendations
San SebastiánSan Sebastián13 May 200813 May 2008
Pablo Alonso, Victor Montori, Andy Oxman, Holger Schünemann
Workshop agendaWorkshop agendaTuesday, 13 May 2008Tuesday, 13 May 2008
Strength of recommendationStrength of recommendation
08.30 – 13.30 h - Plenary08.30 – 13.30 h - Plenary Going from evidence to recommendationsGoing from evidence to recommendations
– Summarising and presenting findingsSummarising and presenting findings– Factors that determine the strength of a recommendationFactors that determine the strength of a recommendation
Balance of benefits and harmsBalance of benefits and harms (Quality of evidence)(Quality of evidence) Values and preferencesValues and preferences Costs (resource use)Costs (resource use)
– Judgements about the strength of recommendationsJudgements about the strength of recommendations– Wording of recommendationsWording of recommendations
15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups Summary of findings and the strength of Summary of findings and the strength of
recommendation for HPV vaccinationrecommendation for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion
Any questions from yesterday about Any questions from yesterday about quality of evidence quality of evidence
or about the agenda for today?or about the agenda for today?
Summarising and presenting Summarising and presenting findingsfindings
AndyAndy
Some GRADE jargon
Quality assessment Summary of findings Evidence profile = quality assessment
+ summary of findings
Quality assessment
Summary of Findings
GRADE Evidence Profile
Headings specify the comparison and applicability of
the SoF table
Up to 7 patient-important outcomes including important outcomes for
which no data are available
The relative effect (e.g. risk ratio)
Illustrative comparative risks present the absolute effect for specified
(“assumed”) baseline or control group risks
How much evidence there is
The quality of the evidence
Comments
FootnotesAbout the interventionAbout the intervention
High and low risk groupsHigh and low risk groups
Judgements about qualityJudgements about quality
Explanation of outcomesExplanation of outcomes
Explanations for SoF tables
Questions about summarising Questions about summarising and presenting findings?and presenting findings?
Strength of recommendations Strength of recommendations and and
values & preferencesvalues & preferences
HolgerHolger
Strength of recommendationStrength of recommendationThe degree of confidence that the The degree of confidence that the
desirable effects of adherence to a desirable effects of adherence to a recommendation outweigh the recommendation outweigh the undesirable effects. undesirable effects.
Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings
Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts
You have to decide where you You have to decide where you want to spend the next five want to spend the next five
years of your live.years of your live.
Which of the following cities Which of the following cities would you prefer to live in?would you prefer to live in?
← Option 1 (Buffalo, purple)
Option 2 → (Rome, yellow)
A bit more practice with the A bit more practice with the voting instrumentvoting instrument
You are hiking.You are hiking.
Which of the following animals Which of the following animals would you prefer to would you prefer to
encounter?encounter?
← Option 1 (bunny, purple)
Option 2 → (tiger, yellow)
You are buying an ice cream.You are buying an ice cream.
Which flavor do you prefer?Which flavor do you prefer?
← Option 1 (strawberry,
purple)
Option 2 → (chocolate,
yellow)
Chocolate
Strawberry
You are buying a new car.You are buying a new car.
Which one would you buy?Which one would you buy?
← Option 1 (Ferrari, purple)
Option 2 → (Volkswagen,
yellow)
Yellow fox
Red Ferrari
What determines your choices?What determines your choices?
What determines your choices?What determines your choices?
•Safety
•Resources
•Preferences (that we cannot rationally explain)
•Risk taking/avoidance
•Burden
•Comfort
•Cost and societal responsibility
Which information do we have?Which information do we have?
•Safety
•Resources
•Preferences (that we cannot rationally explain)
•Risk taking/avoidance
•Burden
•Comfort
•Cost and societal responsibility
Strength of recommendationStrength of recommendationThe degree of confidence that the The degree of confidence that the
desirable effects of adherence to a desirable effects of adherence to a recommendation outweigh the recommendation outweigh the undesirable effects. undesirable effects.
Desirable Desirable effectseffects•health benefitshealth benefits•less burdenless burden•savingssavings
Undesirable Undesirable effectseffects•harmsharms•more burdenmore burden•costscosts
GRADE: Factors influencing GRADE: Factors influencing decisions and decisions and
recommendationsrecommendations Quality of EvidenceQuality of Evidence Balance of desirable and undesirable Balance of desirable and undesirable
consequencesconsequences Values and preferencesValues and preferences CostCost
Categories of recommendationsCategories of recommendations
Although the degree of confidence is a Although the degree of confidence is a continuum, we suggest using two continuum, we suggest using two categories: strong and categories: strong and weak/conditional.weak/conditional.
Strong recommendation:Strong recommendation: the panel is the panel is confident that the desirable effects of confident that the desirable effects of adherence to a recommendation adherence to a recommendation outweigh the undesirable effects.outweigh the undesirable effects.
Weak recommendation:Weak recommendation: the panel the panel concludes that the desirable effects of concludes that the desirable effects of adherence to a recommendation adherence to a recommendation probably outweigh the undesirable probably outweigh the undesirable effects, but is not confident. effects, but is not confident.
Recommend
Suggest
Judgements about the strength Judgements about the strength of a recommendationof a recommendation
No precise threshold for going from a strong to a No precise threshold for going from a strong to a weak recommendationweak recommendation
The presence of important concerns about one or The presence of important concerns about one or more of these factors make a weak more of these factors make a weak recommendation more likely. recommendation more likely.
Panels should consider all of these factors and Panels should consider all of these factors and make the reasons for their judgements explicit.make the reasons for their judgements explicit.
Recommendations should specify the Recommendations should specify the perspectiveperspective that is taken (e.g. individual patient, that is taken (e.g. individual patient, health system) and which outcomes were health system) and which outcomes were considered (including which, if any costs). considered (including which, if any costs).
ValuesValues
Perspective: individual patient vs Perspective: individual patient vs systemsystem
Include a description of how decisions Include a description of how decisions were made about the relative were made about the relative importance (benefits, harms and costs)importance (benefits, harms and costs)
Values reported along with the Values reported along with the research evidence underlying research evidence underlying recommendationsrecommendations
Determinants of the strength Determinants of the strength of recommendationof recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effects
The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted
Determinants of the strength Determinants of the strength of recommendationof recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effects
The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted
Determinants of the strength Determinants of the strength of recommendationof recommendation
Factors that can strengthen a recommendation
Comment
Quality of the evidence Low
Balance between desirable and undesirable effects
Moderately balance
Values and preferences Fair bit of variability
Costs (resource allocation) Relatively high cost
Determinants of the strength Determinants of the strength of recommendationof recommendation
Factors that can weaken the strength of a recommendation. Example:
Decision Explanation
Lower quality evidence □ Yes□ No
Low
Uncertainty about the balance of benefits versus harms and burdens
□ Yes□ No
Moderate y balanced
Uncertainty or differences in values □ Yes□ No
Fair bit of variability
Uncertainty about whether the net benefits are worth the costs
□ Yes□ No
Relatively high cost
Table. Decisions about the strength of a recommendationFrequent “yes” answers will increase the likelihood of a weak recommendation
Implications of a strong Implications of a strong recommendationrecommendation
PatientsPatients: Most people in your situation : Most people in your situation would want the recommended course of would want the recommended course of action and only a small proportion would action and only a small proportion would notnot
CliniciansClinicians: Most patients should receive : Most patients should receive the recommended course of actionthe recommended course of action
Policy makersPolicy makers: The recommendation : The recommendation can be adapted as a policy in most can be adapted as a policy in most situationssituations
Implications of a weak Implications of a weak recommendationrecommendation
PatientsPatients: The majority of people in your : The majority of people in your situation would want the recommended situation would want the recommended course of action, but many would not course of action, but many would not
CliniciansClinicians: Be prepared to help patients : Be prepared to help patients to make a decision that is consistent to make a decision that is consistent with their own valueswith their own values
Policy makersPolicy makers: There is a need for : There is a need for substantial debate and involvement of substantial debate and involvement of stakeholdersstakeholders
Is France right?Is France right? RecommendRecommend
– for or against?for or against?
Strength of recommendationStrength of recommendation– strong or weakstrong or weakFactors that can strengthen a
recommendation Comment
Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effects
The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted
Another exampleAnother example Should oseltamivir be used for treatment of Should oseltamivir be used for treatment of
patients hospitalised with avian influenza patients hospitalised with avian influenza (H5N1)?(H5N1)?
Schünemann HJ et al. WHO Rapid Schünemann HJ et al. WHO Rapid Advice Guidelines for the Advice Guidelines for the pharmacological management of pharmacological management of human infection with avian influenza A human infection with avian influenza A (H5N1) virus. Lancet ID 2007; 7:21-31.(H5N1) virus. Lancet ID 2007; 7:21-31.
Schünemann HJ et al. WHO Rapid Schünemann HJ et al. WHO Rapid Advice Guidelines: Quick and Advice Guidelines: Quick and Transparent. PLoS Medicine 2007; Transparent. PLoS Medicine 2007; 4:e119.4:e119.
Should oseltamivir be used for treatment of patients Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)?hospitalised with avian influenza (H5N1)?
Summary of findings
Transmission: No human to human transmissionPatient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza
Oseltamivir for Avian FluOseltamivir for Avian Flu
Summary of findings: Summary of findings: • No clinical trial of oseltamivir for treatment of H5N1 No clinical trial of oseltamivir for treatment of H5N1
patients.patients.• 4 systematic reviews and health technology 4 systematic reviews and health technology
assessments (HTA) reporting on 5 studies of oseltamivir assessments (HTA) reporting on 5 studies of oseltamivir in in seasonalseasonal influenza. influenza. • Hospitalization: OR 0.22 (0.02 – 2.16)Hospitalization: OR 0.22 (0.02 – 2.16)• Pneumonia: OR 0.15 (0.03 - 0.69)Pneumonia: OR 0.15 (0.03 - 0.69)
• 3 published case series. 3 published case series. • Many in vitro and animal studies. Many in vitro and animal studies. • No alternative that is more promising at present.No alternative that is more promising at present.• Cost: ~ Euro 50 per treatment courseCost: ~ Euro 50 per treatment course
What would you recommend?What would you recommend?
Factors Impact on the strength of a recommendation
Balance between desirable and undesirable effects
Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted.
Quality of the evidence
Higher the quality of evidence, more likely a strong recommendation warranted.
Values and preferences
More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted.
Costs (resource use)
Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.
Judgments about the strength of a recommendation - Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with oseltamivir for treatment of patients hospitalised with
avian influenza (H5N1)avian influenza (H5N1)
Factors Comments
Balance between desirable and undesirable effects
The benefits are uncertain.
Quality of the evidence The quality of the evidence is very low.
Values and preferences All patients and care providers would accept treatment for H5N1 disease.
Costs (resource use) The cost is not high for treatment of sporadic cases.
Questions about Questions about the strength of the strength of
recommendations?recommendations?
Costs (resource use)Costs (resource use)
AndyAndy
BMJ 2008;336:1049-1051 (10 May)
Determinants of strength of recommendationDeterminants of strength of recommendation
Factors Impact on the strength of a recommendation
Balance between desirable and undesirable effects
Larger the difference between the desirable and undesirable effects, more likely a strong recommendation warranted. Narrower the gradient, more likely weak recommendation warranted.
Quality of the evidence
Higher the quality of evidence, more likely a strong recommendation warranted.
Values and preferences
More variability in values and preferences, or more uncertainty in values and preferences, more likely weak recommendation warranted.
Costs (resource use)
Higher the costs of an intervention – that is, the more resources consumed – less likely a strong recommendation warranted.
Why do costs differ from other health outcomes?
Patients receive health benefits and bear the burden of adverse health outcomes; health care cost are typically shared by society as a whole, employers, and patients.
Health care cost may vary widely among and even within jurisdictions and quickly change over time.
What societies can purchase if they forego health care resource use (opportunity cost) varies widely between countries. A year’s supply of an expensive drug may pay half a nurse’s salary in the United States and 30 nurses’ salaries in China.
Resource use issues are highly political.
Despite the differences, approaches to resource use are
similar to other outcomes
guideline panels need consider only important resource implications
decision makers require an estimate of the difference between treatment and control
panels should make explicit judgments about the quality of the evidence regarding incremental resource use
Special considerations of using resource use as an outcome
include which perspective to take which resource implications to include how to make judgments about the
quality of the evidence how to present these implications the potential usefulness – or lack of
usefulness – of a formal economic model in arriving at a wise recommendation or decision
Cost-effectiveness analyses of HPV vaccination
Lancet Infect Dis 2007; 7:289–96
Resource implications of HPV vaccination
Lancet Infect Dis 2007; 7:289–96
Questions about the impact of costs Questions about the impact of costs (resource use) on the strength of a (resource use) on the strength of a
recommendation?recommendation?
Wording of recommendationsWording of recommendations
HolgerHolger
Labeling of Labeling of concept/recommendationsconcept/recommendations
Need to express two levelsNeed to express two levels
1)1) Strong/compulsoryStrong/compulsory
2)weak/conditional/discretionary/2)weak/conditional/discretionary/facultative/guarded/facultative/guarded/
Challenges in wording Challenges in wording recommendationsrecommendations
Need to express two levelsNeed to express two levels Need to express directionNeed to express direction Differences across languagesDifferences across languages
Need codes (letters, symbols, numbers)Need codes (letters, symbols, numbers)
Wording 1 Wording 2 Wording 3 Strong recommendation for We recommend… Clinicians should… We recommend…
Weak recommendation for We suggest Clinicians might… We conditionally
recommend… Weak recommendation against
We suggest...not Clinicians might not…
We conditionally recommend...not
Strong recommendation against
We recommend …not
Clinicians should not…
We recommend …not
Example: Oseltamivir for Avian FluExample: Oseltamivir for Avian Flu
Recommendation: In patients with confirmed or Recommendation: In patients with confirmed or strongly suspected infection with avian influenza strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (oseltamivir treatment as soon as possible (????? ????? recommendation, very low quality evidencerecommendation, very low quality evidence). ).
Schunemann et al., The Lancet ID, 2007
Example: Oseltamivir for Avian FluExample: Oseltamivir for Avian Flu
Recommendation: In patients with confirmed or strongly suspected infection with Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (as soon as possible (strong recommendation, very low quality evidencestrong recommendation, very low quality evidence).).
Values and PreferencesValues and Preferences
Remarks: This recommendation places a high value on the prevention of death in Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. reactions, the development of resistance and costs of treatment.
Schunemann et al., The Lancet ID, 2007
Other explanationsOther explanations
Remarks: Despite the lack of controlled treatment data for H5N1, this is a Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time. alternative pharmacological interventions at this time.
The panel voted on whether this recommendation should be strong or The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.weak and there was one abstention and one dissenting vote.
Schunemann et al., The Lancet ID, 2007
ATS/ERS COPD guidelines
NICE COPD guidelines
GOLD COPD guidelines
RecommendationsRecommendations
Recommendation:Recommendation:For COPD patients with chronic cough For COPD patients with chronic cough productive of sputum, we suggest acetylcystein productive of sputum, we suggest acetylcystein therapy (weak recommendation, moderate therapy (weak recommendation, moderate quality evidence - 2B)quality evidence - 2B)
Values and preferences:Values and preferences:This recommendation places a relatively high This recommendation places a relatively high value on the possible reduction of cough and value on the possible reduction of cough and improvement in quality of life and a low value improvement in quality of life and a low value on the possible adverse outcomes (and resource on the possible adverse outcomes (and resource expenditure) related to mucolytics. expenditure) related to mucolytics.
PICO:Population, intervention, comparison (and outcome)
Questions about Questions about the wording of the wording of
recommendations?recommendations?
Recommendations to use an Recommendations to use an intervention only in the context of intervention only in the context of
researchresearch
AndyAndy
Possible recommendations when there is very low quality
evidence include:
Recommend using an intervention Suggest using an intervention
Recommend using an intervention only in the context of research
Suggest not using an intervention
Recommend not using an intervention
Recommendations to use an Recommendations to use an intervention only in the context intervention only in the context of research may be appropriateof research may be appropriate
Two conditions must be metTwo conditions must be met1.1. There must be inThere must be insufficient evidence sufficient evidence
for a panel to suggest using or not for a panel to suggest using or not using an intervention. using an intervention.
2.2. Further research must have a large Further research must have a large potential for reducing uncertainty potential for reducing uncertainty about the effects of the intervention, about the effects of the intervention, and for doing so at a reasonable cost. and for doing so at a reasonable cost.
ExamplesExamples
National Institute of Health and Clinical National Institute of Health and Clinical Excellence (NICE) Excellence (NICE) – Of its first 95 technology appraisals, eight led to Of its first 95 technology appraisals, eight led to
recommendations for use in the context of research.recommendations for use in the context of research. Centers for Medicare and Medicaid Services Centers for Medicare and Medicaid Services
(CMMS)(CMMS)– June 7, 2000 executive memorandum directing the Secretary June 7, 2000 executive memorandum directing the Secretary
of Health and Human Services to "explicitly authorize of Health and Human Services to "explicitly authorize [Medicare] payment for routine patient care costs...and costs [Medicare] payment for routine patient care costs...and costs due to medical complications associated with participation in due to medical complications associated with participation in clinical trials.“clinical trials.“
– July 9, 2007 the following was added to the Clinical Trial Policy:July 9, 2007 the following was added to the Clinical Trial Policy:– CMMS . . . through an individualized assessment of benefits, CMMS . . . through an individualized assessment of benefits,
risks, and research potential, may determine that certain items risks, and research potential, may determine that certain items and services for which there is some evidence of significant and services for which there is some evidence of significant medical benefit, but for which there is insufficient evidence to medical benefit, but for which there is insufficient evidence to support a “reasonable and necessary” determination, are only support a “reasonable and necessary” determination, are only reasonable and necessary when provided in a clinical trial . . .reasonable and necessary when provided in a clinical trial . . .
Implications for researchImplications for researchCore elements E Evidence (What is the current state of the
evidence?) P Population (What is the population of interest?) I Intervention (What are the interventions of interest?) C Comparison (What are the comparisons of interest?) O Outcome (What are the outcomes of interest?) T Time stamp (Date of recommendation)Optional elements d Disease burden or relevance t Time aspect of core elements of EPICOT s Appropriate study type according to local need
What are examples of interventions What are examples of interventions you can think of for which you might you can think of for which you might
recommend that it should only be recommend that it should only be used in the context of research?used in the context of research?
Questions about Questions about recommendations to use an recommendations to use an
intervention only in the context of intervention only in the context of research?research?
Judgements about priorities Judgements about priorities (importance)(importance)
AndyAndy
Priority setting:Priority setting:From a health system From a health system
or public health or public health perspectiveperspective
A A strongstrong recommendation recommendation may not be may not be importantimportant
Judgements about Judgements about importanceimportance
may depend onmay depend on StrengthStrength of the recommendations of the recommendations SeveritySeverity of the problemof the problem PrevalencePrevalence of the problem of the problem Magnitude of theMagnitude of the net benefitsnet benefits EfficiencyEfficiency (better use of resources) (better use of resources) Total costTotal cost Impact onImpact on inequitiesinequities Potential for improved Potential for improved quality of carequality of care
From the perspective of From the perspective of individual patients and individual patients and
cliniciansclinicians Most patients would want and should Most patients would want and should
receive receive strongly recommendedstrongly recommended courses of actioncourses of action
The majority of people would want The majority of people would want weakly recommendedweakly recommended courses of courses of action, but many would not and clinicians action, but many would not and clinicians should be prepared to help patients make should be prepared to help patients make a decision that is consistent with their a decision that is consistent with their own valuesown values
The issue of priority setting (importance) The issue of priority setting (importance) is usually not important.is usually not important.
Questions about Questions about judgements about priorities judgements about priorities
(importance)?(importance)?
Workshop agendaWorkshop agendaTuesday, 13 May 2008Tuesday, 13 May 2008
Strength of recommendationStrength of recommendation
08.30 – 13.30 h - Plenary08.30 – 13.30 h - Plenary Going from evidence to recommendationsGoing from evidence to recommendations
– Summarising and presenting findingsSummarising and presenting findings– Factors that determine the strength of a recommendationFactors that determine the strength of a recommendation
Balance of benefits and harmsBalance of benefits and harms (Quality of evidence)(Quality of evidence) Values and preferencesValues and preferences Costs (resource use)Costs (resource use)
– Judgements about the strength of recommendationsJudgements about the strength of recommendations– Wording of recommendationsWording of recommendations
15.00 – 17.00 h - Small groups15.00 – 17.00 h - Small groups Summary of findings and the strength of Summary of findings and the strength of
recommendation for HPV vaccinationrecommendation for HPV vaccination17.00 – 18.00 h - Plenary17.00 – 18.00 h - Plenary DiscussionDiscussion