Grading evidence and Grading evidence and recommendationsrecommendations

The GRADE initiativeThe GRADE initiative

Holger Schünemann, MD, PhDHolger Schünemann, MD, PhDAssociate Professor Associate Professor

Italian National Cancer Institute Regina Elena, Italian National Cancer Institute Regina Elena, RomeRome

Professional good intentions and Professional good intentions and plausible theories are plausible theories are insufficientinsufficient

for selecting policies and for selecting policies and practices for protecting, practices for protecting,

promoting and restoring healthpromoting and restoring health..

Iain Chalmers

How can we judge the How can we judge the extent of our confidence extent of our confidence

that adherence to that adherence to aa recommendation will do recommendation will do more good than harm?more good than harm?

GRADEGRADE

Grades of Recommendation Assessment, Development

and Evaluation

What do you know about What do you know about GRADE?GRADE?

o Have prepared a guidelineHave prepared a guidelineo Read the BMJ paper Read the BMJ paper

o Have prepared a systematic review and a Have prepared a systematic review and a summary of findings tablesummary of findings table

o Have attended a GRADE meeting, workshop or talkHave attended a GRADE meeting, workshop or talk

About GRADE*About GRADE*

o Began as informal working group in 2000Began as informal working group in 2000o Researchers/guideline developers with Researchers/guideline developers with

interest in methodologyinterest in methodologyo Aim: to develop a Aim: to develop a commoncommon system for system for

grading the quality of evidence and the grading the quality of evidence and the strength of recommendations that is strength of recommendations that is sensible and to explore the range of sensible and to explore the range of interventions and contexts for which it might interventions and contexts for which it might be useful*be useful*

o 13 meetings (~10 – 35 attendants)13 meetings (~10 – 35 attendants)o Evaluation of existing systems and Evaluation of existing systems and

reliability*reliability*o Workshops at Cochrane Colloquia, WHO, GIN Workshops at Cochrane Colloquia, WHO, GIN

and various conferences since 2000and various conferences since 2000*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005

GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Martin Eccles, professorMartin Eccles, professordd Francoise Cluzeau, lecturerFrancoise Cluzeau, lecturerxx

Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information quality and information

directordirector h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturer jj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Regina Kunx, Associate ProfessorRegina Kunx, Associate ProfessorGillian Leng, guidelines programme directorGillian Leng, guidelines programme director ll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn

James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorHolger J Schünemann, associate professorg,sg,s Tessa Tan-Torres Edejer, medical officerTessa Tan-Torres Edejer, medical officertt Jane Thomas, Lecturer, UKJane Thomas, Lecturer, UKHelena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, professorJohn W Williams Jr, professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww

a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USA USA b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention, USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services, NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network, UKUKi) Fédération Nationale des Centres de Lutte i) Fédération Nationale des Centres de Lutte Contre le Cancer, Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Assistenza Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, Australia Australia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine, PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) National Cancer Institute, National Cancer Institute, ItalyItalyt) World Health Organisation, t) World Health Organisation, Switzerland Switzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, Finland Finland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, USAUSAx) University of London, x) University of London, UKUK

What do users want from What do users want from guidelines?guidelines?

• users looking for different things

• just tell me what to do (recommendation)

• what to do, and on strong or weak grounds• recommendation and grade

• recommend, grade, evidence summary, values• systematic review, value statement

• evidence from individual studies

When to make a recommendation?When to make a recommendation?

•never• patient values differ• just lay out benefits and risks

•when evidence strong enough• when very weak, too uncertain

•clinicians need guidance• intense study demands decision

Why bother about grading?Why bother about grading? People draw conclusions about the

– quality of evidence– strength of recommendations

Systematic and explicit approaches can help– protect against errors– resolve disagreements– facilitate critical appraisal– communicate information

However, there is wide variation in currently used approaches

Who is confused? Who is confused?

Evidence Evidence RecommendationRecommendation

II-2II-2 BB C+ C+ 11 StrongStrong Strongly Strongly

recommendedrecommended

OrganizatioOrganizationn

USPSTFUSPSTF ACCPACCP GCPSGCPS

Still not confused?Still not confused?

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I C+ C+ 11 IVIV CC

OrganizatioOrganizationn

AHAAHA ACCPACCP SIGNSIGN

Recommendation for use of oral Recommendation for use of oral anticoagulation in patients with atrial anticoagulation in patients with atrial fibrillation and rheumatic mitral valve fibrillation and rheumatic mitral valve diseasedisease

Grading SystemGrading System

• current profusion: can there be consensus?

• trade-off benefits and risks• do it (or don’t do it)• probably do it (or probably don’t do it)

• quality of underlying evidence• high quality (well done RCT)• intermediate (quasi-RCT)• low (well done observational)• very low (anything else)

Moving downMoving down

• poor RCT design, implementation• randomization, concealment, follow-up• inconsistency

• indirect• patients, interventions, outcomes• A vs B, but have A to C, B to C

• reporting bias

Moving upMoving up

• magnitude of effect

• dose-response

• biases favor control

Guidelines development Guidelines development processprocess

Prioritise Problems, establish panelPrioritise Problems, establish panel

Systematic ReviewSystematic Review

Evidence ProfileEvidence Profile

Relative importance of outcomesRelative importance of outcomes

Overall quality of evidenceOverall quality of evidence

Benefit – downside evaluationBenefit – downside evaluation

Strength of recommendationStrength of recommendation

Implementation and evaluation of guidelinesImplementation and evaluation of guidelines

Guidelines development Guidelines development processprocess

Prioritise Problems, establish panelPrioritise Problems, establish panel

Systematic ReviewSystematic Review

Evidence ProfileEvidence Profile

Relative importance of outcomesRelative importance of outcomes

Overall quality of evidenceOverall quality of evidence

Benefit – downside evaluationBenefit – downside evaluation

Strength of recommendationStrength of recommendation

Implementation and evaluation of guidelinesImplementation and evaluation of guidelines

Example ACCPExample ACCP• First ACCP guidelines in 1986 (J. Hirsh; J. First ACCP guidelines in 1986 (J. Hirsh; J.

Dalen)Dalen)• Initially aimed at consensus Initially aimed at consensus • Methodologists involved since beginning Methodologists involved since beginning • Now formally convening every 2 to 3 yearsNow formally convening every 2 to 3 years• Seventh conference held in 2003; > Seventh conference held in 2003; >

200.000 copies published in Chest200.000 copies published in Chest• 87 panel members, 22 chapters 87 panel members, 22 chapters • Across subspecialtiesAcross subspecialties• 565 recommendations, 230 new565 recommendations, 230 new• Evidence Based RecommendationsEvidence Based Recommendations• Next conference in 2006Next conference in 2006

Evidence – recommendation: transparent link

Explicit inclusion criteria Comprehensive search

Standardized consideration of study quality

Conduct/use meta-analysis

Evaluate overall quality of evidence

Grade recommendations

Acknowledge values and preferences

What makes guidelines evidence based (in 2005)?

Schünemann et al. Chest 2004

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

Most systems not explicitMost systems not explicit

Options:Options:– strongest outcomestrongest outcome– primary outcomeprimary outcome– benefitsbenefits– weightedweighted– separate grades for benefits and harmsseparate grades for benefits and harms– no overall gradeno overall grade– weakest outcomeweakest outcome

Based on lowest of all the Based on lowest of all the criticalcritical outcomes outcomes Beyond the scope of a systematic reviewBeyond the scope of a systematic review

Quality of evidenceQuality of evidence

““The extent to which one can be confident that an The extent to which one can be confident that an estimate of effect or association is correct.” estimate of effect or association is correct.”

It depends on the:It depends on the:– study designstudy design (e.g. RCT, cohort study) (e.g. RCT, cohort study)– study quality/limitationsstudy quality/limitations (protection against bias; (protection against bias;

e.g. concealment of allocation, blinding, follow-up)e.g. concealment of allocation, blinding, follow-up)– consistency of resultsconsistency of results– directness of the evidencedirectness of the evidence including the including the

populationspopulations (those of interest versus similar; for (those of interest versus similar; for example, older, sicker or more co-morbidity)example, older, sicker or more co-morbidity)

interventionsinterventions (those of interest versus similar; for (those of interest versus similar; for example, drugs within the same class)example, drugs within the same class)

outcomesoutcomes (important versus surrogate outcomes) (important versus surrogate outcomes) comparisoncomparison (A - C versus A - B & C - B) (A - C versus A - B & C - B)

Quality of evidenceQuality of evidence

The quality of the evidence (i.e. our confidence) may also The quality of the evidence (i.e. our confidence) may also be be REDUCEDREDUCED when there is: when there is:

Sparse or imprecise dataSparse or imprecise data Reporting biasReporting bias

The quality of the evidence (i.e. our confidence) may be The quality of the evidence (i.e. our confidence) may be INCREASEDINCREASED when there is: when there is:

A strong association A strong association A dose response relationshipA dose response relationship All plausible confounders would have reduced the All plausible confounders would have reduced the

observed effect observed effect All plausible biases would have increased the observed All plausible biases would have increased the observed

lack of effectlack of effect

Quality assessment criteriaQuality assessment criteria

Quality of evidence Study design Lower if Higher if

High Randomised trial

Moderate

Low Observational

study

Very low Any other evidence

Study quality:

-1 Serious

limitations

-2 Very serious

limitations

-1 Important

inconsistency

Directness:

-1 Some

uncertainty

-2 Major

uncertainty

-1 Sparse or

imprecise data

-1 High probability

of reporting bias

Strong association:

+1 Strong, no

plausible

confounders

+2 Very strong,

no major

threats to

validity

+1 Evidence of a

Dose response

gradient

+1 All plausible

confounders

would have

reduced the

effect

Categories of qualityCategories of quality

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Strength of recommendationStrength of recommendation

““The extent to which one can be confident that The extent to which one can be confident that adherence to a recommendation will do more adherence to a recommendation will do more good than harm.” good than harm.”

quality of the evidencequality of the evidence translation of the evidencetranslation of the evidence into practice into practice

in a specific settingin a specific setting uncertainty about baseline riskuncertainty about baseline risk trade-offstrade-offs (the relative value attached to (the relative value attached to

the expected benefits, harms and costs)the expected benefits, harms and costs)

Clarity of the trade-offs Clarity of the trade-offs between benefits and the between benefits and the

harms harms the estimated size of the effect for each main the estimated size of the effect for each main

outcomeoutcome the precision of these estimatesthe precision of these estimates important factors that could be expected to important factors that could be expected to

modify the size of the expected effects in modify the size of the expected effects in specific settings; e.g. proximity to a hospitalspecific settings; e.g. proximity to a hospital

the relative value attached to the expected the relative value attached to the expected benefits and harmsbenefits and harms

the variation in values between people the variation in values between people

← Option 1 (pink card)

Option 2 → (green card)

You are hiking.You are hiking.

Which of the following animals Which of the following animals would you prefer to would you prefer to

encounter?encounter?

← Option 1 (pink card)

Option 2 → (green card)

You are buying an ice cream.You are buying an ice cream.

Which flavor do you prefer?Which flavor do you prefer?

← Option 1 (pink card)

Option 2 → (green card)

Chocolate

Strawberry

You are buying a new car.You are buying a new car.

Which one would you buy?Which one would you buy?

← Option 1 (pink card)

Option 2 → (green card)

Yellow fox

Red Ferrari

Judgements about the balance Judgements about the balance between benefits and harmsbetween benefits and harms

Before considering cost and making a Before considering cost and making a recommendationrecommendation

Judgements about Judgements about recommendationsrecommendations

1. Benefit and downside evaluation

? ?

2. Recommendation (wording) STRONG WEAK WEAK STRONG Recommend suggest suggest recommend don’t do it probably don’t do it probably do it do it should not do it might not do it might do it should do it

Benefits<< Downsides Benefits ?≤? Downsides Benefits ?≥? Downsides Benefits >> Downsides

Judgements about Judgements about recommendationsrecommendations

• “ “We recommend”…”should” …“Do it”We recommend”…”should” …“Do it”• “ “We suggest”…”may” … “Probably do it” We suggest”…”may” … “Probably do it” • “ “We recommend not”… “may not” We recommend not”… “may not”

…“Probably don’t do it”…“Probably don’t do it”• “ “We suggest not”…”should not”… “Don’t do We suggest not”…”should not”… “Don’t do

it”it”

No recommendationNo recommendation

This could include considerations of costs; i.e. This could include considerations of costs; i.e. “Is the net gain (benefits-downsides) worth “Is the net gain (benefits-downsides) worth the costs?”the costs?”

Should healthy asymptomatic postmenopausal women Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in have been given oestrogen + progestin for prevention in

1992?1992? Quality of evidence across studies forQuality of evidence across studies for

– CHDCHD– Hip fractureHip fracture– Colorectal cancerColorectal cancer– Breast cancerBreast cancer– StrokeStroke– ThrombosisThrombosis– Gall bladder diseaseGall bladder disease

Quality of evidence across critical outcomesQuality of evidence across critical outcomes Balance between benefits and harmsBalance between benefits and harms RecommendationsRecommendations

Will GRADE lead to change?

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and prevention after WHI and

HERSHERS

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and prevention after WHI and

HERSHERS

Further GRADE developmentsFurther GRADE developments

Diagnostic testsDiagnostic tests CostsCosts (Equity)(Equity) Empirical evaluationsEmpirical evaluations Free software applicationFree software application

GRADE Profiler (GRADEpro)GRADE Profiler (GRADEpro)

GRADE profiler (GRADEpro)GRADE profiler (GRADEpro)

GRADE ProfileGRADE Profile

• Excel, HTML, MS Word format

• Linked to REVMAN (direct import from REVMAN)

Footnotes:

1. AUA at 6 mo: Anson 1995; Kabalin 1995; AUA at 12 mo: Sengor 1996; Suvakovic 1996

2. Limitations: concealed allocation to the intervention: yes (Anson), no (Kabalin; Sengor, Suvakovic) subject and outcome assessor unblind (all) Co-interventions: catheter protocol: yes (Kabalin, Sengor, Suvakovic); at physicians discretion: Anson Routine antibiotic use: yes (Kabalin, Suvakovic); at physicians discretion/ unclear (Anson; Sengor) Follow up > 80% (Anson, Kabalin Sengor); not reported (Suvakovic)

3. Quality of evidence: Ambivalence whether to call this low evidence or moderate evidence [strictly speaking, there are limitations in the quality category, as well as the data were sparse, but overall, I would call this moderate quality of evidence]

4. the result is based on only 4 studies, covering a total of 336 patients

5. AUA at 6 mo: Anson 1995; Kabalin 1995; AUA at 12 mo: Sengor 1996; Suvakovic 1996

6. Limitations: concealed allocation to the intervention: yes (Anson), no (Kabalin; Sengor, Suvakovic) subject and outcome assessor unblind (all) Co-interventions: catheter protocol: yes (Kabalin, Sengor, Suvakovic); at physicians discretion: Anson Routine antibiotic use: yes (Kabalin, Suvakovic); at physicians discretion/ unclear (Anson; Sengor) Follow up > 80% (Anson, Kabalin Sengor); not reported (Suvakovic)

7. results based on 336 patients only

8. numbers refer to patients per group

9. Uncler whether this item really applies: a higher score means a better flow: here: TURP achieved a better flow than the laser intervention.

10. 6 Studies: Anson; Costello; Cowles; Donovan; Gujral; Kabalin

11. Limitations: concealed allocation to the intervention: yes (Anson, Cowles, Donovan, Gujral), no (Costello, Kabalin) subject and outcome assessor unblind (all) Co-interventions: catheter protocol: yes (Costello, Kabalin, Sengor); at physicians discretion/ unclear (Anson, Cowles, ); no (Donovan, Gujral) Routine antibiotic use: yes (Costello, Kabalin, Donovan, Gujral); at physicians discretion/ unclear (Anson; Cowles) Follow up > 80% (Anson, Cowles, Donovan, Gujral, Kabalin); < 80% (Costello);

12. would you call this sparse data, too?

13. Increased risk of UTI after laser treatment

14. no separate information provided in the review whether the occurrence of a UTI has been monitored during the total length of the study or during the post-operative period only.

15. Anson 1995; Cowles 1995; Kabalin 1995; Suvakovic 1996 (no event had been reported in two studies: Anson and Kabalin)

16. Limitations: concealed allocation to the intervention: yes (Anson, Cowles), no (Kabalin, Suvakovic) subject and outcome assessor unblind (all) Co-interventions: catheter protocol: yes (Kabalin, Suvakovic); at physicians discretion/ unclear (Anson, Cowles); Routine antibiotic use: yes (Kabalin, Suvakovic); at physicians discretion/ unclear (Anson; Cowles) Follow up > 80% (Anson, Cowles, Kabalin); Availability to follow-up not reported (Suvakovic)

17. Sparse data, based on low number of patients and low number of events

18. Anson 1995; Costello 1995; Kabalin 1995; Sengor 1996

19. Limitations: concealed allocation to the intervention: yes (Anson), no (Costello, Kabalin, Sengor) subject and outcome assessor unblind (all) Co-interventions: Catheter protocol: yes (Costello, Kabalin, Sengor); at physicians discretion/ unclear (Anson); Routine antibiotic use: yes (Costello, Kabalin, Suvakovic); at physicians discretion/ unclear (Anson) Follow up > 80% (Anson, Kabalin, Sengor); follow-up < 80% (Costello)

20. few patients (160 pts.) despite large number of events (69 events)

21. Assessed in 3 studies: Costello 1995; Cowles 1995; Kabalin 1995 but observed only in one study

22. Limitations: concealed allocation to the intervention: yes (Cowles) subject and outcome assessor unblind (all) Co-interventions: catheter protocol: unclear (Cowles) Routine antibiotic use: unclear (Cowles) Follow up > 80% (Cowles)

23. Sparse data: 5 events in 211 patients: Events and numbers seem so sparse that it seems to justify a downgrading by two points to very low evidence (or which of the other categories would apply otherwise?)

24. not unambiguously reported: between 4 and 6 studies (the review did not specifiy the kind of adverse events assessed for 2 studies)

25. number of events is relatively low ==> uncertainty reg. the effect

GRADE Evidence Profile Author(s): Schunemann, Flottorp & Vist Date: 18.06.2005 Question: Should intravenous recombinant tissue plasminogen activator vs no thrombolysis be used for acute ischemic stroke within 3 h? Patient or population: acute stroke and excluded other diagnosis by CT scan Settings: inpatient hospital Systematic review: Wardlaw JM, del Zoppo G, Yamaguchi t, Berge E. Thrombolysis for acute ischemic stroke. The Cochrane Database of systematic reviews. 2003 issue 3

Summary of findings Quality assessment

No of patients Effect

No of studies

Design Limitations Consistency Directness Other

considerations

intravenous recombinant tissue plasminogen activator

no thrombolysis

Relative (95% CI)

Absolute (95% CI)

Quality Importance

Death or dependency at the end of follow up (Barthel Index and modified Rankin score 2-6, where 6 = dead and 0-1 no event Follow up: 3 months)

5 Randomised trials

Serious limitations (-1)1

No important inconsistency

No uncertainty

None 232/465 (49,9%)

280/465 (60,2%)

OR 0.64 (0.50 to 0.83)

103/1 000 ( to )

Moderate

9

Death from all causes during follow-up ( Follow up: 3 months)

6 Randomised trials

Serious limitations (-1)1

No important inconsistency

No uncertainty

None 83/479 (17,3%)

83/478 (17,4%)

OR 0.97 (0.69 to 1.36)

1/1 000 ( to )

Moderate

9

Symptomatic intracraneal haemorrhage (repeat CT scan Follow up: 3 months)

4 Randomised trials

Serious limitations (-1)1

No important inconsistency

No uncertainty

Strong association (+1)2

20/153 (13,1%)

5/153 (3,3%)

OR 3.4 (1.48 to 7.84)

98/1 000 ( to )

High

8

Footnotes:

1. Randomisation uncussessful in the largest study + no intention to treat, lack of information about losses to follow up

2. Strong association with OR 3.4

Comparison of GRADE and other systemsComparison of GRADE and other systems

• Explicit definitionsExplicit definitions• Explicit, sequential judgementsExplicit, sequential judgements• Components of qualityComponents of quality• Overall qualityOverall quality• Relative importance of outcomesRelative importance of outcomes• Balance between health benefits and harmsBalance between health benefits and harms• Balance between incremental health benefits Balance between incremental health benefits

and costsand costs• Consideration of equityConsideration of equity• Evidence profilesEvidence profiles• International collaborationInternational collaboration• SoftwareSoftware• Consistent judgements?Consistent judgements?• Communication?Communication?

Who is interested in GRADEWho is interested in GRADE• WHOWHO• American Endocrine SocietyAmerican Endocrine Society• American College of Chest Physicians (ACCP)American College of Chest Physicians (ACCP)• Italian National Cancer Institute, RomeItalian National Cancer Institute, Rome• Clinical EvidenceClinical Evidence• Norwegian Centre for Health ServicesNorwegian Centre for Health Services• UpToDateUpToDate• Close relationship with Cochrane Close relationship with Cochrane

CollaborationCollaboration• American Society of Clinical Oncology (ASCO)American Society of Clinical Oncology (ASCO)• American Thoracic Society (ATS)American Thoracic Society (ATS)• Urologists worldwideUrologists worldwide

Empirical evaluationsEmpirical evaluations

• Critical appraisal of other systemsCritical appraisal of other systems• Pilot test + sensibilityPilot test + sensibility• ““Case law” + practical experienceCase law” + practical experience• Guidance for judgementsGuidance for judgements

• Single studiesSingle studies• Sparse data or imprecise dataSparse data or imprecise data

• AgreementAgreement• Validity?Validity?• Comparisons with other systemsComparisons with other systems• Alternative presentationsAlternative presentations