GRADE, Summary of Findings and ConQual Workshop · • Moving towards GRADE • Summary of Findings tables ... 4. JBI adopts an approach based on GRADE but sensitive to the ... Measurement

GRADE, Summary of Findings and ConQual Workshop

To discuss…

• Introduction

• New JBI Levels of Evidence and Grades of Recommendation

• Moving towards GRADE

• Summary of Findings tables

• Qualitative Levels

• Conclusion

University of Adelaide 2

JBI Methodology Groups

• Formed to provide guidance on emerging methods of evidence synthesis

• Mixed methods, GRADE, qualitative, umbrella, economic, effects, prevalence, correlation

• Group formed to review JBI Levels of Evidence


Levels of Evidence

• Designate study type

• Better study designs, with greater methodological quality, are ranked higher

• Assist in applying research into practice

• Recommendations assigned a grade

Grades of Recommendation

History




Old levels of evidence


Old Grades of recommendation


How are they used?


Justification for Change • Clarity (not specific)

• Quasi-experimental studies

• Not all types of evidence

• No supporting document

• Experimental studies

• Adopt GRADE

• Difficulties creating recommendations

• Don’t say ‘systematic review’


Working Party Recommendations 1. JBI adopts the GRADE approach to summary of findings tables

for all reviews addressing questions of effect. 2. JBI adopts a modified GRADE approach, based on FAME, for

forming recommendations for practice. 3. New levels of evidence under the following headings:

Intervention/Therapy/ Harms, Diagnostic Accuracy, Prognosis, Economic Analysis, Qualitative research.

4. JBI adopts an approach based on GRADE but sensitive to the nature of qualitative research for qualitative systematic reviews.

5. JBI contact the GRADE working group to discuss GRADE for the use of JBI, and to offer a partnership for developing GRADE for qualitative research.


Why GRADE?

• Grading of Recommendations Assessment, Development and Evaluation (GRADE)

• International working group

• Endorsed by many EBHC organisations (WHO, Cochrane, SIGN, etc)


Kerwin et al. 2012

Why GRADE?


Forming recommendations with GRADE


Balance between benefits, harms

and burdens

Resource use Patients

values and preferences

Quality of Evidence

How do we determine quality of the evidence?

Example meta-analysis discussion

From the examples provided, what information would increase or decrease your confidence in these results?


Discussion results

• Decrease

• Increase


GRADE

• Decrease

– Methodological quality (risk of bias)

– Indirectness (i.e applicability, generalisability, transferability etc)

– Inconsistency (heterogeneity)

– Imprecision (uncertainty)

– Publication bias

• Increase

– Large, consistent, precise effect

– All plausible biases underestimate the effect

– Dose response effect


Quality of evidence: beyond risk of bias Definition: The extent to which our confidence in an estimate of the treatment effect is adequate to support a particular recommendation

Methodological limitations

Inconsistency of results

Indirectness of evidence

Imprecision of results

Publication bias

Risk of bias: Allocation

concealment Blinding Intention-to-treat Follow-up Stopped early

Sources of indirectness:

Indirect

comparisons Patients Interventions Comparators Outcomes

19

GRADEing the evidence • Pre-ranking

– RCTs start as high, Observational studies as low

• Quality of evidence ranges from

HIGH

MODERATE

LOW

VERY LOW

• RCTs start with high quality rating • Can be downgraded 1 or 2 points for each area of concern • Maximum downgrade of 3 points overall

Confidence

GRADE Quality of Evidence

In the context of making recommendations:

• The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation

Likelihood of and confidence in an outcome

Interpretation of grades of evidence

• /A/High: Further research is very unlikely to change confidence in the estimate of effect.

• /B/Moderate: Further research is likely to have an important impact on confidence in the estimate of effect and may change the estimate.

• /C/Low: Further research is very likely to have an important impact on confidence in the estimate of effect and is likely to change the estimate.

• /D/Very low: We have very little confidence in the effect estimate: Any estimate of effect is very uncertain.

Summary of Findings tables

• Standard table format • one for each comparison

• Focus on outcomes

• Includes: • context

• results

• GRADE

• reasons behind decisions

Summary of findings table

• Improve understanding

• Improve accessibility

• Created with GRADEpro http://ims.cochrane.org/revman/gradepro

http://ims.cochrane.org/revman/gradepro

Presented in a summary of findings table




Forming recommendations with GRADE

• Two recommendations

– Strong and Weak

– For or against


Balance between benefits, harms

and burdens

Resource use Patients

values and preferences

Quality of Evidence

Goldet et al. 2013


Systematic review

Guideline development

P I C O

Outcome

Outcome

Outcome

Outcome

Critical

Important

Critical

Low Summary of findings & estimate of effect for each outcome

Rate overall quality of evidence across outcomes based on

lowest quality of critical outcomes

RCT start high, obs. data start low

1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication

bias

Gra

de

d

ow

n

Gra

de

u

p

1. Large effect 2. Dose

response 3. Confounders

Very low Low Moderate High

Formulate recommendations: • For or against (direction) • Strong or weak (strength)

By considering: Quality of evidence Balance benefits/harms Values and preferences

Revise if necessary by considering:

Resource use (cost)

• “We recommend using…” • “We suggest using…” • “We recommend against using…” • “We suggest against using…”

Why still have levels?

• Other JBI resources (not just systematic reviews)

• Assist in pre-ranking

• Address evidence of other types


JBI Levels of Evidence

OLD LEVELS OF EVIDENCE

• Feasibility

• Appropriateness

• Meaningfulness

• Effectiveness

NEW PROPOSED LEVELS OF EVIDENCE

• Therapy, harm

• Diagnostic/screening studies

• Prognosis

• Economic evaluations

• Meaningfulness



Levels of evidence: Diagnostic/screening studies

Level 1 – Studies of Test Accuracy among consecutive patients • Level 1.a – Systematic reviews of studies of test accuracy among consecutive patients • Level 1.b – Studies of test accuracy among consecutive patients Level 2 – Studies of Test Accuracy among non-consecutive patients • Level 2.a – Systematic reviews of studies of test accuracy among non-consecutive patients • Level 2.b – Studies of test accuracy among non-consecutive patients Level 3 – Diagnostic Case control studies • Level 3.a – Systematic reviews of diagnostic case control studies • Level 3.b – Diagnostic case-control study Level 4 – Diagnostic yield studies • Level 4.a – Systematic reviews of diagnostic yield studies • Level 4.b – Individual diagnostic yield study Level 5 – Expert Opinion and Bench Research • Level 5.a – Systematic reviews of expert opinion • Level 5.b – Expert consensus • Level 5.c – Bench research/ single expert opinion


Levels of evidence: Prognosis

Level 1 – Inception Cohort Studies • Level 1.a – Systematic reviews of inception cohort studies • Level 1.b – Inception cohort studies Level 2 – Studies of All or none • Level 2.a – Systematic reviews of studies of all or none studies • Level 2.b – All or none studies Level 3 – Cohort studies • Level 3.a – Systematic reviews of cohort studies (or control arm of RCT) • Level 3.b – Cohort studies (or control arm of RCT) Level 4 – Case series/Case Controlled/ Historically Controlled studies • Level 4.a – Systematic reviews of Case series/Case Controlled/ Historically Controlled studies • Level 4.b – Individual Case series/Case Controlled/ Historically Controlled studies Level 5 – Expert Opinion and Bench Research • Level 5.a – Systematic reviews of expert opinion • Level 5.b – Expert consensus • Level 5.c – Bench research/ single expert opinion


Levels of evidence: Economic evaluations

• Level 1: Systematic review of economic evaluations

• Level 2: Single economic evaluation

• Level 3: Systematic review of expert opinion

• Level 4: Expert opinion


Levels of evidence: Meaningfulness

1. Qualitative or mixed-methods systematic review

2. Qualitative or mixed-methods synthesis

3. Single qualitative study

4. Systematic review of expert opinion

5. Expert opinion


Summary of findings table: Meaningfulness

• Incorporating a ‘GRADE’ like rating of quality for studies related to meaningfulness – can it be done?

• Need to consider what increases or decrease confidence in the results


Discussion Activity

EXAMPLE META-SYNTHESIS

From the examples provided, what information would increase or decrease your confidence in these results?


Synthesised finding 1

Munn & Jordan. The patient experience of high technology medical imaging: a systematic review of the qualitative evidence, JBI Library of

Systematic Reviews, 2011; 9(19): 631-678

People undergoing imaging often expect a health issue to be found during their

scan, which can then lead to anxiety and worry.

Synthesised finding 2

Adjust themselves: Caregivers need to adjust themselves to the caregiving role when living with and

taking care of persons with schizophrenia. The ultimate goal is to integrate the caregiving role into their

lives.

Tungpunkom, Napa, Chaniang & Srikhachin. Caregiving experiences of families living with persons with schizophrenia: a systematic review,

2013; 11(8): 415-564

Discussion Activity


What information will increase or decrease your confidence in the

results?


Discussion Activity (some answers)

• Type of data

• Dependability

• Confidence


What information will increase or decrease your confidence in the

results?


Meaningfulness Summary of Findings Table


Systematic review title:

Population:

Phenomena of interest:

Context:

Synthesised

Finding

Type of data Dependability Confidence Score Comments

Type of data

• Ranking scale consists of 4 levels

– High

– Moderate

– Low

– Very Low

• Begin by pre-ranking papers based on type of data

– High for qualitative studies

– Low for expert opinion


Score of dependability: qualitative research

Measurement Measured by asking the following questions:

1. Is there congruity between the research methodology and the research question or objectives?

2. Is there congruity between the research methodology and the methods used to collect data?

3. Is there congruity between the research methodology and the representation and analysis of data?

4. Is there a statement locating the researcher culturally or theoretically?

5. Is the influence of the researcher on the research, and vice-versa, addressed?

Ranking system: 4-5 ‘yes’ responses, the finding remains unchanged

2-3 ‘yes’ responses: move down 1 level

0-1 ‘yes’ responses: move down 2 levels


Score of dependability: expert opinion

Measured by asking questions related to the appropriateness of the conduct of the research with research aims and purpose:

1. Does the source of the opinion have standing in the field of expertise?

2. Is the opinion’s basis in logic/experience clearly argued?

3. Is the argument developed analytical? 4. Is there reference to the extant

literature/evidence and any incongruence with it logically defended?

5. Is the opinion supported by peers?


Ranking system:

4-5 ‘yes’ responses, the paper remains unchanged

(0)

2-3 ‘yes’ responses: move down 1 level (-1)

0-1 ‘yes’ responses: move down 2

levels (-2)

Score of confidence

Measurement Assign a level of credibility to the findings:

Unequivocal (findings accompanied by an illustration that is beyond reasonable doubt and; therefore not open to challenge) Equivocal (findings accompanied by an illustration lacking clear association with it and therefore open to challenge) Unsupported (findings are not supported by the data, or with no illustration)


Ranking The synthesised findings contains only unequivocal findings Remains unchanged Mix of unequivocal/equivocal findings downgraded one (-1) All equivocal finding down grade 2 (-2)

Mix of plausible/unsupported findings downgraded three (-3)

Not-supported findings downgraded four (-4)

Meaningfulness Summary of Findings Table


Systematic review title: The patient experience of high technology medical imaging: a systematic review of the

qualitative evidence

Population: Persons who had undergone high technology medical imaging

Phenomena of interest: The meaningfulness of a patients experience of undergoing diagnostic imaging using

high technology

Context: Male and Female Adult Patients presenting to a medical imaging department

Synthesised

Finding

Type of

research

Dependability Confidence Score Comments

People undergoing imaging often

expect a health issue to be found during their scan, which can then

lead to anxiety and worry

Qualitative

(HIGH)

(MODERATE)

(LOW)

LOW

A new spin on FAME


The FAME (Feasibility, Appropriateness, Meaningfulness and Effectiveness) scale informs the recommendation. F – Feasibility; specifically: What is the cost effectiveness of the practice? Is the resource/practice available? Is their sufficient experience/levels of competency available? A – Appropriateness; specifically: Is it culturally acceptable? Is it transferable to the majority of the population? Is it easily adaptable to a variety of circumstances? M – Meaningfulness; specifically: Is it associated with positive experiences? Is it not associated with negative experiences? E – Effectiveness; specifically: Was there a beneficial effect? Is it safe? (i.e is there a lack of harm associated with the practice?

JBI Grades of Recommendation


JBI Grades of Recommendation

Grade A A ‘strong’ recommendation for a certain health management strategy where: • it is clear that desirable effects outweigh undesirable effects of the

strategy; • where there is evidence of adequate quality supporting its use; • there is a benefit or no impact on resource use, and • values, preferences and the patient experience have been taken into

account

Grade B A ‘weak’ recommendation for a certain health management strategy where: • desirable effects appear to outweigh undesirable effects of the

strategy, although this is not as clear; • where there is evidence supporting its use, although this may not be

of high quality; • there is a benefit, no impact or minimal impact on resource use, and • values, preferences and the patient experience may or may not have

been taken into account.

Recommendations

People undergoing scanning are aware that is has the power to detect significant health issues, which can lead to anxious waits

and uncertainty after imaging. Once imaging is completed, patients should be given information regarding when they will

receive their results, and what will happen now the images have been taken

(Grade B)

Munn & Jordan. The patient experience of high technology medical imaging: a systematic review of the qualitative evidence, JBI Library of

Systematic Reviews, 2011; 9(19): 631-678


Conclusions

• Improved clarity in levels of evidence

• In line with international organisations

• Changes made to fit with JBI’s broader view of what constitutes evidence

• Guidance provided


Documents

GRADE, Summary of Findings and ConQual Workshop · • Moving towards GRADE • Summary of Findings tables ... 4. JBI adopts an approach based on GRADE but sensitive to the ... Measurement