Standard wording for formulating evidence

conclusions and implications for recommendations

Rode Kruis

Emmy De Buck

Manager Centre for Evidence-Based Practice

Belgian Red Cross-Flanders

I certify that, to the best of my knowledge, no aspect of my current personal or professional situation might reasonably be expected to affect significantly my views on the subject on which I am presenting, other than the following:

Financial relationships:

Employment at the Centre for Evidence-Based Practice of Belgian Red Cross-Flanders

Disclosure of interests

Centre for Evidence-Based Practice (CEBaP) of Belgian Red Cross-Flanders:

provides evidence-based support to all programmes and activities of Belgian Red Cross-Flanders, by developing:

− Practice guidelines

− Systematic reviews

Staff: 4 methodologists, PhD’s

Introduction

In our centre several methodologists (reviewers) work in parallel to develop evidence reviews in a timely way

The evidence conclusions go from the methodologists to other collaborators (colleagues of operational services, experts)

Important to standardize the evidence synthesis and to provide standard wording of the evidence conclusions/statements

Rationale

Information from guideline developers:

− NICE1: “A short evidence statement should be presented alongside the evidence profile, summarising the key features of the evidence on clinical and cost effectiveness”

− SIGN2: no narrative evidence conclusions

− Finnish Medical Society Duodecim3: wording varies depending on the level of evidence

Wording of evidence conclusions (1)

1 http://publications.nice.org.uk/the-guidelines- manual-pmg6/reviewing-the-evidence2 http://www.sign.ac.uk/methodology/index.html3http://2011.colloquium.cochrane.org/sites/2011.colloquium.cochrane.org/files/uploads/users/ u721/Wording%20of%20evidence%20and%20recommendations%20in%20EBM%20Guidelines.doc

Information from systematic review developers:

− Wide variation among authors in the Cochrane Library in reporting results and conclusions1

− The Cochrane Handbook, Chapter 9, Section 9.12:• “A narrative assessment of the evidence can be challenging...”

• “If a descriptive paragraph is provided for the results from each study, this should be done consistently, including the same elements of information for each study.”

• “Organizing the studies into groupings or clusters is encouraged (e.g. by intervention type, population groups, setting etc).”

Conclusion: almost no guidance about wording of evidence conclusions

Wording of evidence conclusions (2)

1 Van Tulder et al. 2003. SPINE 28(12): 290–12992 http://handbook.cochrane.org/chapter_9/9_1_introduction.htm

Target population for reading the evidence conclusions:

− colleagues who write draft recommendations− experts who validate the draft recommendations

Evidence conclusions should contain:

− number and type of studies− intervention, comparison− outcome− level of evidence− direction of effect

Criteria for evidence conclusions (1)

Wording of evidence conclusions should reflect:

− if results were statistically significant or not

− the quality of the evidence

− the difference between “evidence of no effect” and “no evidence of effect” 1

Evidence conclusions should be organized according to the intervention, outcome,…

Categories of evidence conclusions:− Evidence and limited evidence

− No evidence

− Conflicting evidence

Criteria for evidence conclusions (2)

1 Altman and Bland 1995. BMJ 311: 485

Y

Is there any evidence available?

Is there any imprecision?

N

Is the evidence conflicting?

N

Is the body of evidence of low or very low quality?

“No evidence”

Y

Y

N

“Conflicting evidence”

“Limited evidence”

Y “Limited evidence”

N

“Evidence”

Is the intervention considered as common sense?

Y

N

START

Is it animportant practical point

of which the expert panel reaches a consensus and nobody

is likely to question it?

Y

N

Is the balance made by the expert panel positive?

YRecommendation

N No Recommendation

Is the balance made by the expert panel positive?

Y

N

Is the balance made by the expert panel positive?

Y

N

Good Practice Point

Literature review Consensusmeeting with experts

Recommendation

Recommendation

No Recommendation

No Recommendation

No Good Practice Point

Flowchart

Category of evidence Wording of evidence conclusion

No evidence No relevant studies were identified using the above search strategy and criteria.

Conflicting evidence There is conflicting evidence from # experimental studies and/or # observational studies…

(Limited) evidence Wording depends on:

-statistical significance

-level of evidence (LOE)

-imprecision

See extended table

Standard wording

P-value LOE Imprecision Wording

< 0.05 A/B No There is evidence from # experimental studies and/or # observational studies in favour of [intervention] (<Author> <year>, <Author> <year>, etc.).

It was shown that <intervention> resulted in a statistically significant increase/decrease of <outcome>, compared to <comparison> (<Author> <year>).

Evidence is of high/moderate quality.

Yes There is limited evidence ... It was shown that ...

Evidence is of moderate quality and results cannot be considered precise due to limited sample size, lack of data and/or large variability of results.

C/D No There is limited evidence ... It was shown that ...Evidence is of low/very low quality.

Yes There is limited evidence ... It was shown that ...

Evidence is of low/very low quality and results cannot be considered precise due to ...

Standard wording

P-value LOE Imprecision Wording

> 0.05 A/B No There is evidence from # experimental studies and/or # observational studies, neither in favour of the intervention nor the control (<Author> <year>, <Author> <year>, etc.).It was shown that <intervention> did not result in a statistically significant difference of <outcome>, compared to <comparison> (<Author> <year>). Evidence is of high/moderate quality.

Yes There is limited evidence ...A statistically significant increase/decrease of <outcome>, using <intervention> compared to <comparison>, could not be demonstrated (<Author> <year>). Evidence is of moderate quality and results of this study/these studies are imprecise due to limited sample size, lack of data and/or large variability of results.

C/D No There is limited evidence ... It was shown that <intervention> did not result ...Evidence is of low/very low quality.

Yes There is limited evidence ... A statistically significant increase/decrease ... could not be demonstrated ...Evidence is of low/very low quality and results of this study/these studies are imprecise due to ...

Only in case of p<0.05, LOE A/B and no imprecision, a draft recommendation for or against the intervention should be included in the guideline

In the other cases: a draft recommendation may be included in the guideline

The final recommendation will be based on the judgement of the multidisciplinary expert panel, taking into account:

− the scientific evidence

− preferences of the target group

− expertise and practical experience

Implications for recommendations

PICO: Is rice water (I) effective compared to standard Oral Rehydration Solution (ORS) (C) to improve diarrhoea (O) in adults or children with diarrhoea (P)?

Included studies: 2 RCTs: Metha 1986, Fakhir 1990

Example

High [A] Downgrading due to

Limitations of study design

-1 Lack of blinding/blinding unclear; lack of allocation concealment in 1 study

Imprecision -1 Limited sample sizeInconsistency 0Indirectness 0Publication bias 0QUALITY (GRADE) Low [C]

Example

Outcome Comparison Effect Size #studies # parti-cipants

Ref

Stool frequency between 1 and 3 per day

Rice water versus glucose ORS

Day 1: Not statistically significant:4/100 vs 0/100; RR: 9.00, 95% CI [0.49, 165.00] (p=0.14)Day 2: Statistically significant:16/100 vs 6/100; RR: 2.67, 95% CI [1.09, 6.54] (p=0.03)Day 3: Statistically significant:46/100 vs 27/100; RR: 1.70, 95% CI [1.16, 2.51] (p=0.007)Day 4: Statistically significant:68/100 vs 42/100; RR: 1.62, 95% CI [1.24, 2.11] (p=0.0004)

1, 100 vs 100

Metha, 1986

Small stool volume Day 1: Not statistically significant:2/100 vs 0/100; RR: 5.00, 95% CI [0.24, 102.85] (p=0.30)Day 2: Not statistically significant:40/100 vs 30/100; RR: 1.33, 95% CI [0.91, 1.96] (p=0.14)Day 3: Not statistically significant:56/100 vs 42/100; RR: 1.33, 95% CI [1.00, 1.78] (p=0.05)Day 4: Statistically significant:76/100 vs 44/100; RR: 1.73, 95% CI [1.35, 2.21] (p<0.0001)

Duration of purging in the hospital (hours)

Statistically significant:60.2±2.6 vs 78.6±4.6MD: -18.40, 95%CI ; [-20.60;-16.20] (p=0.00001)

1, 20 vs 23

Fakhir, 1990

P-value LOE Imprecision Wording

< 0.05 A/B No There is evidence from # experimental studies and/or # observational studies in favour of [intervention] (<Author> <year>, <Author> <year>, etc.).

It was shown that <intervention> resulted in a statistically significant increase/decrease of <outcome>, compared to <comparison> (<Author> <year>).

Evidence is of high/moderate quality.

Yes There is limited evidence ... It was shown that ...

Evidence is of moderate quality and results cannot be considered precise due to limited sample size, lack of data and/or large variability of results.

C/D No There is limited evidence ... It was shown that ...Evidence is of low/very low quality.

Yes There is limited evidence ... It was shown that ...

Evidence is of low/very low quality and results cannot be considered precise due to ...

Example

PICO: Is rice water (I) effective compared to standard Oral Rehydration Solution (ORS) (C) to improve diarrhoea (O) in adults or children with diarrhoea (P)?

Evidence conclusion:

There is limited evidence from 2 experimental studies in favour of rice water (Metha 1986, Fakhir 1990).

It was shown that rice water resulted in a statistically significant decrease of duration of purging, stool frequency on day 2 to 4 and stool volume on day 4 compared to using standard ORS.

Evidence is of low quality and results cannot be considered precise due to a limited sample size.

Example

The use of standard wording in our evidence conclusions resulted in:

− standardized training of new staff members

− guidelines with uniform evidence statements

− a better interpretation of the available evidence

− more transparency in our methodology

Recommendation for guideline developers:

− use standard wording of evidence conclusions, in order to improve consistency within one guideline and between guidelines

Take home message

Thanks to my colleagues Tessa, Nele and Hans for the fruitful discussions concerning this subject.

Acknowledgements

Thank you for your attention ! Questions ?

Contact: Cebap@rodekruis.be Emmy.Debuck@rodekruis.be

Centre for Evidence-Based Practice