Guidelines International Network North America Webinar ... · Guidelines International Network...
21
Guidelines International Network North America Webinar Series November 19, 2013 Sandra Zelman Lewis, PhD EBQ Consulting
Guidelines International Network North America Webinar ... · Guidelines International Network North America . Webinar Series . November 19, 2013 . Sandra Zelman Lewis, PhD EBQ Consulting
Guidelines International Network North America Webinar Series
November 19, 2013
Sandra Zelman Lewis, PhD EBQ Consulting
Disclosures I receive no reimbursement from industry. I formerly oversaw development of all ACCP
guidelines and consensus statements. This presentation builds upon a process I helped
to create at the ACCP.
Poll 1: In what type of organization do you currently develop guidelines?
a. Medical specialty/professional organization b. Healthcare delivery system c. Academic institution d. Government organization e. Consulting organization f. Other
Objectives and Outline of this Webinar
How do consensus statements (CSs) fit into the realm of EBM?
Why do we need CSs? How can we make CSs reliable and trustworthy? What are the important development steps and how do
they differ from those of EBGs? Important decision points Can we combine these EB and CB processes? Anticipating common pitfalls
Definition of terms
• Guidelines (EBGs) are evidence-based guidelines • Consensus statements same as consensus-based
(CB) statements or CB documents • Either one: guidance document • Evidence-based guidelines have recommendations • Consensus-based statements have suggestions
Presenter
Presentation Notes
Now let’s define our terms before we go forward: When I use the word “guidelines,” I will be referring to methodologically rigorous evidence-based guidelines. Consensus statements can also be called consensus-based statements or consensus-based documents Either are a form of guidance document, as they provide clinical guidance. Evidence-based guidelines have recommendations Consensus-based statements have suggestions
Poll 2: Until now, have you ever developed or overseen the development of consensus-based statements?
a. I have never developed any consensus statements b. I have developed 1 or 2 consensus statements c. I am experienced at developing consensus statements
CDS: Clinical Decision Support CME/CE: Continuing Medical Education KT: Knowledge Transfer QI: Quality Improvement SRs/HTAs: Systematic Reviews, Health Technology Assessments
Presenter
Presentation Notes
The realm of EBM includes 2 pathways: One for evidence-based guidelines and one for consensus-based guidance documents. The former are developed by funneling the scientific evidence through a synthesis process into systematic reviews and health technology assessments which, in turn, inform the evidence-based recommendations. Consensus-based statements include minimal evidence, if any, so the stakeholder expertise plays a greater role but must be agreed upon by the panel as a whole. Achieving that agreement should follow formal methods for consensus achievement. Patient values and preferences, along with recognition of resource barriers, must also be considered in developing the final recommendations and suggestions. Guidelines potentially impact clinical decision support, medical education, healthcare policy, and quality improvement. The consensus-based statements should be restricted to clinical decision support and medical education. Either way, guidance documents employ dissemination, knowledge transfer and uptake, and implementation strategies. Clinical Decision Support: E-decision support tools EMRs/e-reminders Checklists, risk assessment tools, clinical pathways, etc Medical Education Accredited CME/non-CME Simulation/VR Other education Quality Improvement PMs PIMs Other PI projects Healthcare Policy Reimbursement regulations Facilities planning/� approvals
When Evidence Is Weak or Nonexistent Topics with low or poor quality evidence are
common. Quality guideline standards require a minimal
threshold of evidence. Consensus statements (CSs) - at least as important
as EBGs: they fill an unmet need But until now...
• There was no standardization or rigor. • Users unclear whether to trust the products. • Skepticism of both the process and final
statements, resulting in prejudicial bias about CSs.
Presenter
Presentation Notes
What can you do when Evidence Is Weak or Nonexistent? This is the important question that we must address. ■In evidence-based guideline (EBG) development, topics with low or poor quality evidence are common. ■Quality guideline standards require a minimal threshold of evidence, compelling these topics to be handled outside of an EBG process. A common standard is that the research must be published in peer-reviewed publications. ■Consensus statements (CSs) could be at least as important as EBGs since they fill an unmet need. ■But until now... There was no standardization or rigor in the development process. Users had no way to know whether they could trust the final products. There was a tendency to be skeptical of both the process and the final statements, resulting in a prejudicial bias about CSs, in general.
So how can we develop consensus-based statements
that are trustworthy ?
Trustworthy Consensus Statements Five Key Processes to Earn User Trust: 1. Panel structure and panelist approval criteria 2. Comprehensive and systematic literature search 3. Delphi-based consensus achievement process 4. Transparency of process 5. Thorough reviews
Confidence in Panel Composition and Screening
Formal Consensus Achievement
Systematic Searching for Evidence
Rigorous Review Process
Building trustworthiness
Transparency
Presenter
Presentation Notes
There are 5 key processes that must be followed to earn or build the user’s or reader’s trust. [Advance slide] For the first step toward building trust in your guidance document, you must first establish the reader’s confidence in your panel. This is the critical foundation without which the other building blocks will collapse. [Advance slide] Then conduct a comprehensive and systematic PICO-based search for any evidence that might exist. [Advance slide] Use what you find, as well as the expertise of your trusted panelists to develop guidance employing a formal process to achieve panel consensus. [Advance slide] Be transparent about your methods. Provide thorough explanations for the readers and users of the guidance documents. [Advance slide] And finally top off those building blocks with a rigorous review process, equivalent to or even more carefully than you would for an EBG.
Trustworthy Consensus Statements
1. Panel Approval Process • Structure panel with the right stakeholders • Reviewed and approved for qualifications • Important to have higher standards for avoidance of
COIs o No COIs for Chair
Presenter
Presentation Notes
The panel approval process is the foundation for trust in the rest of the process and the final product. It is important that you structure the panel with all of the right stakeholders (all perspectives should be present, including consumers and frontline clinicians) Nominees should be carefully reviewed and approved for qualifications and conflicts of interest Important to have standards for avoidance of COIs, at least as high as standards for EBG panels And the Chair(s) should have no relevant conflicts
Trustworthy Consensus Statements
2. Literature Review Process • Key topics/PICO questions refined • Search strategies based on PICO questions • Systematic searches of at least two databases • Predefined inclusion/exclusion criteria applied to
screen studies • Evidence tables constructed (if applicable) • Quality appraisal of each included study
Presenter
Presentation Notes
The premise of the Literature Review Process is that you will carefully search to ensure whether or not there is evidence relevant to the key clinical questions posed. If so, they will be fully vetted and potentially you will not be creating a consensus statement but an evidence-based guideline. However, if evidence is not found, you have created the basis to justify a consensus-based approach. So you begin by following the steps in an evidence review as if you were creating a guideline. So first the key clinical topics or questions should be chosen and PICO elements defined Develop search strategies based on PICO elements Systematically search for any available evidence, based on the PICO question, in at least 2 databases Apply predefined inclusion and exclusion criteria to screen the studies, if any are netted in the search. Evidence tables should be constructed to include any studies that pass the screenings. And each should be appraised for study quality
Trustworthy Consensus Statements
2. Literature Review Process (con’t) A priori decision: • Threshold for CS/EBG decision
ACCP minimal criteria for sufficient evidence: • 2 RCTs OR 2 observational studies • Quantitatively address comparable outcomes • Must have a quality rating ≥ fair • Review for approval for EBG
• If insufficient evidence, justification for consensus-based approach
Presenter
Presentation Notes
The threshold for defining the difference in evidence between a CS and an EBG should be decided in advance. At ACCP, we set the minimal criteria for sufficient evidence for a guideline at: At least 2 RCTs OR 2 observational studies Must quantitatively address comparable outcomes Must have a quality rating of fair or greater This permits pooling of the data for meta-analyses. The final decision to elevate a CS to an EBG would be made based on a review of an evidence table with standard quality assessments for each study. It is recognized that one exceptional or landmark study of very high quality might accommodate the EBG approach, but only with careful review and approval. If there is insufficient evidence for a guideline, then that provides justification for a consensus-based approach.
Trustworthy Consensus Statements
3. Consensus Development Process • Panel consensus achieved using a Delphi technique • 1-3 rounds • Online survey of suggestions • 5-point scale derived from GRADE grid • Optional open field for comments/feedback
• Response rate of ≥75% of panel required per item • First round nonresponders removed from panel
• Consensus achieved with ≥80% of respondents voting either “strong support” or “weak support”
• If minimum response rate not achieved, then revise or drop item
Presenter
Presentation Notes
The Consensus Development Process must be properly conducted to ensure user confidence. Panel consensus should be achieved using a modified Delphi technique Up to 3 rounds May be conducted online to reduce costs of in-person meetings Use a 5-point scale derived from the GRADE grid (which we will see in the next slide) There should be an optional open field for comments/feedback to facilitate potential revisions to the suggestions For each item, a response rate of at least 75% of panel is required It is important to encourage all panelists to participate in the voting to help achieve the minimal 75% response rate. So they should be informed that panelists not voting in the first round, will not be invited to vote subsequently, and will be removed from the panel and authoring. Consensus is achieved with ≥80% of respondents voting in support of the suggestions, either “strong support” or “weak support” If the minimum consensus level (80%) is not achieved, then revise or drop the item.
Trustworthy Consensus Statements
3. Consensus Development Process (con’t.) GRADE Grid Derivative
1 2 3 4 5 Strong: “definitely do it”
Weak: “probably do it”
No specific recommendation
Weak: “probably don’t do it”
Strong: “definitely don’t do it”
≥80% = consensus
Open field for comments
Presenter
Presentation Notes
This is the modified GRADE Grid. It shows the 5-point agreement scale for which one would need 80% of responders to vote 1 or 2 in order to achieve consensus. The open field is for respondents to write why they could not agree with the suggestions. This information is used in advance of subsequent rounds to modify or revise the suggestions.
Trustworthy Consensus Statements
4. Transparency/Grading • Consensus-based suggestions should not carry
grades • Clearly labeled as consensus based (CB) • No evidence grade • Benefits and harms and the balance of these -
subject to variation – lack of confidence in magnitude of effect
Presenter
Presentation Notes
In drafting the suggestions, there must be transparency of process. Consensus-based suggestions should not carry grades But should be clearly labeled as consensus based (CB) There will be no evidence grade since it is not based on evidence, or not much. Authors can discuss benefits and harms and the balance of these in the text, but not attach a grade. This is subject to future variation as research proceeds and evidence evolves. Confidence in the magnitude of the effect is still to come. At this point, there is not enough confidence to assign a strength to the suggestion. But all suggestions should be marked to indicate that they were based on the consensus of the panel.
Trustworthy Consensus Statements
5. Rigorous Review Process • Executive Committee of panel • Full panel • Invited associations representatives • Non-panelist SME’s - content review • Methodology or guideline committee – methods
review • Board members, if applicable or provide oversight • External organization reviews or public reviews, if
permitted • Journal peer review
Presenter
Presentation Notes
Now that you have laid a strong foundation for trustworthiness and added the powerful building blocks of a systematic literature search, formal consensus achievement, and transparency of methods, a thoughtful and multilayered review process will provide the capstone of reliability. First the chair and other leadership of the panel should provide the first reviews. Once the leadership and full panel signs off on the document, which includes any invited experts or representatives of other disciplines, the review process commences beyond the panel. There should be content reviews provided by subject matter experts who were not participating on the panel and methodology reviews by methods experts. These latter individuals might be a committee that oversees the development process or other methodologists requested to assist at this point. Medical societies need to have their boards sign off on the final version but other guideline developers might also require a final administration or leadership approval. Some developers are permitted to send their guidance documents out for external reviews, possibly for endorsement, with or without changes. And some are permitted to post for public comments and feedback. Finally, the journal or publisher will separately employ peer reviewers.
Poll 3: Do you believe that employing these measures will lead to consensus statements that can be trusted?
a. Yes b. No
Trustworthy Hybrid Documents
Hybrid documents
Combination of both EBGs and CSs
PICO-specific
Provide explanation to readers
Big advantage – provide trustworthy clinical guidance in all topic areas
Important: BE TRANSPARENT - label process
Presenter
Presentation Notes
Remember earlier we discussed that it is common to find areas within a guideline topic for which there is insufficient evidence to develop evidence-based recommendations? So, is it possible to address both types of situations within the same project or document? Yes. We call these Hybrid documents and we aim to make them trustworthy also. Hybrid guidance documents combine both EBG and CS processes Since both are PICO-specific, you can follow the appropriate process for each PICO question individually. Just be sure to provide a thorough explanation to your readers and be clear about which process you followed for which clinical statement (recommendation or suggestion). This hybrid approach affords a major advantage: it provides trustworthy clinical guidance to healthcare providers for all topic areas. Again, and most Importantly, BE TRANSPARENT. Include a label for each clinical statement that indicates which process was utilized for that recommendation or suggestion.
Trustworthy Guidance Documents
Avoiding common pitfalls Do not allow stacking the panel with like-minded
individuals. Do not rely on SMEs who may insist there are no
published studies. Even if they are right, it will provide a reliable foundation for the CB approach
Insist on formal rules for consensus achievement No shortcuts permitted in the final reviews Important: ALWAYS BE TRANSPARENT
Presenter
Presentation Notes
Like guidelines, other guidance documents are subject to ploys and attempts to reduce quality. But that will reduce the trustworthiness of your products. So be prepared to enforce the regulations. Do not allow stacking the panel with like-minded individuals. Make sure you have good balance and wide representation from all perspectives. Do not rely on SMEs who may insist there are no published studies. Even if they are right, a thorough and systematic search will provide a reliable foundation for the CB approach. Insist on formal rules for consensus achievement. It is a good idea to provide the panelists with a list of the rules prior to the voting process. No shortcuts should be permitted in the final reviews. Important: BE TRANSPARENT - label the process that was followed
