April 2009Netta Conyers-Haynes,Principal Consultant, Communications

Kaiser PermanenteNational Guideline Program

Implications of IOM SR Standards

Wiley Chan, MDPhysician, Internal MedicineMethodologist, KP National Guideline Program

Kaiser Permanente: Largest Non-Profit Health Care Program in the United States

Founded in 1945 8 regions in 9 states

and District of Columbia8.6 million members (as of 12/09)

15,129 physicians (as of 12/09)

164,098 employees (as of 12/09)

KP Care Management Institute (CMI)

KP National Guideline Program (NGP)

Kaiser Foundation Hospitals

PermanenteMedicalGroups

KaiserFoundationHealth Plan

Kaiser Permanente

Kaiser Permanente Evidence Network

15 KP National Evidence-Based Guidelines Supported by full Systematic Reviews

3.5 Staff FTEs dedicated to KP NGP work 1 Principal Consultant , 2 Analysts, 2 Project Managers

1.5 Physician FTEs dedicated to KP NGP work 7 Physician EBM Methodologists

External vendor: Doctor Evidence Search & Data Extraction Technology platform

• Data repository, analysis & documentation

Kaiser Permanente National Guideline Program: Process & MethodologyNew

ClinicalIssue

ScheduledUpdate

ClinicalQuestions

(CQ)

ExistingGuideline?

CQMatch?

AssessGuideline

Acceptable?

ExistingSR by CQ?

AssessSR

Acceptable?

New RelevantStudies?

EvidenceSearch

AbstractArticleReview

Inclusion/Exclusion

DataExtraction

CriticalAppraisal

Qualitative/QuantitativeSynthesis

AssessEvidenceQuality

EvidenceSummary Rationale Recommendations

AssessImplementability ofRecommendations

GDT Approval

Internal Review

GQNGD

Approval

NationalGuideline

Yes

Yes

Yes

Yes

Yes

Yes

No

No

No

No

No

No

Implementation

OverviewKey to Abbreviations:• CQ Clinical Question• SR Systematic Review• GDT Guideline Development Team• GQ Guideline Quality Committee• NGD KP National Guideline

Directors

Kaiser Permanente National Guideline Program: Process & MethodologyNew

ClinicalIssue

ScheduledUpdate

ClinicalQuestions

(CQ)

ExistingGuideline?

CQMatch?

AssessGuideline

Acceptable? GDT Approval

Internal Review

GQNGD

Approval

NationalGuideline

Yes

Yes

Yes

Implementation

Acceptable External GuidelineADAPTE/AGREE

Kaiser Permanente National Guideline Program: Process & MethodologyNew

ClinicalIssue

ScheduledUpdate

ClinicalQuestions

(CQ)

ExistingGuideline?

CQMatch?

AssessGuideline

Acceptable?

ExistingSR by CQ?

AssessSR

Acceptable?

New RelevantStudies?

EvidenceSummary Rationale Recommendations

AssessImplementability ofRecommendations

GDT Approval

Internal Review

GQNGD

Approval

NationalGuideline

Yes

Yes

Yes

Yes

No

No

No

NoImplementation

No Acceptable External GuidelineAcceptable External Systematic Review

No New Relevant StudiesAMSTARGRADE

Kaiser Permanente National Guideline Program: Process & MethodologyNew

ClinicalIssue

ScheduledUpdate

ClinicalQuestions

(CQ)

ExistingGuideline?

CQMatch?

AssessGuideline

Acceptable?

ExistingSR by CQ?

AssessSR

Acceptable?

EvidenceSearch

AbstractArticleReview

Inclusion/Exclusion

DataExtraction

CriticalAppraisal

Qualitative/QuantitativeSynthesis

AssessEvidenceQuality

EvidenceSummary Rationale Recommendations

AssessImplementability ofRecommendations

GDT Approval

Internal Review

GQNGD

Approval

NationalGuideline

Yes

Yes

Yes

No

No

No No

No

Implementation

No Acceptable External GuidelineNo Acceptable External Systematic Review Internal Systematic Review GRADE

Kaiser Permanente Systematic ReviewProcesses, Responsibilities, & Tools

Clinical Questions KP: Population, Intervention, Comparison, Outcome

• Timing, SettingEvidence Search

Doctor Evidence with KP input• Masters-trained Medical Librarians (MLIS)

Abstract/Article Review & Inclusion/Exclusion Doctor Evidence

• Dual Inclusion/Exclusion with 3rd party adjudication KP: Evaluate exclusions

Kaiser Permanente Systematic ReviewProcesses, Responsibilities, & Tools

Data Extraction Doctor Evidence

• Dual extraction with adjudication & 3rd party independent QA• Masters-trained & graduate student data extractors• Contact authors to clarify discrepancies in published data

KP: Quality AssuranceCritical Appraisal

KP: Cochrane Risk of Bias, Doctor Evidence platformQualitative/Quantitative Analysis

KP: Doctor Evidence platform

Doctor Evidence Meta-Analysis Page

Doctor Evidence Meta-Analysis Results

Doctor Evidence Meta-Analysis Study Selection

Detailed study- and outcome-specific

descriptions of Cochrane Risk of Bias are exposed by

hovering

Select and de-select studies and Recalculate

Doctor Evidence Study Summary

Detailed study-specific descriptions of

characteristics and outcomes are exposed

by hovering

Kaiser Permanente Systematic ReviewProcesses, Responsibilities, & Tools

Evidence Grading KP: GRADE, GRADEPro

• Developing functionality in Doctor Evidence platformEvidence Summary (SR)RationaleRecommendations & Guideline

KP: GRADE, GRADEPro, Word• Developing functionality in Doctor Evidence platform

Kaiser Permanente National Guideline Program: Process & Methodology

NationalGuideline

DiseaseManagementAccreditation

(NCQA)

Clinician &Staff Education

Tools(Clinical Library)

Patient EducationTools

(KP.ORG,National Patient

Instructions

Regions(EHR, CDS,Operations)

Guideline Implementation

Key to Abbreviations:• EHR Electronic Health Record• CDS Clinical Decision Support• KP.ORG KP publicly-accessible Internet site• NCQA National Committee for Quality

Assurance

IOM Systematic Review StandardsProblem Areas for KP

2.6 Develop a systematic review protocol2.7 Submit the protocol for peer review2.7.1 Provide a public comment period for the

protocol2.8 Make the final protocol publicly available5.2.1 Use a 3rd party to manage peer review5.2.2 Provide a public comment period for the

report5.3 Publish the final report with free public

access

IOM Systematic Review StandardsChallenges

Very resource-intensive Infinite needs - Finite resources Balance between efficiency versus rigor Investment in developing expertise & infrastructure

Migration of existing SRs & CPGs to new methodsUpdating SRs & CPGs

Every 2 years? Dynamic updating, based on evidence & impact?

External reviewPatient & public involvement in SRs and CPGs

IOM Systematic Review StandardsOpportunities

Improved SR & CPG rigor & transparencyCommon evidence-grading & recommendation

strength languageCollaborate with other SR & CPG developers

Prioritize & coordinate topics Avoid duplication Target funding Share data extractions?

Devote more energy to CPG implementationCollaborate with performance metric developers

Implications of IOM SR Standards

Wiley.Chan@kp.org