SES:059 Beyond Collaboration: Real-time Integration with the C-Suite:

Using CLER, ACGME, and GME Data Innovatively to Drive Institutional Change and Improve the Clinical Learning

Environment

David Entwistle, M.A., President & CEO Stanford Health Care Ann M. Dohn, MA, DIO, Director of GMENancy Piro, PhD, Senior Program Manager

Conflict of Interest

SPEAKERS:• David Entwistle, MA ‐ ACGME Board of Directors, Member• Ann Dohn, MA – No Conflicts of Interest• Nancy Piro, PhD ‐ No Conflicts of Interest

Session Objectives

1. Better understand all the new/possible data sources DIOs, Program Directors, Hospital Leadership and Administrators can leverage for innovation

2. Understand how to integrate GME infrastructure with institutional organizational assets to effectively leverage this data collaboratively in high level institutional decision‐making e.g., resource allocation

3. Learn how to apply and fully utilize the data for collaborative change in the Clinical Learning Environment

4. Understand ways to establish integrative and collaborative processes which enable the enhancement and alignment of GME with the overall health care delivery system. 

Agenda

DiscussI. The availability of multi‐sourced data and use of this data by 

GME and the C‐Suite to spearhead strategic innovation in the learning environment

II. Tools that facilitate collaboration using existing data sources e.g., CLER that can integrate with C‐Suite strategic priorities

III. Specific processes that illustrate how this data can be used to collaborate with hospital leadership

Background – Historical Perspective

• Historically, many GME decisions have been made in silos devoid of any meaningful connection between:– the C‐Suite, GME, Residents, the Clinical Learning Environment (CLE)and the Hospital & Medical School mission/vision.

C‐SUITE GME/PDs

Clinical LearningEnviron‐ment

HospitalMissionVision Med School

MissionVision

Residents

Background – Different Perspectives

• The C‐Suite, GME, Residents, the Clinical Learning Environment (CLE) and the Hospital & Medical School bring different perspectives to the table, e.g.,– Service vs Education– Revenue Units vs Academic Time– UME vs GME– Patient Care vs Protected Time

• Today we have the availability of multiple data sources which can enable the “means” to facilitate this evolving partnership. ..

• Purpose– Mission, Vision, Values

• Strategies– Complex Care, Coordinated Care

• Objectives– Goals, Operating Plans, Dashboards

• Performance– Improvement and Management

Understanding C-Suite Priorities

8

STANFORD OPERATING SYSTEM…Standard Operating System…

Data Sources

GMEC‐Suite

Program 

Evaluations

Program QualityEvaluations

Safety Reports & Surveys

National Patient Volumes 

Benchmarking  Data 

ACGME RRC Accreditation Decisions 

Hospital IP/OP Utilization Statistics ‐Volumes

Hospital Strategic Objectives

FinancialReports

ACGME & GME Wellness Survey

CLER Data

Patient Satisfactionassessments

ACGME & Internal Surveys

Brainstorming new data sources –what have we missed ?

Making data meaningful – speaking the C-Suite “Language”

• Applying and fully utilizing the data for collaborative change in the Clinical Learning Environment

• Establishing and maintaining a data‐based dialogue with the C‐Suite is the first step in true integration between GME and the C‐Suite.  

Building Integrative and Collaborative Processes

• Setting up integrative and collaborative processes which enable the enhancement and alignment of GME with the overall health care delivery system. 

MULTI‐SOURCE DATA PROCESSES COLLABORATIVE 

DECISIONS

Process Example – Program Expansion and Funding

14

Historical Perspective – A3 “What was the Problem”

Understanding some of the issues surrounding C-Suite views on residency/fellowship expansion.

• SERVICE vs Education• Budget Cycle vs Academic Year• Hospital growth, expansion and strategies not tied to the educational cycle• Cost of trainees

15

Understanding some of the issues surrounding Program Views on Residency/fellowship Expansion.

• EDUCATION vs Service• Newly Recognized Areas of Training…

– Epilepsy– Peds Epilepsy– Neuroimmunology– Selective Pathologies

• Faculty recruitment / retention‐I want a fellow to “teach” …”

16

How Did We Design a New Process?

First Steps  • Talked with Stakeholders to determine their current and future needs and issues.

– Discussions with C‐Suite, Dean, Chairs, DFAs (Department Business Managers), GMEC, Program Directors and Program Coordinators.

• Aligned the educational needs and issues with Institutional Deadlines– ACGME requirements for funding guarantees– NRMP Quota Deadlines

• Preliminary Process Designed …tweaked and retweaked• Debriefed new process with GMEC

17

Timeline

• Application materials/requests due to GME – July 1st for September Meeting• Second Biannual meeting – Requests due to GME October 1st for January Meeting

18

• Members:– CEO and President, Stanford Health Care (or designee)– Dean, Stanford Medicine (or designee)– Associate Dean GME, Chair GMEC– DIO, Director of GME– COO, Stanford Health Care– CMO, Stanford Health Care– Two Department Chairs– Two GMEC resident reps (1 yr term)– One (1) Program Coordinator

New Program Expansion/Funding GMEC Subcommittee

Looking into the Data:

20

Program Expansion / Funding Dashboard with Multi-Source Data

21

Determining the need for special program reviews

– More data required– Program too small– Long interval since last formal review

22

• Results of the special review if applicable• Focus on educational quality of programs• Alignment with hospital mission/strategic plans• Interval growth in teaching faculty, specialty institutes and departments that offer educational opportunities

• Long term plans for departmental growth• Program specific growth in ambulatory and inpatient volume (historical and projected)• Workforce needs

Discussion Points for GMEC Expansion/Funding Subcommittee

Decision Making Rationale

24

Summary of Program Expansion Committee Results

Decision Making RationaleResults of Funding Decisions

GME/CEO Collaborative Expansion/Funding Process Points

• Subcommittee recommendations go to the GMEC for final approval.• Positions are not granted or funded outside this process.• The process is transparent with results immediately posted on the GME website.

Real transparency isn’t this !!!

28

If Funding is Denied: Programs Can Re-apply for Funding

Deny

Program Expansion/Funding Committee Decision

Can re‐apply after two years but decision will again be based on same parameters including 

educational quality and priorities

“The course of true love never did run smooth….”William Shakespeare

• What would we do differently?– Align Meetings with NRMP / Recruitment Deadlines– Increase communication among all the stakeholders during process development– Put final process on website sooner

30

Maintain Open Communication

Expansion FundingProcess

Program Directors

Dean

C‐SuiteVPs

DEPTChairsDFAs

31

Discussion – How is this accomplished at your institution? What would you like to change?

Part Two: Using CLER Results to Improve our Clinical Learning Environment

CLER Example – Integrating GME into the Institutional Culture of Safety – Breaking Down the Silos

Why the Clinical Learning Environment is so important...

• Essential to:– Training of Medical Students, Residents & Fellows to become competent physicians

• Trainee Supervision• Trainee Wellness, Fatigue Management & Mitigation• Trainee Professionalism

– Patient Care, Healthcare Quality and Safety• Effective Transitions of Care• Addressing Healthcare Disparities

Culture of Safety

• GME is pivotal to the institutional safety culture …• Residents are imperative to the overall safety culture…

Do we have any data from residents that will inform GME and the C‐Suite?

The answer is “CLEAR”…..e.g. CLER !

• First CLER Site Visit – provided us with a qualitative report – what to do with it?• Second CLER Site Visit – provided us with another qualitative report

– We attempted to analyze trends between the first/second reports to see if our efforts were working,  but it was challenging:

• the majority of data was narrative• some questions changed• Multi‐site comparisons weren’t possible• Lack of normative data

• Next Steps:  Transform CLER data to a format that would enable collaboration with hospital leadership to implement C‐Suite strategic priorities.

But just HOW can we do that? 

Building Integrative and Collaborative Processes

• Establish integrative and collaborative processes which enable the enhancement and alignment of GME with the overall health care delivery system. 

MULTI‐SOURCE DATA PROCESSES COLLABORATIVE 

DECISIONS

We had CLER data… but how do we transform it to make it meaningful ?

Quantification of Narrative CLER findings … Intra Institutional Trends

CLER AREAS‐Questions

 = No Response

Residents47

Faculty55

PDs44

Residents51

Faculty56

PDs47

Patient SafetyKnew  Hospital and Clinics’ priorities in the area of patient safety

75% 79% 93%

Experienced an adverse event or near miss at 

75%

Had the opportunity to participate in a patient safety investigation such as an RCA (PGY3 and above residents)

30% 64% 66%

Had personally reported a patient safety event at  Medicine using the hospital’s reporting system

53% 25% 15%

Believe their residents and fellows report the event using the hospital’s reporting system

59% 52%

Believe their residents and fellows rely on a nurse to submit the report

18% 13% 7%

Believe their residents and fellows / or  rely on their supervisor to submit the report

29% 20% 35%

Believe their residents and fellows care for the patient but do not / chose not submit a report

10% 9% 7%

Have reported a near miss at  Medicine in the past year

29%

Received feedback on the outcome of a report that had been filed by any means.

28%

Participated in an interprofessional safety investigation (PGY 3 and above) .

38%

Have received education about proactive risk assessments (e.g., Failure Mode and Effects Analyses)

45%

Spring 2017Summer 2014

STEP 1: Identify each reported area and its respective percentages by resident, faculty or PD group

Normative data (Safety as an example) from CLER reports

STEP 2: Add National Normative data from ACGME 

Take Aways

• Identified the need to improve resident involvement in Safety initiatives• Formed a Resident Safety Council – Good initial results…

Displaying normative data from CLER to monitor trends to assess the success of continuous quality improvement efforts.

STEP 3: Test significance to monitor progress on our initiatives

Understand how to use normative data from CLER to monitor trends to monitor the success of continuous quality improvement efforts together with tests of significance

• Step 4: Color Code your data Red/Green with respect to National Normative Data

CLER AREAS‐QuestionsSignificant Difference (p < .05) Direction of Difference NORM

Q Number

Population Positive NegativeChi‐

squarep‐value Z‐Score p‐value Residents Positive Negative Faculty PDs

Healthcare QualityHad participated in QI project of their own design or one designed by their program or department (PGY2s and above)

Significant Difference/Stanford Higher than Norm

76% B9 8298 6306 1992 7.223 p<.01 2.688 0.007 91% 52 5 NR NR

Project was in some way linked to the hospital’s goals Not Significant

52% B10 6154 3200 2954 3.069 p>.05 ‐1.752 0.08 40% 23 34 NR NR

Didn't know if project was in some way linked to the hospital’s goals

40% NR NR

Engaged in inter‐professional QI teams working on performance improvement projects

Significant Difference/Stanford Lower than Norm

75% B11 3133 2350 783 6.994 p<.01 ‐2.645 0.008 59% 34 23 NR NR

Residents have access to organized systems for collecting and analyzing data for the purpose of QI Not Significant

63% B12 8549 5386 3163 0.089 p>.1 0.298 0.77 65% 37 20 77% 82%

Knew the hospital’s priorities with regard to healthcare disparities Not Significant

55% B13 8607 4734 3873 0.389 p>.1 ‐0.624 0.53 51% 29 28 45% 60%

Transitions in CareKnew the hospital’s priorities for improving transitions of care

86% 92% 98%

Used a standardized process for sign‐off and transfer of patient care during change of duty Not Significant

88% B14 8471 7454 1017 0.004 p>.1 ‐0.064 0.95 88% 50 7 NR NR

If a standardized process is used, written templates of patient information are used to facilitate the hand‐off process.

Significant Difference/Stanford Lower than Norm

78% B15 7262 5664 1598 7.227 p<.01 ‐2.688 0.007 64% 36 21 NR NR

201457 Residents

55 Faculty Members44 PDs

2014ACGME Norm Data

Quantification of CLER findings across External Institutions in a Scorecard Format

CLER AREAS‐Questions

2014ACGME Norm Data

NORMResidents

47Faculty55

PDs44

Residents51

Faculty56

PDs47

Residents 53 Faculty  37 PDs         31 Residents 58 Faculty  55PDs       39

Residents12

Faculty  13

PDs      4   

Residents12

Faculty  17

PDs3    

Patient SafetyKnew  Hospital and Clinics’ priorities in the area of patient safety

74% 75% 79% 93% 64% 58% 68% 100% 100% 100%

Experienced an adverse event or near miss at 

68% 75% 77% 71% 75% 100%

Had the opportunity to participate in a patient safety investigation such as an RCA (PGY3 and above residents)

30% 64% 66% 0% 62% 100%

Had personally reported a patient safety event at  Medicine using the hospital’s reporting system

47% 53% 25% 15% 20% 34% 24% 23% 25% 33% 92% 50% 66%

Believe their residents and fellows report the event using the hospital’s reporting system

59% 52% 56% 59% 47% 33% 33% 77% 50% 8% 88% 100%

Believe their residents and fellows rely on a nurse to submit the report

18% 13% 7% 18% 8% 7% 20% 11% 18% 11% 0% 0%

Believe their residents and fellows / or  rely on their supervisor to submit the report

29% 20% 35% 45% 31% 21% 29% 38% 46% 56% 23% 50% 12%

Believe their residents and fellows care for the patient but do not / chose not submit a report

10% 9% 7% 12% 6% 14% 17% 4% 3% 0% 0% 0% 0%

Have reported a near miss at  Medicine in the past year

20% 29% 13% 17% 50%

Received feedback on the outcome of a report that had been filed by any means.

47% 28% 35% 31% 38% 25%

Participated in an interprofessional safety investigation (PGY 3 and above) .

41% 38% 53% 57% 83% 43% 69% 51% 50% 82% 100%

Have received education about proactive risk assessments (e.g., Failure Mode and Effects Analyses)

45% 35% 41%

Have received training on how to disclose medical errors to patients and families

51% 66% 92%

If they were to be involved in a major patient safety event resulting in a patient 92% 93% 100%

Institution "A"Spring 2017

Institution "C"Winter 2014

Institution "C"Fall 2015

Institution "A"Summer 2014

Institution "B"Winter 2015

Institution "B"Summer 2017

How can all this CLER data be used to make changes in the Clinical Learning Environment at your institutions

• Investigate and leverage CLER Trends, Significance and Normative Data– Focus on areas where the institution is lower than:

• the ACGME CLER National Norms• comparative institutions

– Trending changes across CLER visits• Input to the AIR (ACGME Required Annual Institutional Review)

Analysis of the Effectiveness of our Initiative

• This full integration also engages our residents and fellows in the design of healthcare organizations that will deliver care across a complex continuum of care. 

• Collaborative real‐time integration will concomitantly promote ACGME’s overall vision to support rethinking our healthcare model to enduring institutions. 

Going beyond CLER …

• What else do we know about the Clinical Learning Environment?– A wealth of data coming in ….

• CLER Reports• GQ (Graduation Questionnaire – Medical Students)• Internal Resident Surveys• ACGME Resident and Faculty Surveys• Resident and Faculty Program Evaluations• Trainee Alumni Surveys• ACGME Accreditation Reports• LCME Reports

What do we know about the Clinical Learning Environment?A wealth of data coming in ….

Clinical Learning

Environment

CLER Reports

GQ

ACGME Resident and 

Faculty Surveys

Internal Reviews & Resident Surveys

Trainee Alumni Surveys

LCME Reports

ACGME Accreditation 

Reports

Trainee Alumni Surveys

Resident and Faculty Program 

Evaluations

Group Exercise

• Discuss how the increased availability of multi‐sourced data facilitates innovative collaboration between GME, PDs and the C‐Suite spearheading strategic changes in the learning climate. (15 minutes – interactive)

All these data sources enable further discussion and collaboration between GME and the C-Suite

Questions