Competency-Based Education: Milestone or Millstone?

Competency-Based Education

Milestone or Millstone?

Geoff Norman, PhDMcMaster University

Competency-Based Education

• Basic Tenets

– PG education should be directed at defined competencies, not defined time blocks

– PG trainees should be evaluated on achievement of these competencies (milestones) to varying levels, culminating in independent practice

The Basic Assumptions

• Achievement of a competency can be visualized as a smooth curve of increasing competency with time and practice.

• At some point the competency is sufficient to permit independent professional practice

– The Entrustable Professional Activity (EPA)

• Different individuals will achieve different competencies at different rates.

Entrustable Professional Activity (EPA)(Wijnen-Meier, 2012)

EPA• a critical unit of professional work to be

entrusted to a trainee once competence at a sufficient level has been reached.

• independently executable and should be observable and measurable in process and outcome.

“Adoption of OBE would better equip medical graduates to respond effectively and efficiently continue to expand the depth and breadth of the requisite competencies”

– Carnegie Foundation Report, 2010

“OBE is an intuitive approach that engages the range of stakeholders… it encourages a student-centered approach and at the same time supports the trend for greater accountability and quality assurance.” Harden, 2007

• Dissemination

– The Scottish Doctor (Scotland)

– ACGME Competencies (US)

– Tomorrow’s Doctor (UK)

–CanMEDS roles (Canada, Netherlands, 24

others)

The Scottish Doctor

• CAN DO– Clinical Skills– Practical procedures– Patient Investigation– Patient Management– Health Promotion– Communication– Medical Informatics

• APPROACH and PRACTICE– Basic, social and clinical sciences– Attitudes, ethics and legal– Decision making , reasoning, judgment

• PROFESSIONAL– Role within health service– Personal development

ACGME COMPETENCIES

• Patient Care

• Medical Knowledge

• Practice Based Learning and Improvement

• Systems Based Practice

• Professionalism

• Interpersonal Skills and Communication

• Source: ACGME Core Competencies | University of Maryland Medical Center http://umm.edu/professionals/gme/competencies

CanMEDS Competencies

CAnMEDS Roles

• Adopted by 26 countries worldwide

• Implemented (?) in all RCPS training programs

The ISSUES

• Problems with:

– Objective proliferation

– De-emphasis of “softer” competencies

– Mastery modelling

– Sampling and statistical precision

– Assessment – macro or micro

– Logistics and remediation

Objective Proliferation

• Each major goal decomposed into objective, sub-objectives

• Number of sub-objectives become unmanageable

• Even at sub-sub goal, it is still not sufficiently specific

CanMEDS Roles

Level 2 Level 3 • Expert 6 28• Communicator 5 17• Collaborator 2 16• Manager 4 13• Advocate 4 13• Scholar 4 25• Professional 3 14

• TOTAL 28 126

Subgoals proliferate

• CanMEDS 126

• Scottish Doctor 86

• ACGME (Internal Medicine) 142

however…. still not specific

CanMEDS sub – sub - goals

1.1.1 Effectively perform a consultation, including the presentation of welldocumented assessments and recommendations in written and/or verbal form response to a request from another health care professional

1.2.1 Apply knowledge of the clinical, socio-behavioural, and fundamental biomedical sciences relevant to the physician’s specialty

1.5.2 Demonstrate effective, appropriate, and timely performance of therapeutic procedures relevant to their practice

3.1.8 Describe the principles of team dynamics

4.1.4 Describe principles of healthcare financing, including physician remuneration, budgeting and organizational funding

4.2.4 Set priorities and manage time to balance patient care, practice requirements, outside activities and personal life

6.3.1 Describe principles of learning relevant to medical education

Scottish Doctor Goal and subgoal

GOAL

‘‘Take a history from patients, relatives and others”

Sub- GOAL

• All age groups; local/multi-ethnic factors; a wide range of different contexts; and a patient-centred, sensitive, structured and thorough approach with demonstration of principles of good communication

(And, by the way, how many kinds of history are there?)

ACGME Competencies

“Ability to perform and interpret common laboratory tests”

But there are 39 “common laboratory tests” in internal medicine (Wigton, 1989)

De-emphasis of soft competencies

“an outcome-based approach highlights areas in the curriculum which may be neglected whether they are traditional aspects of medical practice such as ethics and attitudes….” Harden, 2007

************************

“educational processes, which influenced the development of values, insight, and judgement, could not be subsumed into a curriculum model that focused strictly on behavioural objectives. When students learned affectively, socially, culturally, aesthetically, or ethically from experience, it was not possible to specify goals or assess them objectively but that did not mean such types of learning were unimportant” Morcke, 2012

“qualities, such as humanism, altruism, professionalism, and scholarship, are difficult to define in objective, measurable terms, and thus may receive short shrift in assessment” Norman, Bordage, Norcini, 2014

Mastery Modelling

• The identification of a single smooth curve crossing a threshold (the EPA):

– results in a formidable measurement problem

– is dependent on a precise specification of the EPA

– Is inconsistent with actual accumulation of skills / competencies

Growth of competence over time

training deliberate professional practice

Dreyfus & Dreyfus, 1986

proficient

expert

competent

advanced

novice

However…

• It’s not that easy to identify the EPA level

(Wijnen-Meier, 2012)

• 35 MD students• 28 FY1 residents• 23 FY2 residents …. and their supervisors

• Rate 10 clinical and 4 general activities1= Not yet able to do a task2= May act under full supervision3 = May act under moderate supervision4= May act independently5= May act as instructor

• 10 Clinical Activities– e.g. draw up an examination plan for a new

patient at the Outpatient department

– e.g. To treat a skin laceration of about 4 cm at one of the extremities.

• 4 General activities– e.g. write an evidence-based case-report for

colleagues with limited preparation time (three hours).

Can’t tell them apart

• “Principal component analysis on ‘clinical activities’ shows a one-factor solution that explains 53% of the variance”

• “Principal component analysis on the part ‘‘general activities’’, revealed a one-factor solution explaining 65% of the variance”

• “Reliability analysis showed high internal consistency (alpha = .80) between supervisors”

Level of Competence

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

MD GY1 PGY2

Clin Self

Clin Super

Gen Self

Gen Super

(Wijnen-Meier, 2012)

Level of Competence

MD GY1 PGY2

Clin Self

Clin Super

Gen Self

Gen Super

Mastery modelling

• Even simple behaviours, learned on controlled settings, with multiple measurements, do not follow a smooth master curve

The Learning Curve

Ward ST, Mohammed MA, et al. BMJ Open Resp Res, 2014

297 residents, 36,730 colonoscopies

“Cecal Intubation Rate” (CIR) at 90% achieved at 233 procedures

Based on 297 data points, 36,730 observations

234 290

The “Pursuit Rotor”

Learning on Pursuit Rotor (Adams, 1952)

– 39 subjects, 2 groups (massed, distributed

– Record time when pointer in contact with brass disc

– Recording every 10 sec.

Massed• 6 min./day x 5 days = 30 min.

Distributed• 36 x 10 sec. trials / day x 5 days with 40 sec. break = 30

min.

Average of 20 students

Average of 19 students

48 108

60 145

Sampling Issues

Because of content specificity, it is not feasible to collect sufficient data to provide sufficient precision to permit defensible identification of time to competency

McMaster Undergraduate Medical Program (classes 2005-2016)

McMaster Undergraduate Medical Program (class of 2014)

McMaster Undergraduate Medical Program

(Individual Students in Class of 2014)

McMaster Undergraduate Medical Program

(Individual Students in Class of 2014)

McMaster Undergraduate Medical Program

(Individual Students in Class of 2014)

Assessment Issues--Macro or Micro

“General competencies are what most governing bodies and educational leaders responsible for physician training like to stress.”

“It is not necessary to choose between competencies and activities. Rather, by acknowledging that both are relevant pieces of the training process and that each represents a different dimension of the same overall objective of professional training, we can reconcile the concepts of competence and clinical practice.”

Ten Cate and Scheele, 2008

Can we assess competencies reliably?

• MICRO level sub or sub-sub goals– Assessment of individual activities, like suturing,

venipuncture

• MACRO level – overall goals– Assessment of CanMEDS roles directly, using ongoing

assessment like mini-CEX (Sherbino, 2013) or summative assessment like ITER ( Tromp, 2012; Ginsburg, 2013)

Global Rating Checklist

N = 21 18 9 N = 21 18 9

MICRO Level (Dwyer et al, 2014)6 stations – Clinical Radiograph, Procedure39 Residents5 point global rating, specific checklist

• Some junior residents outperformed some fellows

• But despite same training, differences were present between groups

“Despite identical intensive teaching to a set curriculum within a CBME model, junior residents were not able to demonstrate or apply clinical skills as well as senior residents, suggesting that experience is a critical factor”

Simple mastery model may not be appropriate

MACRO level Summative

• Tromp, 2012

– Competency Assessment List COMPASS

• 7 CanMEDS roles; 19 sub-objectives

– Completed by trainer over successive 3 month periods

– 68 trainees

Results“Compared with T1, most trainees received higher ratings in T2. A relative small group of trainees with no increase in ratings was found in all roles: Medical Expertise (21.3%), Communication (31.3%), Management (22.5%) Collaboration (23.8%), Social Accountability (22.5%), Science and Education (22.5%) and Professionalism (16.3%).”

Change over 3 month about 1.0/6, ES = .6

“Scores of the Compass ... show[ed] excellent internal consistency ranging from .89 to .94.”

But that‘s too high. Does not differentiate roles.

• Ginsburg, Eva, Regehr

– ITER with 7 CanMEDS roles, 19 items and 5 point scale

– 63 residents, average 9 rotations over 2 years

“The reliability of a single score (end of rotation evaluation) was 0.11 for both the year one and year 2 data”

“The unrotated factor solution revealed two factors with the first accounting for 66% of the variance and every item loading more or less equally… the second factor accounted to 5.4% of the variance.”

- Reliability too low

- Unable to discriminate roles

Conclusions - MACRO summative

Summative assessments are incapable of differentiating mastery at different levels of different competencies.

– The EPA model fails

MACRO Ongoing

• Sherbino et al., 2013

– 4 week EM rotation

– 155 medical students

– 1819 evaluations of encounters

– 7 CanMEDS roles with descriptors and 10 point scale

– Evaluator assessed Medical Expert and 1 or 2 other roles

Proportion of completed ratings

0

20

40

60

80

100

120

Column1

Percent of variance in Scores

2

Student

Rater

Item

Error

67%

20%10%

3%

“a single rater’s rating had a reliability of 0.12. Averaging across a minimum of 17 raters was required for an acceptable reliability of 0.7”

“[Factor] analysis revealed a single factor ..accounting for 87% of the variance in scores… the next largest factor accounted for less than 4% of the variance….Loadings ranged from 0.88 to 0.97”

• Few assessments of roles other than Medical expert

• Unable to discriminate roles

• Reliability too low

Conclusions - MACRO ongoing

Ongoing assessments are incapable of differentiating mastery at different levels of different competencies.

– The EPA model fails

Conclusion

• We will never be able to assess competencies at the required level of specificity with the required level of precision to make defensible judgments of “time to attain the EPA level of proficiency” across the full range of competencies

• The notion that CBE will achieve efficiency through individually reduced training time is fundamentally flawed

Residual benefits

CBE does represent an important approach to identifying core competencies

– Useful for curriculum design to ensure uniform exposure

– Useful for formative assessment using sampling strategies

– Useful for summative assessment (licensure, certification) using sampling strategies

• Ferguson PC, et al. J Bone Joint Surg (2013)

Introduction of CanMEDS CBE increased number of assessments/ per resident

No of assessments Old / New

0

5

10

15

20

25

30

35

40

Old New

PGY1

PGY2

PGY3

“The difference between an optimist and a pessimist is that the optimist believes this is the best of all possible worlds….

and the pessimist is afraid he’s right.

J.R.Oppenheimer