Insightful practice and competency assessment: how do we comply with the “autoregulation”

McGill Annual refresher course

Ernest Prégent, M.D. CCFP(em) CSPQ FCCFPPractice Enhancement Division

Collège des médecins du Québec

…from quality assurance to quality improvement

McGill Annual refresher course

• My team and I, do not have any affiliation with a commercial organization

• Emergency medicine specialist (Sacré-Coeur Hospital)• Université de Montréal, Université Laval

Disclosure

December 2019 3

• Understand the concept of autoregulation and the linkwith our medical practice

• Identify the tools used in the assessment process• Evaluate the risk factors present in a medical practice • Integrate QA/QI before and after an inspection visit and

respond to the CPD mandatory regulation in Québec

OBJECTIVES

2019 McGill Annual Refresher Course 4

Context: Historical, legal, medicolegal, social

The physicians’ assessment programs Risk-factor based, Process, Efficiency

Remediation…helping

QA and QI of your practice ?

Outline of the Presentation

2019 McGill Annual Refresher Course 5

Quality medicine at the service of the public

Continuing Professional

DevelopmentCommittee

Professional Inspection Committee

Practice Enhancement Division

Inquiries Division

Medical Education Division

Administrative Services Division

LegalServices Division

Executive Office President and Chief Executive OfficerSecretary

The Board of Directors

Collège des médecins du Québec

6

• How would you evaluate the practice of the physiciansitting next to you ?

• How (process/tool) would you evaluate your practice ?

• How would you like that the physician sitting next to you, evaluate your pratice ?

As a baseline…

McGill Annual Refresher Course 7

A bit of epidemiology

8

Age distribution of active physicians

50%

2018-2019

50%

% and age distributionactive 2018 vs « red line …later »

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-99

Tranches d'âge

64,9 ans

MAJ 2019

Mean age of active MD

49,7 ans

Active physicians ≥ 70 yoJan 2019

614

330

90

20 4

915

359

114

28 50

100

200

300

400

500

600

700

800

900

1000

70-74 75-79 80-84 85-89 90-97

MD

actif

s

2010 2011 2013 2014 2016 2017 2018 2019

MAJ 2019

2010 : 1058 MD

2011 : 1097 MD

2013 : 1230 MD

2014 : 1250 MD

2016 : 1317 MD

2017 : 1373 MD

2018 : 1389 MD

2019 : 1421 MD

• Only College with licensing and regulatory activities• Reporting to the Ministry of Justice• Mission: quality medicine so as to protect the public

Self-regulation: a peer-managed profession

Context

12

• A framework law adopted in 1973• Office des professions du Québec

• Code of Ethics of physicians determines the duties and obligations to be discharged by every member of the Collège des médecins du Québec (CMQ) (art. 87)

• A professional inspection committee (PIC) is established within each order by the Board of Directors (art. 109)

• Self-Regulated Profession• Public representatives

Professional Code

13

Background19901973

screening

needs assessment

remediation

evaluation

Level 1

Level 2

Level 3

1996

1996: A study* on a random sample of 100 family physicians reveals that 95% had an adequate quality of practice

1997: the Practice Enhancement Division develops inspection programs based on indicators of quality of care 40 programs have been developed over the

years (15 still active, 4 major)

*Goulet et al. (2002), “Performance assessment: Family physicians in Montreal meet the mark!”, Can.Fam.Physician,

vol 48 pp 1337-1344

2008 College of Physicians and Surgeons of Ontario (CPSO) annual report◦ > 89 % of physicians selected randomly have a satisfactory practice

Everything is « context », right ?

What we, and the public, have been… and are reading

• Foundation document for all studies of health care quality

• Health outcome: end results of medical care measuresby health status (outcome) and patient satisfaction

Groundbreaking 1989 essay Avedis Donabedian

18

• Medical quality: the degree to which health services for individuals and populations increased the likehood of desired health outcomes and are consistent with currentprofessionnal knowledge

• Purpose of oversight: ensure that proper structures in health care delivery and processes ensuring good qualityand measure patient outcomes in ways that enhanceimprovement efforts

Institute of Medicine (IOM)

19

Safety Culture…since the Err is human

20

Culture of

Safety

Communication patterns & language

Feedback, reward, and corrective

action practices

Formal and informal leader

actions & expectations

Teamwork processes

(e.g., back-up behavior)

Resource allocation practices

Error-detection and correction

systems

• Massachusetts General Hospital, over 7 months from2013-2014

• Drug labelling errors/ incorrect dosing/ Drug documentation mistakes/ Failing to properly respond to changes in a patient’s vital signs

« In the news: medication errors in 50% of surgeries »

21

How dangerous is health care?• Less than one death per 100 000 encounters

– Nuclear power– European railroads– Scheduled airlines

• One death in less than 100 000 but more than 1000 encounters– Driving– Chemical manufacturing

• More than one death per 1000 encounters– Bungee jumping– Mountain climbing– Health care

Pour plus de modèles : Modèles Powerpoint PPT gratuits

http://www.modeles-powerpoint.fr/

• « Strong evidence suggests that none of us are good at knowing what we don’t know. »

Davis et al. JAMA 2006

24

• « We believe that protecting the integrity of a peer-defined, discipline specific credential is not the role of the government, health care delivery systems, or payers…it belongs to those of us who practice the discipline, maintaining highly specialized knowledge and demonstrating that we have done so. »

Baron, Braddock NEJM 2016

25

• « Wide variation in rate of opioid perscribing existedamong physicians practicing within the sameemergency department, and rates of long-term opioiduse where increased among patients who had not previously received opioids and received treatmentfrom high-intensity opioid prescribers. »

NEJM 2017 Barnett et al. Opioid-prescribing patternsof emergency physicians and risk of long-term use

26

• Retrospective analysis Stanford University: more thantwo fold increased risk for ED visit or inpatientadmission for overdose

• During the past 15 years, opioid prescriptions increasednearly 3-fold and one third of fatal opiod overdoses involve also benzodiazepines

• In 2001, 9% of opioid users also used a benzo… while in 2013, 17% (relative increase of 80%)

BMJ, march 2017 Concurrent prescribing of opioids and benzodiazepines

27

Competence by Design (CBD)

http://www.royalcollege.ca/common/documents/canmeds/framework/competence_continuum_diagram_e.pdf

• How should we evaluate a medical practice…and how?• Who should evaluate the practice ?• Medical records: what aspects? How many?• Current knowledge? ( exam? Revalidation ? Recertification?)• Simulation of a frequent clinical condition ?

Let’s go back to the original question…

29

• An inspector, disconnected from the reality of the practice, « makes the call… » ?

• Random inspection process, and I may be the « luckyone » ?

• The goal of the inspection is to absolutely find problems• I heard that the success rate of remediation activities is

near 0%, so they can radiate physicians or push them to retirement

Inspection process…and urban legends ?

30

Professional inspection

Cette photo par Auteur inconnu est soumis à la licence CC BY-SA

https://fr.wikipedia.org/wiki/Craniectomiehttps://creativecommons.org/licenses/by-sa/3.0/

Knows

Knows how

Shows how

Does

Miller’s pyramid of competenceMiller G.E. The assessment of clinical skills/performance.Academic Medicine 1990; 65(9): s63-s67

do we to measure?Performance assessment:peer review, direct observation,structured oral interview

Competence assessment

Performance assessment:peer review, direct observation,structured oral interview (SOI)

Personalized or “one size fits all”?

Physicians in practice differ in terms of: specialty

demographics (gender, age)

scope of practice (types of patients, medical problems, techniques and procedures)

practice setting (e.g. hospital, private office)

clinical environment (solo or group practice)

attitudes, skills, knowledge, personality

o The Board of Directors appoints an elected physician from theBoard + 9 physicians registered on the Roll for at least 10years+ a member named by the Office. (art. 1)

o CMQ members do not participate during discussion/decisionperiod

PrésidentAdministrateur

Obstétrique-gynécologie

Chirurgie générale

Médecine de familleSLD

Responsable InspectionDr Marc Billard CMQ

Médecine de famillecabinetMédecine internePsychiatrie

Directeur DAE

Dr Ernest Prégent, CMQAdjointe administrative

Urgenceou Radiologie

Anesthésiologie ou Pédiatrie

EvaluationsVIP individuelles

Niveau 3

Programme (70/2) + 60*+ secteurs CH**

Niveau 2

VIP d’établissementsavec CMS***

Niveau 1

VIP individuelles annulées

Total

2009 146 1362 54 1562

2010 151 2363 45 2559

2011 146 2669 44 2859

2012 154 465 1632 *** 45 2296

2013 198 703 1510 *** 79 2490

2014 190 295 1806 *** 89 2380

2015 210 953* 1288 *** 93 2544

2016 207 1028* ** 4937 113 6285

2017 220 921* ** 5633 112 6886

2018 210 1257* ** 3861 103 5431

type de visites

année

MAJ 2019

MDs evaluated since 5 years

Mds visited % active MDs

level 1 15 719 75%

level 2 4454 21%

level 3 1037 5%

• Tools• Programs and results• Risk factors identified• Quality indicators

In sequence…

37

• Review records and other documents• Conduct a structured oral interview• A guided interview or direct observation• Obtain a physician practice profile • Competency assessment questionnaires• Psychometric tests

Tools that can be used

39

• Professional inspection questionnaire, call • RAMQ profile: level of activity, samples of specific

conditions• A « peer » needed?• Visit or …

Pre-visit

40

• Review of questionnaire, office, EMR• Medical records review: legibility, documentation, complaints,

histories, functional inquiries, physical exam (+ and – pertinent findings) diagnoses, investigations, results recorded, particulars of any referral

• EMR: use and tools, content, confidentiality• Equipment, registry

During the visit…

41

Physician A11 cases

9

8

7

45

10

1

2

3

6

11

Structured Oral Interview (SOI)

OSCE vs SOI

Sample case in family medicine

Evaluators Candidate

Scoring of the SOI

Score for a case =percentage of expected answers

mentioned by the candidate across all questions

Global score given by the two evaluators not consideredin the scoring of the SOI

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 5 10 15 20

Cases

G coefficients and number of casesUnbalanced (c:p) design

G Study

D Study

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 5 10 15 20Cases

95% confidence intervals for a score of 60%

(pxc) designWith 15 cases,

the margin of error is 8%.

Programs

050

100150200250

2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017Inquiries division (n=666) Professional reporting (n=545)70-y.o. program (n=294) >35 years of practice (n=221)Executive Committee (n=43) Restrictive permit (n=41)Abnormal billing (n=31) Professional claim (n=31)Locum only (n=29) Impaired physicians (n=28)Control group (n=24)

• Since 1997, the PIC has been using screening programs to target physicians for inspection

Peer review programs

2 082 peer reviews2006-2017

Relative efficiency of programs*

*Initial visits only (control visits excluded)

Efficiency of inspection programs

♦ Odds ratio95% C.I.

Programs PIC 2009-2018

60 Y.O. 23 SYNDIC (inquiries) 585

EXERCICE EN CABINET 3 SIGNALEMENT(reporting) 519

MÉTHADONE 6 D>35-70/2 357

PERMIS RESTRICTIF 34 SUIVI MD EN DIFFICULTES 15

GROUPE TÉMOIN 9 CHANGEMENT DE CHAMP 8

ÉCHEC AUX EXAMENS 2 COMITÉ DE RÉVISION 15

MD DÉPANNEUR 19 CMS + ESTHÉTIQUE 13

RÉCLAMATIONS 31 DPC 5

MD MIGRATEUR FERMÉ COMITÉ EXÉCUTIF 24

MD EXPERT 4 DIPLÔME 35+ 142

PROFILS ATYPIQUES 19 VISITES ANNULÉES 777

MAJ 2019

Levels of interventionfollowing an inspection visit

Level 2

Level 1

Level 0

Unsatisfactoryquality of practice

Satisfactoryquality of practice

Remediation prescription;limitation; retirementLevel

3

Recommendations + Control visit

Recommendations

Letter of satisfaction

Individual visits/year

18 13 12 3 8 15 7 11 5

63 60 73 72 7674

67 6476 85

2523

17 2537 30

35 2618 28

40 55 44 54

77 71 101 106114 92

120

20

40

60

80

100

120

140

160

180

200

220

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

MAJ 2019

Individual visits/yearincluding canceled visits for retirement, limitation or death

18 13 12 3 8 15 7 11 5

11

6360

7372 76 74

67 64 76 85

19

2523

1725

37 3035

26 1828

2

40 55 44 5477 71

101106 114

92

134 35 37 37 62 74 83 113 112 103

120%

20%

40%

60%

80%

100%

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 TémoinMAJ 2019

• Retrospective analysis of inspection visits* made since 2006

• Potential risk factors considered in the analysis: Demographic factors: gender, age (

Univariate logistic regression

RISK FACTOR Odds ratio(95% C.I.) p-valueRelative

risk*

Low record keeping score 8.41 (6.27-11.27) < 0.0001 3.41

Diploma outside Canada and USA 2.13 (1.53-2.97) < 0.0001 1.46

Low CPD score 2.10 (1.51-2.92) < 0.0001 1.47

Solo practice 1.59 (1.06-2.38) 0.03 1.27

Male gender 1.57 (1.03-2.40) 0.04 1.35

Age ≥ 70 1.44 (1.05-1.98) 0.02 1.24

Constant 0.10 < 0.0001

Multivariate logistic regression model (2006-2015)

*For each risk factor, the odds ratio was converted into a relative risk using the formula : , where P0 is the incidenceof non-satisfactory visits in the unexposed group (the group without the risk factor). 𝑅𝑅𝑅𝑅 = 𝑂𝑂𝑅𝑅

1− 𝑃𝑃0 + (𝑃𝑃0 × 𝑂𝑂𝑅𝑅)

Physicians with none of the model’s risk factors

(n=128)

Physicians with at least one of the

model’s risk factors (n=1 892)

4%

96%43%57%

Satisfactory (0-1)Not satisfactory (2-3)

Level of decisionfollowing peerreview

Level of decision following peer reviewversus number of risk factors

Physicians with none of the model’s risk factors

(n=128)

4%

96%

Physicians with all of the model’s risk factors (n=66)

8%

92%

Level of decision following peer reviewversus number of risk factors

Satisfactory (0-1)Not satisfactory (2-3)

Level of decisionfollowing peerreview

Effect of Age on Results of Peer Review2001 to 2014

Peer reviews cancelled were included in the calculations as non satisfactory.

Pour plus de modèles : Modèles Powerpoint PPT gratuits

Results CPD per decadevisits from 2007

23 88

197 344

316 295

4 13

51 91

146 124

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

25-34 35-44 45-54 55-64 65-74 75-99

http://www.modeles-powerpoint.fr/Graph1

25-3425-34

35-4435-44

45-5445-54

55-6455-64

65-7465-74

75-9975-99

#REF!

#REF!

23

4

88

13

197

51

344

91

316

146

295

124

Sheet1

25-34234

35-448813

45-5419751

55-6434491

65-74316146

75-99295124

Confidential ans optional evaluation of theprofessional inspection visit

Énoncé de la question COTE (1-4)1- The letter annoncing the visit was clear enough on the mechanics of the visit2- The delay between the letter and the visit was long enough3- The questionnaire was easy to fill in4- The inspector was objective5- I could make my point in different aspects6- The mechanics of the visit has reflected accurately my practice7- The visit has allowed me to enhance my medical practice

Comments :

COTE (1-4)1 – Totally at variance2 – Partially at variance3 – Partially in agreement4 – Totally in agreement 62

Revalidation is discussed around us …

63

What I would like to share…

In order to ensure the lifelong quality of their medical practice, physicians should: Maintain or improve their medical record keeping skills

Participate in CPD activities that correspond to their needs

Engage in group practice

Be attentive to signs of cognitive, sensory-motor and physical decline associated with aging and adapt their practice accordingly, or retire before health issues affect their ability to provide safe and effective care to their patients.

Questions, comments…

Pour plus de modèles : Modèles Powerpoint PPT gratuits

http://www.modeles-powerpoint.fr/

Insightful practice and competency assessment: how do we comply with the “autoregulation”…from quality assurance to quality improvementDisclosureOBJECTIVESOutline of the PresentationCollège des médecins du Québec As a baseline…A bit of epidemiologyAge distribution of active physicians% and age distribution�active 2018 vs « red line …later »Active physicians ≥ 70 yo �Jan 2019�ContextProfessional CodeBackgroundDiapositive numéro 15Diapositive numéro 16Everything is « context », right ?Groundbreaking 1989 essay Avedis DonabedianInstitute of Medicine (IOM)Safety Culture…since the Err is human« In the news: medication errors in 50% of surgeries »How dangerous is health care?Diapositive numéro 23Davis et al. JAMA 2006Baron, Braddock NEJM 2016NEJM 2017 Barnett et al. Opioid-prescribing patterns�of emergency physicians and risk of long-term useBMJ, march 2017 Concurrent prescribing of opioids and benzodiazepinesCompetence by Design (CBD)Let’s go back to the original question…Inspection process…and urban legends ?Professional inspectionMiller’s pyramid of competence�Miller G.E. The assessment of clinical skills/performance.�Academic Medicine 1990; 65(9): s63-s67Personalized or “one size fits all”?Bylaws of the Professional Inspection CommitteeEvaluationsMDs evaluated since 5 years In sequence…Diapositive numéro 38Tools that can be used Pre-visitDuring the visit…Diapositive numéro 42Diapositive numéro 43Diapositive numéro 44Diapositive numéro 45Diapositive numéro 46ProgramsDiapositive numéro 48Diapositive numéro 49Efficiency of inspection programsPrograms PIC �2009-2018Levels of intervention�following an inspection visitIndividual visits/yearIndividual visits/year�including canceled visits for retirement, limitation or deathDiapositive numéro 55Univariate logistic regressionMultivariate logistic regression model (2006-2015)Diapositive numéro 58Diapositive numéro 59Effect of Age on Results of Peer Review�2001 to 2014Results CPD per decade�visits from 2007 Confidential ans optional evaluation of the professional inspection visitRevalidation is discussed around us …What I would like to share…Diapositive numéro 65Diapositive numéro 66