Multisource feedback & its utilityA N E P I C D E B A T E

D R . J O A N S A R G E A N T

D R . K A R E N M A Z U R E K

Resolution

"Be it resolved that based on the evidence that MSF can lead to changes in physicians' behaviour, that MSF should be an essential component of any system of revalidation"

MSF – The Scientific Evidence

J O A N S A R G E A N T P H D

P R O F E S S O R A N D H E A D , D I V I S I O N O F M E D I C A L E D U C AT I O N

D A L H O U S I E U N I V E R S I T Y, H A L I FA X , N S , C A N A D A

I A M R A M O N T R E A L O C T O B E R 2 0 1 5

In Canada, PAR - Continued Development of Suites of InstrumentsAnesthesia

Surgery,

Medicine, Pediatrics, Psychiatry

Episodic Care

Laboratory Medicine

Radiology

Family Medicine (Revised)

Anesthesia (In process of being revised)

7

Evidence in support of PAREach suite was examined for psychometric properties -◦ Descriptive data on each item◦ Range, means (s.d.), unable to assess

◦ Reliability◦ G-study/ Cronbach’s alpha

◦ Factor analysis◦ Association between scores and ◦ Sociodemographic variables◦ Rater familiarity

About 20 scientific publications◦ Violato/Lockyer/Fidler and others

Validity

There is a body of evidence that is coherent (hangs together) and that supports the use of the results of an assessment for a particular purpose.

Items and scales are developed through focus groups with targeted stakeholders

Items correlate in intended ways producing fairly robust factors (scales)

Scores increased on 2nd iteration for colleague and co-worker

ReliabilityThe results of the assessment would be the same if repeated under similar circumstances.◦ Stability of the instrument

Factor analysis/ Cronbach’s alpha for instruments and scales >0.8

G-studies: Ep2 approximately 0.7 for most instruments with 8 medical colleagues & co-workers (15-40 items) and 25 patients with 15-40 items

Note - High stakes Ep2 > 0..8

10

Utility: Changes to practiceChanges contemplated or initiated based on feedback(Violato et al, Acad Med, 1997; Hall et al, CMAJ 1999; Fidleret al, Acad Med 1999; Sargeant 2003; Overeem 2007)◦ About 50% of MDs report making changes in practice◦ Changes: specific aspects of care; communication with

patients, colleagues and co-workers; stress management◦ Changes requiring support or expenditures less likely ◦ Change is influenced by multiple factors; e.g, perceptions

of data credibility; fairness of process; specificity of data; beliefs about change or ability to effect change

◦ Limitations: self-report

11

So wouldn’t you agree…..

Lots of evidence from PAR?

More evidence - Systematic review of MSF studies (Violato et al 2014)

A systematic literature review for English-language studies (1975 to 2012) was conducted.

Search parameters: multisource-feedback, 360-degree evaluation, assessment of medical professionalism

48 studies (Canada, US, UK, NL, China and elsewhere met the inclusion criteria)

Domains Assessed by MSF•Professionalism

•Clinical competence

•Communication with patients and families

•Case and office management

•Collegial relationships

•Best for humanistic domains that can’t be assessed in other ways (Archer et al 2005, 2006; Overeem 2010)

ConclusionsMSF has evidence of reliability, validity and feasibility and can ◦ assess core competencies of physicians ◦ identify strengths and weaknesses within competencies ◦ provide feedback for professional development

Validity-Recruiting reviewers1. Ramsey e tal, JAMA 1993 – ratings similar for physician and

supervisor selected reviewers

2. Archer and McAvoy, Medical Education 2011 – ratings were higher for physician-selected reviewers than referring-body selected reviewers (high-stakes assessment)

16

Validity- factors influencing reviewers’ MSF assessmentsCampbell et all. BMJ, 2011 -◦ Country primary medical qualification◦ Locum status◦ Clinical specialty◦ Contractual role◦ Familiarity (% reporting ‘daily’ or ‘weekly’ contact ↑)

Sargeant et al, Acad Med 2003, Med Ed 2007 –◦ Familiarity ↑◦ Those less familiar selected “unable to assess” more frequently◦ “ Raters default to ‘3’ (average) if they don’t know you”◦ “need to be able to observe someone to assess them”

17

Factors influencing feedback acceptance and use

Physicians are more satisfied with feedback that includes narrative comments (Overeem et al., 2010)

Physicians need specific feedback to guide them in improvement (Sargeant et al 2007, 2008)◦ “being a ‘4’ doesn’t tell you what to do to improve”◦ need narrative for specific guidance on “how” to improve

18

Factors influencing feedback acceptance and use

Agreement with MSF report is positively correlated with scores (Sargeant 2003, 2008)

Impacts of this -◦ Low scores can cause distress and result in demotivation

(Sargeant 2008, Overeem 2010)◦ Facilitated reflective feedback can enhance feedback

acceptance (Denisi, Kluger 2000; Goodstone, Diamante 1998; Sargeant et al 2015)

19

What are you doing with THIS evidence??

MSF- The Practical IssuesKAREN MAZUREK

DEPUTY REGISTRAR

COLLEGE OF PHYSICIANS AND SURGEONS OF ALBERTA

Why PAR?believe that all physicians can benefit from continuous quality improvement

Wanted to build a culture of CQI

Did not want a hunt for “bad apples”Public believed it was already being done

22

How does PAR work?

Every physician, once every 5 years

25 patients, 8 colleagues, 8 co-workers, self-assessment

Report gives results and comparisons; how to use results; how to claim CPD credits

3-month follow-up feedback

23

PAR Survey

Self-Analysis

Response

PARCommittee

Review

IndividualizedAssessment

Advice toPhysician

Feedback

Office Peer Review

CompetenceAssessment

Diagram 1

The College’s PAR Program

Participation in PAR1999-2014 totals

GP/family physicians = 5,546Surgeons = 1,600Anesthesiologists = 571Medical specialists = 1,940Psychiatrists = 754Pediatricians = 563Episodic care = 1,122Diagnostic Imaging = 494Lab Medicine = 256

Total to date = 12,846

25

Raw scores

Communication

Humanistic

Office mgt.

Telephone

Staff

4.2 4.4 4.6 4.8 5.0

PAR Results

communication

humanistic

office mgt.

telephone

staff

T-Score Format

Presentation of feedbackSargeant et al, Academic Medicine 2003◦ Physicians who receive high scores agree with colleague feedback; those who receive low

scores are neutral or disagree with results.◦ Facilitation of feedback should be considered

Eva KW, Regehr G. Commentary. CMAJ, 2013

◦ “Cognitive dissonance is the discomfort created by trying to maintain 2 conflicting beliefs at the same time”

◦ “Easier to question the data than to question oneself, this pair of conflicting beliefs will often be resolved by discounting the feedback rather than altering one’s sense of self*”

*Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract 2012;17:15–26.)

Self selection of ratersRamsey et al, JAMA 1993

– ratings similar for physician and supervisor selected reviewers

Archer and McAvoy, Medical Education 2011

– ratings were higher for physician selected reviewers than referring-body selected reviewers (high-stakes assessment); self selection of raters should end

OutcomeViolato, C, Lockyer, JM, Fidler H, British Medical Journal 2003◦ “Many surgeons in this study used the feedback to

contemplate or initiate changes to their practice”.

Lockyer, J, Violato C, Fidler H, Teaching and Learning in Medicine 2003◦ “Surgeons made few changes in practice in response to feedback data”

Ferguson et al, BMC Medical Education 2014◦ systematic review found 16 studies in the World literature looking at MSF

and change in physician behavior - only one found a measurable improvement.

PAR Feedback% of respondents who report making at least one practice change

51% Jan to June 2013

48% July to Dec 2013

54% Jan to June 2014

62% July to Dec 2014

56% Jan to June 2015

Cost of PAR$200 per physician; $40 per year

2013/2014 financial data - $600 to 700 per physician; $120 to $140 per year

3x the cost!!!

Alberta survey - 2015Sent surveys to all physicians/surgeons in Alberta (not provisional licenses)

How good is existing program (PAR)?

2215 responses out of 9021 = 25% response rate

Previous CPSA record was Physician survey 2014 with RR of 14%. Typically <10%

99% probability that these results are within 2.4% of actual true value

Please rate how successful the existing Physician Achievement Review (PAR) Program is in assessing the following dimensions. (1-10 where 1=not at all, 5=fairly, and 10= extremely)

303 Text responses: Substantially negative – we grouped into 8 key themes

“Too easy to cheat. Probably little use to 90% of profession”

“To me the PAR is an incredible waste of time and money. I do not know one MD who actually pays it much attention, either filling it out or implementing changes” “…Patients uniformly complain only about waiting times in parking and really have little useful to say about the quality of care that is delivered. Comments from peers as well as other physicians in the community are uniformly positive and there is never anything in the way of constructive criticism or useful feedback…”

IrrelevantSubjective, biasedEasy to cheatDiscriminatoryConfusingTime-consumingRepetitiveWaste of money

Why Canadian regulators would consider abandoning ParThe impact on the physician is not what we hoped

After 15 years we don’t know that it improves care for patients

It costs too much money

Physicians do not like it

Questionswhose side are you on?

are you using MSF now?

what do you see as its strengths and weaknesses?

how do you see its role in assessment and revalidation?