Clinical Skills Verification Rater TrainingMODULE 3

Setting the Clinical Skills Verification Examination Climate and Giving Feedback

Karen Broquet , M.D.Eugene Beresin, M.D.

Nyapati Rau , M.D.

Module 3Pre-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

b. Clinical Formulationc. Differential Diagnosisd. Treatment Planning

Module 3Pre-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

a. Arrange Mock Oral Board-style evaluationsb. Use Standardized Patientsc. Perform the exercise during routine workd. Make use of the Hawthorne Effecte. Add a multiple choice test following the interview

Module 3Pre-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

a. Any rating form can be used

b. Process notes provided by the residentc. Faculty should use a rating form he/she most prefersd. Faculty should all use the same rating form

Module 3Pre-test

4. Which of the following is considered formative assessment?

a. Year-end reviews of performance

b. Pass/fail examinationsc. Specific feedback of behavior in real timed. Anonymous 360 degree evaluations sent to the residente. Periodic summative evaluations

CSV Exam MUST Assess…

• Ability to establish doctor-patient relationship

• Conduct a clinical interview• Oral presentation of patient history

and clinical findings

But can (and should) ALSO be….

• Valuable educational experience• Opportunity for

• Direct observation of clinical skills by faculty

• Formal evaluation of clinical skills

• Focused feedback

Feedback is Formative

Process by which the teacher provides the learner with information about his/her performance for the purpose of improving performance (Ende 1983)

• Without specific and focused feedback, learners:• Rely on self-assessment (unreliable!)• May establish undesired behaviors and/or

extinguish desired behaviors

Evaluation is Summative

“Did I Pass the exam!?!”

Feedback = Coach

• Timely, immediate, ongoing

• Direct• Verbal• Face to face• Guiding future

performance

Evaluation = Judge

• Summative• Not ongoing,

occurs at the end of time together

• Assesses whether or not learner met performance standards

Quality of both feedback and evaluation depends on the observer’s detection, perception and recall of the resident’s

performance

How to maximize this:• Incorporate direct observation into clinical

teaching routine• Brief (daily rounds) and not so brief (CSV exam)• Normalizes observation/assessment experience• Hawthorne effect: the more we know we are

assessed, the better we perform• Write it down!

• Specific, accurate feedback is better feedback• Also helps with accurate evaluation

• Prepare for the encounter

Setting Expectations

• How long will the encounter be?• Who will be observing?• What are the goals of the encounter?

• i.e., is this a formal CSV evaluation?• Assessment parameters?• What rating instrument/form will be used?• When will “results” be given?• What are the consequences of “passing” or

“failing” a CSV exam?

What Rating Form/Instrument to Use?

• Multiple forms accepted by the ABPN

• Forms vary by level of detail• All faculty/CSV raters in a program

should be trained on and use the same form

How Long Should the CSV Exam Be?

• ABPN: “about” 30 minutes for interview and 15 minutes for presentation

• AADPRT CS Assessment Task Force: 30-45 minutes for interview, 60-75 minutes total

• Programs should choose one format and use consistently

Guidelines for giving effective feedback – whatever the setting/encounter

Effective/Meaningful Feedback is…

• approached with teacher and learner working as allies with common goals

• given immediately in a private environment• well timed and expected• based on first-hand (observed) data• focused on specific behaviors, not

generalizations• limited to behaviors that are remediable

Effective/Meaningful Feedback is…

• regulated in quantity (no more than 2-3 items for improvement)

• limited to actions and decisions, not assumed intentions or interpretations

• phrased in descriptive, nonjudgmental language

• closed with suggestions for improvement and a mutually agreed upon action plan

Barriers to Effective Feedback

Faculty• Time• Lack of feedback skills• Feels like being critical• Want learners to like us • Gestalt vs specific

observations• Impact of CSV ‘failure”

Resident• Time• Lack of feedback

experience • Feels like being criticized• Don’t want to expose

weaknesses

Residents CRAVE feedback

To minimize barriers, faculty can…

• Understand the profound and beneficial educational value of feedback

• Provide the proper climate and preparation for the experience

• Elicit reciprocal feedback from trainees• Gain the skills for giving both constructive

and reinforcing feedback• Use a consistent, common rating instrument

Cultural Phenomena – International Medical Grads

• Lack of feedback Experience• Diverse cultural backgrounds• Power dynamics between teacher and learner• Accepted communication patterns• Attitudes towards critique• Vulnerability in evaluation • Stress and difficult social situations • Experience with discrimination

Important for all learners – especially IMGs

• Learner centered approach• Explicitly prepare the resident for the

expectations of the exam and the provision of feedback

• The importance of the teacher and the IMG working together cannot be over emphasized.

• Create a safe learning environment and clear expectations and educational objectives

Important for all learners – especially IMGs

• Invite learner to indicate his/her learning needs and provide a self-assessment.

• Normalize feedback as regular educational activity that includes areas to be reinforced and areas requiring improvement.

• Be clear when feedback is formative or summative (i.e evaluation)

Module 3Post-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

b. Clinical Formulationc. Differential Diagnosisd. Treatment Planning

Module 5 Post-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

Diagnosis, biopsychosocial formulation and treatment planning are not assessed

Module 5Post-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

a. Arrange Mock Oral Board-style evaluationsb. Use Standardized Patientsc. Perform the exercise during routine workd. Make use of the Hawthorne Effecte. Add a multiple choice test following the interview

Module 5Post-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

c. Perform the exercise during routine work

Module 5Post-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

a. Any rating form can be used

b. Process notes provided by the residentc. Faculty should use a rating form he/she most prefersd. Faculty should all use the same rating form

Module 5Post-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

d. Faculty should all use the same rating form

Although the ABPN has approved a variety of different rating forms for the CSV exam, consistent use of and feedback regarding the same form by all raters in a program is associated with improved inter-rater reliability

Module 5Post-test

4. Which of the following is considered formative assessment?

a. Year-end reviews of performance

b. Pass/fail examinationsc. Specific feedback of behavior in real timed. Anonymous 360 degree evaluations sent to the residente. Periodic summative evaluations

Module 5Post-test

4. Which of the following is considered formative assessment?

c. Specific feedback of behavior in real time

Annual performance reviews, 360 degree evaluations and pass-fail examinations are all types of summative assessment.