Upload
christina-beasley
View
222
Download
0
Embed Size (px)
Citation preview
Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment
Kim T. Mueser
Dartmouth Psychiatric Research Center
Gary Bond: Man Without a Planet or Brother from
Another Planet?
• Innovative thinker• Math major• Beloved by students & colleagues alike• Unflappable • Force to be reckoned with in psychiatric rehabilitation• Honest, direct, modest, sensitive, humorous• Clearly not of this world, but is he without a planet or…
Fidelity: Adherence & Competence
Adherence • The extent to which the defining characteristics of a
standardized intervention are implemented in practice
Competence• The skillfulness & effectiveness with which a
standardized intervention is implemented
Potential Data Elements of Fidelity Assessments
• Record reviews• Clinician/administrator interviews• Consumer interviews• Knowledge tests• Role play tests• Direct observation• Audio/videotape observation
Methods for Assessing Fidelity Depend on:
Definition of the “treatment provider”:• Individual• Team• AgencyDefining elements of practice:• Organization & staffing (e.g., ACT team, voc unit)• Structure, aim, & scope of services (e.g., rapid job search,
services provided in community)• Clinician:consumer interactions (e.g., structure, content, & skill of
clinician working with consumer)
Practices That Have Emphasized Organization &
Structure
Assertive Community Treatment:• 1:10 staff:client ratio• Shared caseloads• Daily team meetings• Services provided in community• Direct, not brokered, service provision
Supported Employment:• Zero exclusion• Focus on competitive jobs• Rapid job search• Follow-along supports• Consumer preferences
Practices That Have Emphasized Clinician:Client
Interactions
Integrated Dual Disorder Treatment, Illness Management & Recovery:
• Psychoeducational skills• Motivational interviewing• Cognitive-behavioral teaching skills
Relevance of Clinical Skills to All Evidence-Based Practices
• Importance of the relationship between clinician & consumer
• Utility of specific skills in changing behavior or achieving specific objectives
• Differences between clinicians in their effectiveness despite adherence to organizational & structural aspects of practice
• Potential to improve quality of services through targeted training of specific skills
Assessment of Clinical Skill at Implementing a Practice
• Limited utility of interviews with clinicians or consumers as skills must be inferred
• Knowledge tests: informative about clinician’s understanding of practice, but not ability to do it
• Role play tests: informative about clinician’s ability to use specific clinical skills in simulated situations
• Direct observation: informative about clinician’s use of clinical skills during actual interactions, but may be reactive to presence of observer
• Audio/videotaped observation: informative about clinician’s use of clinical skills during actual interactions, but requires dealing with concerns about recording interactions
Advantages of Formal Assessment of Clinical Skill at
Practice• Identifies areas in need of subsequent training• Facilitates understanding of differences in effectiveness of
clinicians implementing same practice• Useful for research focused on understanding differences between
consumers in benefit from a practice• May lead to certification of clinicians in practices for which fidelity
is defined mainly in terms of the interaction between the clinician & client:
• Illness Management & Recovery• Family Psychoeducation (e.g., multi-family groups, behavioral family
therapy)• CBT (e.g., for psychosis, PTSD)
Two Examples of Clinical Skill-Based Fidelity Scales
• IMR Clinical Competency Scale
• CBT for PTSD Fidelity Scale
IMR Clinical Competency Scale
• Developed by Gingerich, Mueser, & Meyer • Initial step of brainstorming core skills necessary to implement
IMR, based on principles & specific strategies outlined in manual• Initial draft of competency items, behavioral anchors, & multiple
choice questions• Feedback on initial draft from 25 experienced IMR clinicians &
trainers• Revised draft & additional feedback obtained• Scale piloted with experienced IMR clinicians & trainers• Feedback obtained, revisions made
Overview of Scale
• 19 items, 5-point ratings, behaviorally anchored• Administered to individual providers with
experience treating at least 3 individual clients with IMR or leading/co-leading 1 group
• Ratings based on:• Observation of 2 IMR sessions • Interview with clinician, including role plays• Review of written materials
Overview of Scale (cont.)
• Administered by experienced IMR clinician, trained in implementing scale
• Requires about 6 hours to complete• Organization of scale:
• Process skills• Motivational enhancement strategies• Educational strategies• Cognitive-behavioral strategies• IMR group leader skills• IMR curriculum
Materials in the Evaluators’ Manual
1. Steps for Evaluators (in checklist form)2. Competency Items in the MN IMR Clinical
Competency Scale• Item description• Skill requirements• Primary measurement• Missed Opportunity/Comment Boxes
Materials in Manual, cont’d
3. Interview Guide• Questions for the interview section• Examples of role plays for evaluation
4. Scoring Manual• 5-point behaviorally anchored scale• Descriptors for 3 anchor points
5. Workshop materials
Steps for Evaluators Completing the Scale
1. Clinician meeting-preparation for evaluation2. Observe or listen to audiotapes of 2 sessions
(group or individual or both)3. Review written materials4. Conduct clinician interview-evaluate skills not
demonstrated in observation and follow-up any questions from observation
5. Feedback session-offer praise for strengths and suggestions for areas of improvement
11 Items to be rated using behavioral observation
• 1. Therapeutic relationship• 2. Structure of sessions• 3. Efficient use of time• 4. Motivational enhancement
strategies• 6. Goal follow-up• 7. Educational strategies
• 8. Use of reinforcement • 9. Home assignments• 11. Relaxation Training (if one of
sessions includes this topic)• 15. Group skill: Involving all
members of group• 16. Group skill: Enlisting support
between group members• 17. Group skill: Tailoring IMR
materials to experiences and goals of each group member
Items to be rated using Written Materials
• 5. Goal Setting• 6. Goal Follow-up• 10. Relapse Prevention Training• 18. Comprehensive Use of IMR Curriculum• Questions about written materials should be
followed up during the clinician interview
How to evaluate the goal tracking sheets?
• What to look for?• Recovery goal • Short-term goals-how many and are they related to the long-
term goal• Steps-how many and are they small enough• Any modifications?
• Remaining questions for Clinician Interview• See Interview Guide• Additional questions for clarification:
• Review how the client decided on the goal • Why did the client choose the short-term goals listed?
Items to be rated using the Interviewer Guide (* Role
Plays)• 11. Relaxation Training*• 12. Behavioral Tailoring and
other medication adherence strategies
• 13. Social Skills Training*• 14. Coping Skills Training• 19. Knowledge of IMR
modules
• Followed up from written materials review:
• 5. Goal Setting• 6. Goal Follow-up• 10. Relapse Prevention
Training• 18. Comprehensive Use of
IMR Curriculum
Feedback Meeting With Practitioner & Supervisor
• Conducted as soon as possible after completion of the interview
• Explain purpose of meeting is to review results of evaluation
• Give each person a copy of the Scoring Summary Sheet• Start by noting practitioner’s strengths, followed by areas
that might be improved• Briefly review ratings and the basis for them• Provide suggestions for addressing problems areas
CBT for PTSD Program• 12-16 individual sessions, for consumers with SMI & PTSD• Components of intervention:
• Orientation, crisis plan, breathing retraining (1 session)• Psychoeducation about PTSD (2 sessions)• Cognitive restructuring, taught as a skill to manage distressing feelings &
address trauma-related thoughts & beliefs (9-13 sessions)
• Supported by one RCT in SMI; multiple other studies underway in other special populations (e.g., addiction, disaster survivors, adolescents)
CBT for PTSD Fidelity Scale
• Ratings based on audio recordings• 17 items rated on 5-point Likert scales• Narrative feedback on specific ratings• Used to facilitate training & certification of
clinicians in treatment model• Used to verify fidelity to model over treatment
study
Items on CBT for PTSD Fidelity Scale
• 1. Agenda setting• 2. Homework review• 3. Overview of program• 4. Crisis plan• 5. Breathing retraining• 6. Use of Educational materials• 7. Psychoeducation• 8. Cognitive restructuring
• 9. Problem solving • 10. Trauma focus• 11. Assign homework• 12. Manual adherence• 13. Teaching effectiveness• 14. Interpersonal effectiveness• 15. Pacing & efficient use of time• 16. Reduction of client distress• 17. Overall session quality
NJ-Dartmouth PTSD Study• Kim Mueser, Dartmouth, PI• Stan Rosenberg, Dartmouth, Co-PI• Jen Gottlieb, Dartmouth, Project Manager• Steve Silverstein, UMDNJ, Site PI• Weili Lu, UMDNJ, Site Project Manager• Phil Yanos, UMDNJ, Clinical Supervisor• Stephanie Marcello, UMDNJ, Clinical Supervisor• Stephanie Dove-Williams, UMDNJ, Interviewer• Danielle Hawthorne, UMDNJ, Interviewer• Danielle Paterno, UMDNJ, Interviewer• Lindy Fox, Dartmouth, Assessment Supervisor• Rosemarie Wolfe, Dartmouth, Data Manager• Eric Slade, University of Maryland, Health Economist
Training Model
Annual Training
Weekly On-site Group Supervision
Practice Cases Assigned & Sessions Recorded
Fidelity Monitoring Via Tapes
If fidelity for sessions 4-16<3.5
Training Model: Monitoring treatment fidelity
• Frontline clinicians were provided with fidelity ratings for all sessions for their first practice cases
• If the overall fidelity ratings for sessions 4-16 was above 3.5, clinician was certified to take on protocol case
• If overall fidelity was below 3.5, then the clinician took a second practice case
• 17/20 clinicians were certified after 1 case, 3 after a second practice case
ParticipantsCompleters(n=26)
Participants(n=34)
Male 27% 29%
Age (M+/- SD) 46.8+/-10.6 45.6 +/-10.9
Race AA 27% 35%
EA 58% 50%
Hisp. & Other 15% 15%
CBT Session Progress (N=26)
Session Progress
0
10
20
30
40
50
60
1stsession
4thsession
7thsession
10thsession
13thsession
16thsession
Session Number
Score
BDI
PCL
Conclusions• Clinical skills are useful to rate for both training
purposes & to verify fidelity• Audio-based fidelity assessments are feasible to
conduct with frontline clinicians working with regular clients
• Clinical skills assessments have potential to improve other practices (e.g., ACT, supported employment)