Upload
jody-rice
View
215
Download
1
Embed Size (px)
Citation preview
Current Examples of Joint HSR/SR Evaluation
Edward J. Miech, Ed.D.Research Scientist
Health Services Research & DevelopmentVA-CASE (VISN11 VERC)
Roudebush VA Medical CenterIndianapolis, IN
July 14, 2010
VA Cancer Care Collaborative, Phase II
• 21 VA Teams Working on 5 Different Cancer Types– Lung, Prostate, HCC, Head & Neck, Colon
• SR-based Collaborative• Each team has a coach who is experienced with
SR, team dynamics, VA context
VA Cancer Care Collaborative, Phase II
• Each site receives monthly visits by an Industrial Engineer
• Learning Sessions include sessions on SR principles, concepts & tools
• Successor to VA Cancer Care Collaborative-Phase I
• Co-Chairs: Ken Klotz and Kim Voss
Team Development Measure
• CCC II planning committee interested in learning more about team dynamics
• In May 2010 (during Action Period 1) individual CCC II team members were invited to complete online version of Team Development Measure
• n=97; 18 Teams ≥ 3 members completed TDM
Team Development Measure
• 31-item instrument where respondents rate level of agreement with statements about team cohesion, communication, roles and goals
• Respondents retain anonymity• Takes 5-10 Minutes To Complete
Team Development Measure
• Examples of Statements– Team members say what they really mean. – The goals of the team are clearly understood by
all team members. – The work I do on this team is valued by the other
team members.
• 4-point Likert response scale: – Disagree Strongly, Disagree, Agree, Agree Strongly
Team Development Measure
• Permission Granted To Use Freely In VA• VA Guide To Using TDM Developed Under
Leadership of DeDe Ordin and Brian Mittman – 12-page detailed handbook, includes questions– published February 2010
• Freely Available on QUERI website at http://www.queri.research.va.gov/ciprs/TDM_Module.pdf
Team Development Measure
Preliminary Results
Team Development Measure
• Coaches Receive Customized Reports For Each of 18 CCC II Teams– Team Reports will include distribution of
individual responses for each TDM item– Visual depiction of members’ different
perceptions of current state of team
Team Development Measure
Example of TDM Customized Team Report
Number of respondents = 6Mean score = 65Lowest score = 50Highest score = 88 The graph below shows how many team members see the team at what stage.
Baseline, June 2010
INSERT SNAG IT OF TDM HERE
INSERT SNAG IT OF TDM HERE
INSERT SNAG IT OF TDM HERE
Team Development Measure
• Coaches have option to lead 1-hour facilitated discussion
• Facilitated discussion led by coach– Where team is now– What next stage looks like– How the team might get to next stage
Team Development Measure
• Data captured for TDM is quickly analyzed and reported back to CCC planning committee
• TDM Data Cross-Referenced With Other Measures
• At level of team, 1. diverse perceptions of how members see the
team leads to 2. an evidence-based discussion that3. promotes further team development and growth
Team Development Measure
• Tool = Online Instrument (Kwik Surveys)• All-Digital Data Capture and Organization– Downloads automatically into Excel
• Respondents Retain Individual Anonymity – Allows for candid responses to questions
• Multi-Level Data– Tracked at both individual and team level– Summary Reports Useful at both Collaborative
and Team levels
PCMH Collaborative LS1 - West
• Learning Session with approximately 300 participants taking place right now in San Diego
• 300-clicker ARS system in use to capture baseline and endpoint data (i.e., pre- and post-)
Baseline ARS at PCMH LS1 - West
Selected Items
How many collaboratives have you participated in, including this one?
1 2 3 4 5 6
41%
36%
10%
4%3%6%
1. 12. 23. 34. 45. 56. 6 or more
How many years have you worked for the VA?
1 2 3 4 5 6 7
7%
16%18%
8%
16%
14%
21%
1. < 12. 1 - 23. 3 - 54. 6 - 105. 11 - 156. 16 - 257. 26 +
What is your job title?
1 2 3 4 5 6 7 8 9 10
18%
1%
8%
26%
9%
2%3%
18%
6%
8%
1. Administrative Assoc2. HPDP/BHC3. LVN/LPN4. Nurse5. PCMH Faculty & Staff6. Pharmacist7. Physician8. Social Worker9. Technician/MA10. VA Faculty Leader
How many VA project-based teams (QI, Systems Redesign, Lean, etc.) have you personally been a
member of, including this one?
1 2 3 4 5 6
26%
20%
29%
3%
8%
15%
1. 12. 23. 34. 45. 56. 6 +
Did you complete the prework before you came to LS #1?
1 2 3 4 5 6
54%
15%
4%2%
10%
16%
1. Yes, all2. Yes, most3. Yes, some4. No5. Did not know about
pre-work6. N/A
I had enough knowledge about the tools available to complete the prework.
1 2 3 4 5
12%
34%
5%
18%
31%1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree
I had enough resources, including time, to complete the prework.
1 2 3 4 5
7%
20%23%
41%
10%
1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree
Exploratory Data Analysis: Correlations
• “How many Collaboratives have you participated in?” and– How many project-based teams have you been
on? (r = .55, p<.0001)– Did you complete the LS1 pre-work? (r = .21
p<.001)
ARS data from CCC II LS1-New Orleans
• Statistically significant correlations between participation in any prior VA Collaborative and– Higher levels of self-reported familiarity with
Systems Redesign tools (r=.59, p>.0001)– Higher levels of perceived support from site
leadership for Collaborative work (r=.42, p<.02)– Higher letter grades given by participants to LS1
(r=.41, p<.02)
Common Features: ARS & Online TDM
• All-Digital Capture (with built-in reporting features)
• Respondents retain anonymity• Responses can be tracked at individual level• Data can be segmented, aggregated, analyzed
for correlations, etc.
VA Yellow Belt Training Program
• In VA Yellow Belt training program participants actively learn about Systems Redesign and Lean principles, concepts, and tools through simulations, group application exercises, didactic presentations, and discussion
• Enrollment capped at 50 participants per session • Courses usually taught by faculty from the
Purdue PharmaTAP program • Sessions generally take place for 3.5 consecutive
days in an off-site location such as a hotel
VA Yellow Belt Training Program
• Over 400 participants in VA Systems Redesign Yellow Belt training across 12 different sessions held between August 2009 (pilot in Indianapolis) and June 2010 (Boston, MA)
• Pre- and post-data collected at 11 of these 12 sessions
Example: Interim YB Findings
Example: Interim YB Findings
• Analyses found extremely strong correlations between:– the items “I believe this Yellow Belt workshop will
be relevant to my job” and “I am positive about Systems Redesign” (r=.68, p<.0001);
– the items “I found this workshop to be practical” and the overall letter grade given to the workshop (r=.67, p<.0001)
Example: Yellow Belt in Connecticut
• I am familiar with many of the tools used in Systems Redesign. (n=34)
ARS Advanced Features
• Multi-Voting• Priority Ranking – 1st choice =10 points– 2nd choice =5 points– 3rd choice= 1 point
• Impact-Effort Matrix– Graph two dimensions along x-axis and y-axis
• Conditional Branching
Next Frontier
• Capturing Multi-Level Data From Complex, Multi-Site Initiatives in a Single Project File
• Flexibly Interrogating/Analyzing Evidence– Keywords– Codes– Indexing
Next Frontier
• Adding Value to Source Data with – Applying codes and tags– Linking to Analytic Memos– Performing Context Matrix Analyses
• Rapid Reporting of findings/results to initiative organizers/planners on rolling basis
• Capturing, Organizing, Analyzing and Reporting Rich-Context Data occurs on ongoing, rolling basis
• Digital Tool = NVivo8
Acknowledgements
Heather Woodward-Hagg– Systems Redesign, Chief, Roudebush VA– Co-Director, VA-CASE (VISN11 VERC)
Mary Sherrill– Co-Director, VA-CASE (VISN11 VERC)
Dawn Bravata– Co-Coordinator, Stroke QUERI