Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Module One:
Maximizing CTSA Program Impact
Philip L. Lee
Clear Impact
www.clearimpact.com
CTSA Program
Common Metrics Initiative
Based on the principles of Results-Based Accountability from Trying Hard is Not Good Enough: How to
Produce Measurable Improvements for Customers and Communities by Mark Friedman (Trafford 2005)
1. The Common Metrics
2. Expectations
3. Turn-the-Curve Thinking
a. Analogy
b. Application
4. Scorecard
Module One
1. Median IRB Review Duration (Time from IRB
submission to IRB approval)
2. Pilot Funding Publications and Subsequent Funding
3. Careers in clinical and translational research (T1-T4).
The Common Metrics
Expectations
1. Define the desired impact of the CTSA Program’s support and
innovations:
A skilled CT research workforce
Improved translation: more & better treatments & cures get to
more patients faster & at less cost
Q.: How will the Common Metrics will be used?
2. Gauge the extent to which that impact is occurring:
% of K & T scholars who become CT researchers
Improved translation speed, cost, quantity & quality
Rates of innovation – develop, demonstrate, disseminate, and
uptake – in CT research workforce development and translation
The Common Metrics will not be used to:
1. Answer questions of contribution/attribution:
Why scholars did (or did not) become successful
researchers
Why translation did (or did not) improve
2. Define, assess or compare performance:
Performance can only be defined, assessed or compared
when all of the factors that influence a measure are known
and considered.
The Common Metrics will be inherently insufficient to define,
assess, or compare performance.
Expectations
Expectations
1. Align decision making across the CTSA Program:
Hub management
Network management
Q.: How will Results-Based Accountability (RBA) and
the Scorecard be used with the Common Metrics?
2. Focus decision making on the Common Metrics:
the rate of scholars going into CT research
the speed, cost, quantity and quality of translation
the rate of innovation in CT workforce development and
translation
3. Standardize the systematic and rigorous use of data and analysis
to determine what will work to “turn the curve” for the Common
Metrics
4. Make decision making collaborative and transparent
5. Share with and learn from other stakeholders at the hubs and
across the Network
Expectations
6. Keep decision making dynamic:
improve strategies on an ongoing basis, incorporate
learning/insights as they are garnered
anticipate and adapt nimbly to changing circumstances – new
challenges and opportunities
7. Trade accountability for flexibility:
Accountability focused on impact, rigorous use of data and
analysis, collaborative and transparent
Flexibility customize strategies to the unique
circumstances of a hub and its stakeholders
8. Develop, implement, and continue to manage a plan for at least
one Common Metric now … eventually for all.
Expectations
We are implementing the Common Metrics
collaboratively. Together we will learn how to
ensure the usefulness of the Common Metrics for
maximizing the impact of the CTSA Program.
Expectations
PM% of studies that achieved accrual goal
within time specified in study design
Apply to your Common Metric
1. Median IRB Review Duration
(Time from IRB submission to IRB approval)
1. Pilot Funding Publications and Subsequent Funding
2. Careers in clinical and translational research (T1-T4).
How are we doing?
Why?What are we going to do?
Who can help?Options?
Change the trend?
“Turn the Curve”?
A Disciplined Approch to Decision MakingTurn-the-Curve Thinking
Start with the End
Where are we headed?STRATEGY
Work backwards to the Means
Baseline History and Forecast
History Forecast
Turning the Curve
OK?
Return
on
Investment*
* The “ROI” is not financial, it is the change in the trend line.
13
How Are We Doing?
PM
50 %
Current Value:Q2 2016
% of studies that achieved accrual goal within time
specified in study designPM
PM
50 %
Current Value:Q2 2016
% of studies that achieved accrual goal within time
specified in study designPM
Force Field Analysis
Factors Contributing?
Factors Restricting?
The Story Behind the Curve
Root Causes (ask “Why?” five times)
Positive and negative
current and anticipated
internal and external
Prioritize – Which are the most important to
address to turn the curve?
Do we need additional data/analysis?
The Story Behind the Curve
PM
What is the story
behind the curve?
• Non-investigator clinicians not screening patients for potential study
participants during clinic consultations. (See: CTSI report)
• Lack of systems for data-driven cohort discovery in planning clinical studies
• Inadequate knowledge/skills in recruitment planning/implementation
• Study designs that impede participation
• Sites inconveniently located.
% of studies that achieved accrual goal within time
specified in study designPM
50 %Current Value:
Q2 2016
Who are partners who might have a role to
play in turning the curve?
Does the story behind the curve suggest any
new partners?
Partners
Do we know what would work to turn the curve?
Low-cost/no-cost ideas?
Off-the-wall ideas?
Research/evidence-based?
Do we need additional information?
What Would Work?
PM
Partners?
What would work?
• University/Medical Center Leadership
• University Communications School
• IT staff, Other Hubs
• Education and training department
• Develop strategies to motivate non-investigator clinicians to screen patients.
(Research: German Heart Center Munich)
• Social media (See: eRecruitmentWhitePaper.pdf)
• IT Solutions:
• Enterprise cohort discovery database.
• Cohort discovery tools.
• Provide education to inexperienced investigators in how to develop and
implement recruitment strategies.
% of studies that achieved accrual goal within time
specified in study designPM
50 %Current Value:
Q2 2016
What are our actions (in order of priority)?
Criteria
• Leverage
• Feasibility
• Specificity
• Consistent with organizational values
Strategy
PM
Strategy to
turn the curve?
% of studies that achieved accrual goal within time
specified in study designPM
Research, develop, and institute targeted strategies to
motivate non-investigator clinicians to screen patients.
Engage institutional leadership.
Develop and implement social media strategies.
Partner with the communications school.
50 %Current Value:
Q2 2016 STRATEGY
PM
55 %Current Value:
Q2 2017
% of studies that achieved accrual goal within
time specified in study designPM
PM
What is the story
behind the curve?
Targeted clinician motivation programSee EvaluationofTargetedClinicianMotivationProgram.docx
Social media campaignsSee Evaluationof2016RecruitmentSocialMediaCampaigns.docx
% of studies that achieved accrual goal within time
specified in study designPM
55 %Current Value:
Q2 2017
PM
55 %Current Value:
Q2 2017
% of studies that achieved accrual goal within time
specified in study designPM
Expand clinician motivation program to all clinicians.
Increase number of trials using social media campaigns.
Strategy to
turn the curve?
STRATEGY
1. New data
2. New story behind the curve
3. New partners
4. New information on what works
5. Changes to strategy
6. Adjourn
Network/Hub/Division/Core/Team
Agenda
1. How are
we doing?
2. What is the story behind the curve?
2. What is our strategy to turn the curve?
(c) Results Leadership Group, LLC 28
Turn-the-Curve Plan
the use of measures/data
and transparency in the
management of the CTSA Program.
but
encourages
does not
discourage
Together build a culture that
% Job Training Trainees Placed in Jobs
100%
0%
Yr1 Yr2 Yr3 Yr4 Yr5 Yr6 Yr7
What’s the story
behind the curve?
Accrual Rates
Story behind the curve?
What works?
Partners?
(c) Results Leadership Group, LLC 31
Getting to Our Best Thinking
"All the significant
breakthroughs
were
'break-withs’
old ways of thinking.”
Thomas Kuhn
Goals,Values& Strategies
From Ed Batista: Double-Loop Learning www.edbatista.com/2006/12/doubleloop_lear.htmlMore on Double-Loop Learning: www.edbatista.com/2006/10/chris_argyris_d.html
Results
Most Learning (Single-Loop)Improvement within an existing system thatrests on underlying assumptions thatare implicit and unchallenged.
UnderlyingAssumptions
Results
Most Learning (Single-Loop)Improvement within an existing system thatrests on unchallenged assumptions thatare implicit and unchallenged.
UnderlyingAssumptions
Goals,Values& Strategies
More on Double-Loop Learning: www.edbatista.com/2006/10/chris_argyris_d.html
Implicit assumptions
How to
(1) Identify
and
(2) “break with”?
The discipline of working with
mental models
starts with turning the mirror inward,
learning to unearth
our internal pictures of the world,
to bring them to the surface,
and hold them to rigorous scrutiny.
Peter Senge
The Fifth Discipline
To tell, sell, persuade,
decide
To inquire to learn
Discussion vs. Dialogue
To evaluate and select
the best
To integrate multiple
perspectives
To uncover and examine
assumptions
To justify/defend
assumptions
“I wonder which of these is
the right one?”“I wonder how these pieces
combine to create a whole?”
The purpose of a dialogue is to go beyond any one individual's understanding.
In dialogue, individuals gain insight that simply could not be achieved individually.
Suspend Judgment
Listen
Inquire*
Explore Assumptions
* Start with questions
Dialogue
The governing Values of Model I are:
• Achieve the purpose as the actor defines it
• Win, do not lose
• Suppress negative feelings
• Emphasize rationality
Primary Strategies are:
• Control environment and task unilaterally
• Protect self and others unilaterally
Model I
Usually operationalized by:
• Un-illustrated attributions and evaluations e.g.. “You seem unmotivated”
• Advocating courses of action which discourage inquiry e.g.. “Lets not talk about the past, that’s over.”
• Treating ones’ own views as obviously correct
• Making covert attributions and evaluations
• Face-saving moves such as leaving potentially embarrassing facts unstated
Consequences include:
• Defensive relationships
• Low freedom of choice
• Reduced production of valid information
• Little public testing of ideas
Model I
The governing values
of Model II include:
• Valid information
• Free and informed
choice
• Internal commitment
Strategies include:
• Sharing control
• Participation in design
and implementation of
action
Operationalized by:
• Attribution and evaluation illustrated with relatively directly observable data
• Surfacing conflicting view
• Encouraging public testing of evaluations
Consequences should include:
• Minimally defensive relationships
• High freedom of choice
• Increased likelihood of double-loop learning
Model II
Scorecard Software and the CTSA Program
Common
Metric
With Turn-the-Curve Page
Common Metrics and Performance Measures
Common Metric
Performance Measures
Performance Measures
With Turn-the-Curve Page
50
1. Watch lecture recordings and scorecard videos
2. Develop and enter your Common Metric turn-the-curve plan into
Results Scorecard.
3. Develop your own analogy to explain the turn-the-curve steps (like
Mark Friedman’s leaking roof analogy). Explain the turn-the-curve
process to a colleague using your analogy.
4. Sorting Exercise
Interim Learning Activities
Mark Friedman, Trying Hard Is Not Good Enough (Trafford,
2005)
Peter Senge, The Fifth Discipline: The Art and Practice of
the Learning Organization (Doubleday 1990); Fifth
Discipline Fieldbook (Doubleday)
Mark K. Smith, Chris Argyris: theories of action, double-
loop learning and organizational learning
(2001)http://infed.org/mobi/chris-argyris-theories-of-action-
double-loop-learning-and-organizational-learning/
Portions of these materials draw upon the work of:
Acknowledgement