Charles E. Cunningham, Ph.D.Professor
Department of Psychiatry & Behavioural Neurosciences
Jack Laidlaw Chair in Patient-Centred Health Care
Michael G. DeGroote School of Medicine
Faculty of Health Sciences
McMaster University
Modeling Implementation and Knowledge Translation Strategies Using
Discrete Choice Conjoint Experiments
Goals of the Talk
Brief Introduction to Preference Modelling
Discrete Choice Experiments
Highlights from Studies of KTE Preferences
Addiction Professionals
Children’s Mental Health Professionals
Educators
Implications?
Why Study Implementation?
“Multiple studies across differing treatments found that training (workshop, manual, and supervision) was not enough to produce proficient change in therapist adherence, competence, and skill. In turn, Therapists were not necessarily able to engender client change.”
Source: Biedas and Kendall (2010). Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clinical Psychology Science and Practice 17, 1-30.
For treatments to be effective and sustained in practice settings, treatment developers should consider design features that increase the appeal to the therapists who are ultimately responsible for using them.
Source: Journal of Consulting and Clinical Psychology, 2013, 81, 6, 999-1009,
Modeling the Implementation of School-based Mental Health Strategies
CIHR Team in Access to Children’s
Mental Health Services
Principal Investigator Melanie Barwick (University of Toronto)
Co-Investigators (Selected) Kathy Short (Hamilton-Wentworth District School Board
Research Staff Heather Rimas Stephanie Mielko Jenna Ratcliffe Cathy Campbell (Team Secretary) Yvonne Chen (McMaster HRM Program)
Funding Canadian Institutes of Health Research Jack Laidlaw Chair in Patient-Centred Health Care
Discrete Choice Conjoint Survey
Sample Size = 1010 Educataors
Return Rate = 82%
17 4-Level KT Attributes
Partial Profile Design
17 Choice Tasks Per Informant
999 Near Orthogonal Versions of the Survey
Source: Cunningham, Barwick et al. School Mental Health (2014)
Assume you are considering changing your teaching practice to improve the behavioral and emotional outcomes of students:
Click below the practice change strategy you would prefer:
Next
Strategy 1 Strategy 2 Strategy 3
Learning requires 2 days
My colleagues endorse this 67%
Other schools say this strategy works
Learning requires 1 day
My colleagues endorse this 100%
This strategy is promising but
unproven
Learning requires 3 days
My colleagues endorse this 33%
Research says this strategy works
Note: 999 versions – 1 randomly assigned to each participant
Partial Profile Designs
Reduce Effect of Dominant Attributes
Reduce Response Error
Reduce Total Design Error
Produce Similar Utility Coefficients
Improve Predictive Validity of Simulations
(Reduce MAE)
Source: Chrzan, K. (2010). Using partial profile choice experiments to handle large numbers of attributes. International Journal of Market Research, 52, 827-840.
Data Analysis
Hierarchical Bayes or Conditional Logit
Importance Scores (Relative Influence on Choices)
Zero Centered Utility Coefficients (Strength of Preference)
Latent Class Analysis
Multi-level Latent Class Analysis
Randomized First Choice Simulations
Source: Hauber, B., Gonzalez, J., Groothuis-Oudshoorn, C., Prior, T., Marshall, D.,Cunningham, C., . . .
Bridges, J. (2016). Statistical methods for the analysis of discrete-choice experiments: A report of the ISPOR
conjoint analysis good research practices task force. Value in Health, 19(4), 300-315.
Why Use Discrete Choice Experiments?
Approximate Complexity of Real World Choices
Each Level Experimentally Manipulated
Each Attribute in the Context of Others
Complexity Activates Decision Making Heuristics
Users Inform Implementation Planning Tradeoffs
Reduce Influence of Social Desirability Biases
Segments with Different Preferences? Latent Class Segmentation Analysis
Demand Sensitive (23%)
Change Ready (77%)
Source: Cunningham, Barwick et al., School Mental Health (2014)
Covariates in Latent Class Analysis
Can be Linked to Latent Class Membership
Can be Included in the Formation of Latent Classes
Improve Model Fit
Enhance the Interpretation of Segment Membership
But - Increase Number of Parameters Estimated
What Attitudes Distinguish Change Ready vs Demand Sensitive Segments?
Change Ready Educators:
Anticipate More Benefits to Practice Change
Find the Social Context to be More Influential
Report Higher Change Self Efficacy
More Intent on Participating
Relative Importance of KT Design Attributes
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttribute
Attributes of the Implementation
Process
Sensitivity to Variation in an Attribute’s Levels
Importance= Variation In Each Attribute’s Utility/Total Utility Variation
Attributes of the Social Context Influencing the Decision to Adopt Mental Health Practice Changes
Relative Importance of KT Attributes: Colleague Support for Change
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
enta
tio
n A
ttrib
ute
Demand Sensitive Change Ready
Support of Colleagues for Change Exerts an Important
Influence
Source: Cunningham, Barwick et al. School Mental Health (2014)
Colleague Support
-2
-1.5
-1
-0.5
0
0.5
1
1.5
0% Support
this Option
33% Support
this Option
67% Support
this Option
100% Support
this Option
Utility
Valu
e
Change Ready Demand Sensitive
*
Both Segments Prefer Programs
Supported by 100% of Colleagues
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Design Attributes: Administrative Support
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttribute
Demand Sensitive Change Ready
Administrative Support
Important
Source: Cunningham, Barwick et al., School Mental Health (2014)
Support by Administrators
-100
-80
-60
-40
-20
0
20
40
60
80
Does Not
Support
33% 67% 100%
Utility
Valu
e
KT Choices of Both Segments Sensitive to Administrative Support
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Design Attributes: Union Endorsement
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttribute
Demand Sensitive Change Ready
Union Endorsement Exerted a Strong Influence on Choices
Source: Cunningham, Barwick et al. School Mental Health (2014)
Union Endorsement
-2
-1.5
-1
-0.5
0
0.5
1
1.5
Does Not
Support
33% Support 67% Support 100% Support
Utility
Valu
e
Change Ready Demand Sensitive
*
Decisions of Both Segments Sensitive to Union Support
Source: Cunningham, Barwick et al. School Mental Health (2014)
Attributes of Mental Health Practices that Influence Implementation Decisions
Relative Importance of KT Attributes: Compatibility with Practice
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
enta
tio
n A
ttrib
ute
Demand Sensitive Change Ready
Compatibility with Teaching Practice Highly
Important
Source: Cunningham, Barwick et al. School Mental Health (2014)
Compatibility with Teaching Practice
-2
-1.5
-1
-0.5
0
0.5
1
1.5
0% 33% 67% 100%
Utility
Valu
e
Change Ready Demand Sensitive
Educators More Likely to Choose Mental Health Practices
Consistent with Teaching Practice
Source: Cunningham, Barwick, et al., School Mental Health (2014)
Relative Importance of KT Design Attributes: Supporting Evidence
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttribute
Demand Sensitive Change Ready
Supporting Evidence
Moderately Strong Importance
Source: Cunningham, Barwick et al. School Mental Health (2014)
Supporting Evidence
-1.5
-1
-0.5
0
0.5
1
Promising butUntested
Other Schools Saythis Works
Research Says thisWorks
Research & OtherSchools Say this
Works
Utility
Valu
e
Change Ready Demand Sensitive
Both Segments Prefer Options Based on Research + Experience of Other Schools
Demand Sensitive Educators Value Experience of Other Schools vs Research
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Attributes:Provincial Curriculum Links
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
en
tatio
n A
ttrib
ute
Demand Sensitive Change Ready
Provincial Curriculum Links Moderately Important to
Both Segments
Source: Cunningham, Barwick et al. School Mental Health (2014)
Links to Provincial Curriculum
-1.5
-1
-0.5
0
0.5
1
0% 33% 67% 100%
Utility
Change Ready Demand Sensitive
Source: Cunningham, Barwick et al. School Mental Health (2014)
Attributes of the Implementation Process that Influence Decisions
Relative Importance of KT Design Attributes: Qualities of Presenters
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttribute
Demand Sensitive Change Ready
Qualities of Presenters Very
Important Influence
Source: Cunningham, Barwick et al. School Mental Health (2014)
Qualities of the Presentor
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Not EngagingNor Expert
Engaging NonExpert
Non EngagingExpert
Engaging Expert
Utility
Valu
e
Change Ready Demand Sensitive
Demand Sensitive Educators Value Engaging More than Expertise
Educators Prefer an Engaging Expert
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Attributes:Coaching to Support Implementation
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
en
tatio
n A
ttrib
ute
Demand Sensitive Change Ready
Coaching to Support Implementation More
Important to the Change Ready Segment
Source: Cunningham, Barwick et al. School Mental Health (2014)
Coaching Support to Improve Skills
-2
-1.5
-1
-0.5
0
0.5
1
No Coaching If I Want If I Need It All Participants Get
Coaching
Utility
Valu
e
Change Ready Demand Sensitive
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Attributes:Workshop Size
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
en
tati
on
Att
rib
ute
Demand Sensitive Change Ready
Workshop Size Exerts a Moderately Important Influence on Choices
Source: Cunningham, Barwick et al. School Mental Health (2014)
Workshop Group Size
-1
-0.5
0
0.5
1
I Learn This Alone Group of 10 Group of 50 Group of 100
Utility
Coeffic
ient
Change Ready Demand Sensitive
Both segments prefer information delivered in small groups of 10
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Attributes: Training Time Demands
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imp
lem
en
tatio
n A
ttrib
ute
Demand Sensitive Change Ready
Training Time Demands Exert More Influence on Demand Sensitive Educators
Source: Cunningham, Barwick et al. School Mental Health (2014)
Training Time Demands
-1
-0.5
0
0.5
1
1 Day 2 Days 3 Days 4 Days
Utility
Valu
e
Number of Training Sessions
Change Ready Demand Sensitive
Both Segments prefer 1 day workshops
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Attributes:Follow-up Support
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Demand Sensitive Change Ready
Follow-up Support More Important to Change
Ready Educators
Source: Cunningham, Barwick et al. School Mental Health (2014)
Follow-up Support
-1.5
-1
-0.5
0
0.5
None 1 2 3
Utility
Valu
e
Number of One Hour Follow-up Sessions
Change Ready Demand SensitiveDemand Sensitive Educators Prefer No Follow-up Support
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Design Attributes: Selection Process
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Demand Sensitive Change Ready
Selection Process of Relatively Low Importance
Source: Cunningham, Barwick et al. School Mental Health (2014)
KT Selection Process
-0.5
-0.25
0
0.25
0.5
Ministry Selects Board Selects Individual Schools
Select
Individual Teachers
Select
Utility
Valu
e
Change Ready Demand Sensitive
Source: Cunningham, Barwick et al. School Mental Health (2014)
Relative Importance of KT Design Attributes: Internet Options
0 1 2 3 4 5 6 7 8 9 10 11 12
Presentor Qualitites
Colleague Support
Compatibility
Administrative Support
Union Endorsement
Coaching
Supporting Evidence
Provincial Curriculum Links
Workshop Size
Skills vs Knowledge
Follow-up Support
Observability Trialability
Training Time Demand
Selection Process
Universal vs Targeted
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Demand Sensitive Change Ready
Internet Options Exert Little Influence on Choices
Source: Cunningham, Barwick et al. School Mental Health (2014)
Internet Training Options
-1
-0.5
0
0.5
1
No Internet Options Moderated Internet
Discussion
Internet Learning +
Moderated Discussion
Internet Learning
Activities
Utility
Valu
e
Change Ready Demand Sensitive
Source: Cunningham, Barwick et al. School Mental Health (2014)
Limited Interest in Internet Options
Randomized First Choice Simulation
Standard Dissemination 1 Day Large Group (N=50) No Internet 67% Focus on Knowledge No Coaching or Follow-up
Enhanced Dissemination 3 Day Small Group (N=10) No Internet 67% Focus on Skills Coaching and 3 One Hour Follow-ups
Internet Dissemination 3 Day Individual Learning Internet Learning and Moderated Discussion 67% Focus on Skills No Coaching but 3 Internet Follow-up Sessions
Source: Cunningham, Barwick et al. School Mental Health (2014)
0
20
40
60
80
100
Standard
Dissemination
Enhanced
Dissemination
Internet
Dissemination
1.4
98.4
0.3
53.3
44.8
1.9
Perc
ent
Change Ready
Demand Sensitive
Randomized First Choice Simulation
Source: Cunningham, Barwick et al. School Mental Health (2014)
Is this Initiative Supported by My Principal, Colleagues, and Union?
Is this Compatible with My Practice & Consistent with Provincial Curricula?
What is the Evidence that This Works?
Does this Work for Other Schools?
Is It Supported by Research?
Is the Presenter an Engaging Expert?
Is this Consistent with My Learning Preferences1 Day, Small Group (n=10), Skill Focused, Supported by Coaching,1 to 2 Follow-ups
Multi-Stage Implementation Decisions
Modeling Evidence-based Practice Dissemination in Addiction Agencies Serving Women
SELECTED PROJECT PARTNERS
Members of Research Team Joanna Henderson Alison Niccols Maureen Dobbins Wendy Sword Karen Milligan Ellen Lipman Lehana Thabane Louis Schmidt
Research Team Heather Rimas (Research Coordinator) Stephanie Mielko (Research Assistant) Yvonne Chen (HRM Graduate Student) Ainsley Smith
Research Support Canadian Institutes of Health Research Jack Laidlaw Chair in Patient-Centred Health Care
Discrete Choice Conjoint Survey
Sample Size = 1379
Return Rate = 60%
16 4-Level KT Attributes
20 Choice Tasks Per Informant
999 Versions of the Survey
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Latent Class Segmentation Analysis
Process Sensitive (30%)
Outcome Sensitive (52%)
Demand Sensitive (18%)
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation Attributes
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
KT Choices Most Sensitive to Percentage of Clients Benefiting
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation Attributes
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
Compatibility of KT Content Important
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation Attributes
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
Participants Sensitive to the Support of Colleagues &
Administrators
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation & Exchange Attributes to Professionals
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
Attributes of the KT Process Exerted Less Influence on
Choices
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation & Exchange Attributes to Professionals
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
Content Selection Process of Moderate Importance
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
-60
-40
-20
0
20
40
60
Government Administrators Me Co-Workers & I
Content Selection Process
Utility
Valu
es
Outcome Process Demand
Content Selection Process
Participants Prefer Local Collaborative Content
Selection
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Relative Importance of Knowledge Translation Attributes: KT Presenter
0 2 4 6 8 10 12
Benef it t o Client s
Compat ibilit y
Evidence
Ease of Implement at ion
Administ rat ive Support
Co-Worker Support
Cont ent Select ion
Present or Credent ials
Push Pull
Knowledge vs Skills
Endorsement
Modalit y
Act ive Learning
Group Size
Implement at ion Support
Time Cost
KTE A
ttri
bute
s
Importance Score
Credentials of KT Presenter Important
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
-60
-40
-20
0
20
40
60
Administrator Former Client Researcher Clinician
Presenter
Utility
Valu
es
Outcome Process Demand
KT Presenter
Participants Prefer Presenters Who are Clinicians vs
Researchers
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Simulating Response to KT Options
Standard KT
1-Day Time Committment
Large Group Format
Focus is 67% Knowledge 33% Skill
Easy to Apply
No Review Questions or Practice
No Implementation Follow-ups
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Enhanced KT
4-Day Time Commitment
Small Group Format
Focus is 67% Knowledge 33% Skill
Difficult to Apply
Review Questions and Practice
3 Implementation Follow-ups
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Simulating Practice Change Options
66.3
44.3
81
33.7
55.7
19.1
0
20
40
60
80
100
% S
hare
of Pre
fere
nce
Standard Enhanced
Outcome Process DemandShare of Preference is the Percentage of
Participants Predicted to Prefer the Standard or Enhanced KT Strategies
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
62.5
29.1
7.5
24
37.5
70.9
92.5
76
0
20
40
60
80
100
Baseline Enhanced + Evidence Enhanced +
Evidence+Colleague +
Administrative Support
Government Selection &
Endorsement
Standard
Enhanced
Percent Predicted to Prefer Enhanced KT
Source: Cunningham, Henderson, Niccols, et al., Addiction (2012)
Modeling Implementation of EBPs in Children’s Mental Health Agencies
CIHR Team in Access to Children’s
Mental Health Services
Principal Investigator Melanie Barwick (University of Toronto)
Selected Research Staff Heather Rimas Stephanie Mielko Yvonne Chen (McMaster HRM Program) Raluca Barac
Funding Canadian Institutes of Health Research Jack Laidlaw Chair in Patient-Centred Health Care
Discrete Choice Conjoint Survey
Sample Size = 563 Mental Health Service Providers
31 Agencies
14 4-Level KT Attributes
18 Choice Tasks Per Informant
Partial Profile Design
999 Versions of the Survey
Source: Cunningham, Barwick et al., Administration and Policy in Mental Health (2018).
Segments with Different Preferences? Latent Class Segmentation Analysis
Accelerated Adopters (12%)
Gradual Adopters (88%)
Source: Cunningham, Barwick et al., Administration and Policy in Mental Health (2018).
Attributes of the Social Context Influencing the Decision to Adopt Mental Health Practice Changes
Relative Importance of KT Design Attributes: Supervisor Support
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Supervisor Support for EBP
% Clients Benefiting
Skill vs Knowledge
Colleagues Support EBP
Initial Training on Internet
Engaging Expert Trainer
Active vs Passive Learning
Follow-up Trainin
Effectivenes in Other Agencies
Group Size
Initial Training Time
Observability Trialability
Selection Control
% Change in Practice
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Gradual Adopters Accelerated Adopers
Accelerated Adopters Most Sensitive to Variation in
Supervisor Support for EBP
Source: Cunningham, Barwick et al. Administration and Policy in Mental Health (2018)
Relative Importance of KT Design Attributes: Trainer Qualitites
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Supervisor Support for EBP
% Clients Benefiting
Skill vs Knowledge
Colleagues Support EBP
Initial Training on Internet
Engaging Expert Trainer
Active vs Passive Learning
Follow-up Trainin
Effectivenes in Other Agencies
Group Size
Initial Training Time
Observability Trialability
Selection Control
% Change in Practice
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Gradual Adopters Accelerated Adopters
Engaging Expert Trainer Most Important to Gradual
Adopters
Source: Cunningham, Barwick et al. Administration and Policy in Mental Health (2018)
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Not Engaging Not
Expert
Engaging Not
Expert
Expert Not
Engaging
Engaging Expert
Utility
Accelerated Adopters
Gradual Adopters
Source: Cunningham, Barwick et al., Administration and Policy in Mental Health (2018).
Trainer Qualities
Attributes of the Implementation Process that Influence Decisions
Relative Importance of KT Design Attributes: Initial Training Time
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Supervisor Support for EBP
% Clients Benefiting
Skill vs Knowledge
Colleagues Support EBP
Initial Training on Internet
Engaging Expert Trainer
Active vs Passive Learning
Follow-up Trainin
Effectivenes in Other Agencies
Group Size
Initial Training Time
Observability Trialability
Selection Control
% Change in Practice
Relative Importance
Imple
menta
tion A
ttri
bute
Gradual Adopters Accelerted Adopters
Initial Training Time Demand of Relatively Low Importance
Source: Cunningham, Barwick et al. Administration and Policy in Mental Health (2018)
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
1 Day 2 Days 3 Days 4 Days
Utility
Initial Training Time
Accelerated Adopters
Gradual Adopters
Source: Cunningham, Barwick et al., Administration and Policy in Mental Health (2018).
Initial Training Time
Relative Importance of KT Design Attributes: Initial Internet Training
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Supervisor Support for EBP
% Clients Benefiting
Skill vs Knowledge
Colleagues Support EBP
Initial Training on Internet
Engaging Expert Trainer
Active vs Passive Learning
Follow-up Trainin
Effectivenes in Other Agencies
Group Size
Initial Training Time
Observability Trialability
Selection Control
% Change in Practice
Relative Importance
Imple
menta
tion A
ttri
bute
Gradual Adopters Accelerated Adopters
Initial Internet Training Appealed to Accelerated
Adopters
Source: Cunningham, Barwick et al. Administration and Policy in Mental Health (2018)
-1
-0.5
0
0.5
1
No Internet
Learning
33% Initial
Training On Line
66% Initial
Training Online
100% Initial
Training Online
Utility
Accelerated Adopters
Gradual Adopters
Source: Cunningham, Barwick et al., Administration and Policy in Mental Health (2018).
Initial Internet Training
Relative Importance of KT Design Attributes: Initial Training Time
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Supervisor Support for EBP
% Clients Benefiting
Skill vs Knowledge
Colleagues Support EBP
Initial Training on Internet
Engaging Expert Trainer
Active vs Passive Learning
Follow-up Trainin
Effectivenes in Other Agencies
Group Size
Initial Training Time
Observability Trialability
Selection Control
% Change in Practice
Internet Options
Relative Importance
Imple
menta
tion A
ttri
bute
Gradual Adopters Accelerated Adopters
Initial Training Time of Relatively Low Importance
Source: Cunningham, Barwick et al. Administration and Policy in Mental Health (2018)
Mental Health Professionals
Both Segments Prefer
100% Support of Supervisors and Colleagues for EBP Changes
EBP Changes Benefiting All Patients
Proven in Research and Other Agencies
Trainers Who are Engaging Experts
Selected by Individual Programs within Agencies (Rather than Ministries)
Focus on 33% Knowledge 67% Skills
Observational Learning, Practice, and Feedback
Mental Health Professionals
Accelerated Adopters 12%
Greater Intent to Participate
100% of Initial Training on Line
Devote More Time (4 Days) to Initial Training
Would Make More Changes to Their Practice
Introduce only Minor Modifications to EBPs
Gradual Adoplers 88%
Lower Intent to Participate
33% of Initial Training on Line
Devote Less Time (2 Days) to Initial Training
Would Make Fewer Changes to Practice
Preferred More Follow-up Training
Introduce Moderate Modifications to EBP
Randomized First Choice Simulation
Basic Dissemination
2 Days of Initial Training
2 Days of Follow-up Training
33% Skills 67% Knowledge
25% Change in Practice
Enhanced Dissemination
4 Days of Initial Training
4 Days of Follow-up Training
67% Skills 33% Knowledge
50% Change in Practice
Source: Cunningham, Barwick Administration and Policy in Mental Health (2018)
0
20
40
60
80
100
Basic
Implementation
Enhanced
Implementation
49.2 50.8
80.3
19.7
Perc
ent
Accelerted Adopters
Segment 2
Randomized First Choice Simulation: Basic vs Enhanced
Source: Cunningham, Barwick Administration and Policy in Mental Health (2018)
0
20
40
60
80
100
Basic
Implementation
Enhanced
Implementation
23.6
76.4
37.4
62.7
Perc
ent
Segment 1 Segment 2
Randomized First Choice Simulation: Basic vs Enhanced + 67% Supervisor Support
Source: Cunningham, Barwick Administration and Policy in Mental Health (2018)
Selected Implications
Segments with Different KT Design Preferences
Change Ready versus Demand Sensitive
Outcome Sensitive vs Process Sensitive vs Demand Sensitive
Accelerated Adopters vs Gradual Adopters
Different Pattern of Preferences Across Professional Groups
Social Context Exerts More Influence Than Evidence Base
Administrators, Supervisors, Principals
Colleagues
Unions
Most Prefer Learning in a Small Group (N=10 Social Context)
An Engaging Expert Facilitator Important
Interest in Internet Options Varies Across Professional Groups
Highest in Mental Health Professionals in Accelerated Adopters Segment
Merci!