Exploring the Effectiveness of Online “Remote” Training and ......Exploring the Effectiveness of Online “Remote” Training and Coaching 30th Annual Research and Policy Conference

Exploring the Effectiveness of Online “Remote” Training and Coaching

30th Annual

Research and Policy Conference on

Child, Adolescent and Young Adult Behavioral Health

8 March 2017

Janet S Walker, PhD

Celeste Seibel, MS

Engaging Youth and Young Adults

in Wraparound

Presenters

Janet Walker, Ph.D.—Director, Pathways RTC; Co-Director National Wraparound Initiative and National Wraparound Implementation CenterCeleste Seibel, MS —

Co-Investigator for Achieve

My Plan (AMP) training

Projects; Lead Trainer;

Pathways RTC

RTC on Pathways to Positive Futures

“Rehabilitation Research and Training Center” funded by the federal government (HHS/SAMHSA)

• First funded in 2009, currently second 5-year cycle

• Focused on improving outcomes for youth and young adults with serious mental health conditions.

• pathwaysrtc.pdx.edu, then

– Publications search, enter author/title/key words, or

– Featured publications

Agenda• Why enhance Wraparound?

• The Achieve My Plan (AMP) enhancement

• Why enhance training?

• “Remote” training and coaching– Training and coaching process

– The Virtual Coaching Platform (VCP)

• Findings from the evaluation of AMP trainings– Practice change

– Training satisfaction and impact

– Implementation drivers and barriers

• Lessons learned

AMPlify!

Why enhance Wraparound?

Research showed that few youth meaningfully participated in their education, care, and treatment team planning:

◦ Schools/IEP

◦ Systems of care

◦ Wraparound

Professionals were also dissatisfied with the level of youth participation in wraparound

Ongoing experiences reinforce this: Limited voice and choice

download at

pathwaysrtc.pdx.edu

Engagement/Participation in

Wraparound

Walker & Schutte 2005 (youth up to age 18)

• Youth present more than half the meeting only 39% of the time

– ~50% if youth age 14 or older

• Youth observed often to be completely disengaged

• Youth more dissatisfied overall, less comfortable, saw more conflict on team

• Providers cited lack of youth involvement as one of the top “worst aspects” of team meeting

Walker, Pullman, Moser et al., 2012 (youth up to age 20)

• Youth less satisfied, older youth more dissatisfied than younger youth; youth rated their participation lower than caregivers did

• Some evidence of “crowding out” (youth versus caregiver), this also increased with age

What’s to be Gained?

• Youth voice and choice is part of the first principle of Wraparound

• Engagement increases when people feel they have voice and choice– Improved engagement with the team should lead to

• Better tailoring of services and supports to reflect youth priorities and to fit with needs and motivation

• Increased engagement in services and supports / higher “dose”

• Team alliance, which may have an independent effect on outcomes

• Acquiring self-determination skills—learning to make plans and achieve goals—is a key developmental task and a valuable asset in life









• Lessons learned

AMPlify!

Youth Participation in Wraparound:

Best Practices

• Best practices derived from research, vetted and refined through work with our advisors and testing for AMP

• Supporting participation includes:– Organizational support– Coaching– engagement, preparation,

follow up, participation skills– During the meeting: meeting

structures and interactions that promote youth participation

– Accountability

AMPlify!download at pathwaysrtc.pdx.edu

Achieve My Plan (AMP)

• Research project to develop and test an “enhancement” intervention to increase youth participation in planning

• Advisory Board—Emerging adults and youth, caregivers, providers, research staff

• What would be the characteristics of an enhancement with best chance of success?

– Feasible within resources of agencies– Appealing

• Address concerns of providers and caregivers• Be engaging for youth

– Increases participation in ways that are obvious (and/or measurable) and positive

What sort of “enhancement”?

• A structured process for incorporating a series of best practices into regular Wraparound practice

– Increase voice and choice

– Practice self-determination: Work with the young person to come up with activities where they can take the lead• Model, coach and teach self-determination skills

– Prepare the young person for collaboration

• Not qualitatively different, just more structured and intentional focus on “active ingredients”

– Curriculum is structured into “modules” and “boosters,” with elements that are recycled as needed

Top 10 Engagement Tips

download at

pathwaysrtc.pdx.edu

download at pathwaysrtc.pdx.edu

Key things that are different in AMP

• Focus on teaching skills for self-determination

– Not overly concerned about specific goals/activities or the perfect plan

– Practice and experience success at being a change agent for own life

– Learning to work productively and pragmatically, access support and help

• Training approach that focuses on concrete, observable skills

– Tools and curriculum are good, but in no way sufficient to ensure youth-driven conversation

Randomized study

Comparison of Wraparound “as usual” with Wraparound plus AMP enhancement

• Findings showed that, relative to control

–Youths’ active participation as assessed by all stakeholders (youth, care coordinators, family and other team members) was higher

–Video showed youth spoke more, made more meaningful contributions; team had more positive and fewer negative interactions

– Youth reported higher therapeutic alliance with team

• AMP enhancement was delivered by university interns (undergrad, MSW) with 6 weeks of training









• Lessons learned

AMPlify!

0

0.5

1

1.5

2

2.5

3

Theory …+ Demonstration

…+ Practice …+ Feedback …+ Coaching

Knowledge

Skill

Practice Change

Transfer of training to practiceEffect sizes for training outcomes

Joyce & Showers, 2003

“Gold Standard” for Transfer*

• Initial training includes clear conceptual model and concrete skills/competencies

• Observation of practice (live or video)

• Feedback using a reliable tool

– Development of initial competence

– Benchmark for competent practice

– Periodic checkups

*e.g., Dorsey et al., 2013;Garland, 2013; Herschell,

Kolko, Baumann, & Davis, 2010; Milne, Sheikh, Pattison, &

Wilkinson, 2011









• Lessons learned

AMPlify!

AMP Training Content

• Monthly video-conferences cover:

– Young adult engagement techniques

– AMP curriculum modules

– Theory of change

– Advanced skills (ex. supporting a young person to make a decision, opening a topic, what to do when things don’t go as planned)

AMP Training Process

– Review: the AMP trainers review a module from the AMP curriculum with the trainees during a video-conference.

– Observe: The trainees watch a video recording of a lead coach working through that module with a young person.

– Practice: The trainees record themselves doing the module with a young person and upload it to the secure training website– the Virtual Coaching Platform (VCP).

AMP Training Process, Cont.

– Feedback: An AMP trainer reviews the trainee’s video and provides feedback on areas of his/her practice that are strong as well as areas for growth.

– REPEAT: This process continues through all of the curriculum modules. Trainees practice/record/get feedback for each module with one “practice person” and one young adult from their caseload.

AMP Certification

• Trainees must demonstrate the ability to deliver the AMP curriculum to fidelity

• Trainees must pass the AMP certification quiz with a score of 80% or better

The VCP

Feedback Report

Some helpful things about the

VCP • Trainees can watch example videos through the VCP

• Trainees can watch their videos that they uploaded through the VCP

• The trainee or the supervisor can clip interesting/important interaction videos and share them with each other

• The supervisor can send the trainee a feedback report that links comments to specific segments, so the trainee can re-watch certain segments to better understand the feedback









• Lessons learned

AMPlify!

Training evaluation study

• 40 participants who completed the training (5 did not, mostly due to changing jobs)

• Assessed

– Prior to training: Skills self-efficacy

– ~1 month: first full submitted video, practice skill

– 6 weeks: IOTTA (Impact of Training and Technical Assistance) training satisfaction items

– ~4th month: final full video, practice skill

– Post: Skills self-efficacy, IOTTA, including items on implementation supports and barriers

Pre- and Post- Self-efficacy

• Trainees’ ratings of confidence regarding skills in specific areas

Paired samples t-test, all significant

Early and late skills as

demonstrated in videos

• Trainer and research assistant coded each video to ensure reliability

Paired samples t-test/Wilcoxon signed rank, all significant

Pre- ratings: Self vs video

Post-ratings: Self vs video

Aspects of training impact and

satisfaction• Ten-point scales from none/terrible/completely

disorganized, etc.; to fabulous, unsurpassed credibility, extremely confident, etc.

Contributions to mastery

• To what extent did these training elements contribute to any increase in mastery you experienced?

Important barrier Important barrier

Implementation drivers and barriers

• How much did these factors contribute to the degree to which you used what you learned…

Important barrier Important driver

AMP “family”

• “Original” AMP with Wraparound and older youth (to 18)—randomized study– “Original” AMP with care coordinators in several

states

• Study of AMP enhancement for Wraparound with young adults in MA—writing up now

• Now developing two versions that are stand-alone or in conjunction with larger team

– AMP-TF , delivered by “transition facilitators”

– AMP+, peer-delivered– finishing study in KY and OR









• Lessons learned

AMPlify!

Lessons Learned

• Approach appears to have potential to be quite effective

– Self-ratings and video ratings of skills increase significantly

– High level of satisfaction

• Can be challenging to ensure that people complete all aspects of training

– supervisory/administrative support and messaging is key

Pathwaysrtc.pdx.edu

Research & Training Center for Pathways to Positive

Futures, Portland State University

Acknowledgments/Funders

The development of the contents of this presentation were supported by funding from the NationalInstitute of Disability and Rehabilitation Research, United States Department of Education, and theCenter for Mental Health Services Substance Abuse and Mental Health Services Administration,United States Department of Health and Human Services (NIDRR grant H133B990025). The contentdoes not represent the views or policies of the funding agencies. In addition, you should notassume endorsement by the Federal Government.

Documents

Exploring the Effectiveness of Online “Remote” Training and ......Exploring the Effectiveness of Online “Remote” Training and Coaching 30th Annual Research and Policy Conference