Emerging Practices in VR: Moving from Pilot Project to System Integration 6 th Annual Summit on VR Program Evaluation & Quality Assurance September 16,

Emerging Practices in VR: Moving from Pilot Project

to System Integration

6th Annual Summit on VR Program Evaluation & Quality Assurance

September 16, 2013

Chip Kenney, Deborah Greene

Session Objectives• Share methods used by Southeast TACE

to evaluate integration of emerging practices in VR systems

• Review/Update findings from 2012 evaluation of Southeast TACE pilot projects designed to introduce practices.

• Present protocols and tools used to evaluate impact and measure/track systems integration.

TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013

2

Overview

• Since late 2009, the Southeast TACE has been partnering with a number of state vocational rehabilitation (VR) agencies to integrate innovative practices within the VR service delivery mainstream.

• Various states are piloting or integrating approaches such as Customized Employment and Asset Development.


3

Overview, cont.

• Five of these “Innovations Projects” have progressed to a point where their impact can be assessed to inform future action at local, state and regional levels.

• In 2012 TACE worked with project teams to evaluate individual project experiences as well as the Innovations initiative as a whole, in terms of their impact on the adoption of best practices within VR systems in the Southeast.


4

Overview, cont.

• All 5 projects are now taking steps to integrate piloted practices into VR delivery systems

• Three additional Innovations Projects are under way

• Evaluation of project activities continues through networking, tracking tools and Integration Scale measures


5

Overview, cont.Current status

•5 Southeast states, 5 state agencies

•8 projects in various stages of design or implementation, focused on…

Aspects of Customized Employment (7) Asset Development (1)

•5 have completed a pilot & are implementing further integration strategies

•2 projects from original count on hold due to agency internal changes


6

Overview, cont.Summary of evaluation findings

•VRCs in 4 states have viable option to serve jobseekers with most significant disabilities, where previously they had no effective option

•Results directly attributable to pilot projects

•Value of investment evidenced in… States’ further strategies to fully adopt the practice Refinements/Evolution of the practice itself Continuous improvement of the Innovations initiative

•New design/evaluation tools (Roadmap, Integration Scale, Community of Practice)


7

Innovations strategy: Goal

Increase the capacity of state agencies and their community partners to improve employment and self-sufficiency outcomes for individuals with significant and most disabilities


8

Innovations strategy: Premise Adoption of innovative practices requires more than training, hence a phased process from pilot to sustainability to integration•Incubator to understand/address feasibility and systems implications before taking it to scale

•Partners can try the practice out, get their feet wet, make it their own, get buy-in

•Admin. and service delivery partners experience and influence the process – credibility, support for the practice and its feasibility

•Phases designed according to local needs, resources, readiness, logistics


9

Approach to Innovation Projects


10

Innovations strategy: Process • Meet with agency leadership/staff re. goals, success

measures, strategy

• Agree on multi-phase design

• Assign project manager/team

• Develop project roadmap (workplan, resources)

• Identify pilot sites, partners, resources

• Conduct performance-based training

• Facilitate partnerships and infrastructure supports

• Evaluate pilot phase

• Refine plans for expansion/sustainability/integration

• Launch subsequent phases


11

Description of Pilots

Original 5 projects•Florida (2)

Customized Employment - Transition Discovery Certification - Transition and Adult

•Mississippi Customized Employment - Adult

•Georgia Customized Employment - Transition and Adult

•Kentucky Customized Employment - Adult


12

Description of Pilots, cont.

3 new projects since 2012

•Kentucky Asset Development

•Georgia Discovery in Residential Setting - Adult

•North Carolina Customized Employment - Transition


13

Evaluation, cont.

Design principles•Demonstrate value of Innovations approach to support adoption of emerging practices

•Assess impact to date of individual projects and initiative as a whole

•Directly connect results with the intervention

•Establish process/protocols for ongoing evaluation and continuous improvement


14

Evaluation, cont.

Targets•Outputs

Trainings conducted, numbers trained Meetings facilitated Products generated (reports, tools, samples)

•Impact Change in attitude/behavior, i.e., awareness/

understanding/utilization of the practice Systems changes, e.g., partnerships and

infrastructure measures to operationalize the practice


15

Evaluation, cont.

Targets, cont.•Return on investment: Start-up and operational costs, value-added

•Customer satisfaction: Agency/Partner assessment of project experience and quality of TACE services

•Jobseeker outcomes Completed profiles Job placements Financial security measures


16

Evaluation, cont.• Subjects: Projects that…

Completed the pilot or training phase (at least), and Took or are considering measures to ramp up

• Types of data Quantitative (trainings conducted, # participants,

active practitioners, etc.) Qualitative (partnerships & infrastructure, attitudes &

behavior)

• Overall Innovations initiative as a TACE technical assistance activity


17

Evaluation, cont.Data sources•Tracking tools, reports and other documentation (communications, products, etc.)

•Southeast TACE team (staff & consultants involved in development of initiative, design and execution of projects, etc.)

•Agency leadership

•Project leads & participants (incl. jobseekers and their families)

•Communities of practice


18

Evaluation, cont.Process•Agree on measures, design evaluation strategy with input from TACE team, agencies, projects

•Develop/Update evaluation Work Plan

•Design data collection and interview protocol (source documents, key informant interviews, focus group)

•Collect/Analyze prelim. Data

•Draft individual project reports and aggregate discussion paper


19

Evaluation, cont.Process, cont.•Conduct focus group among project leads

•Synthesize findings – focus group report

•Disseminate white paper summarizing findings and recommendations (TACE website)

•Publish article in Journal of Rehabilitation Administration

•Construct new evaluation tools Project Roadmap Integration Scale


20

Evaluation, cont.Process, cont.•Convene communities of practice

Customized Employment Asset Development

•Update evaluation measures and strategy Outputs (projects, trainings, TA activities, certified or

authorized practitioners) Systems integration/impact indicators (policies,

MOUs, curricula, expansion activities) Operations indicators – changes in attitudes, practices Client impact indicators – quantitative, qualitative


21

Evaluation, cont.Findings: Infrastructure•CE/Discovery defined/endorsed as discrete services in policy/regulation (MS, GA, FL, KY)

•CE/Discovery incl. in VR strategic plans (FL, GA)

•Fee code changes (MS, GA, FL, KY)

•Long term supports FL: DD agencies GA: MOUs between GVRA and DD/MH KY: HCBS waiver – require referral to VR MS: HCBS waiver – in discussion


22

Evaluation, cont.Findings: Capacity-building•Internal Capacity-building

Creation/Modification of positions to support CE Creating mentor/support teams

•Provider/Partner Capacity-building Outreach/Funding strategies to train/develop a pool of

VR-approved CRPs (GA) Communities of practice (FL, MS, GA) Mentoring strategies (MS, FL, KY) University-based training or credentialing program in

progress (GA, FL, KY)TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013

23

Evaluation, cont.Findings: Partnerships•School Districts (early collaboration for transition)

•Employment First

•DD and MH agencies

•Universities: capacity-building, evaluation

•Joint outreach & tracking strategies

•Shared recognition of practices as cross-system continuum of services requiring collaboration

•Reduced ‘finger pointing’


24

Evaluation, cont.

Findings: Systems Change•Awareness of CE as an option and increasingly requesting CE or Discovery for a given jobseeker

•Understanding of the innovation as a change in VR practice

•Use of CE by VR systems practitioners


25

Roadmap• Step-by-step guidance in developing Innovation

projects

• Ensures successful implementation, fidelity Objectives, success measures, evaluation Phases Stakeholders, partners Expectations Advance work Workplan, resources Deliverables


26

Roadmap, cont.• Tools to plan project, conduct meetings,

negotiate agreements, e.g. SWOT analysis Resource assessment Readiness/Feasibility assessment Risk assessment Meeting agenda template

• Use for design, development, tracking and evaluation


27

Integration Scale

Key indicators

•Agency Buy-in

•Policy and Procedures

•Capacity/Resources

•Data Collection

•Accountability/ Results Reporting

•Evaluation/Impact/

Continuous Improvement TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012

28

Integration Scale: Agency Buy-In

• Agency Leadership

• Agency Champion

• Administrative Level

• Field Service Level

• Partner Level

• Success Measures incorporated in strategic plan, agency identity, accountability


29

Integration Scale: Policy/Procedures

• Policy documents, communications

• Fee structure that adequately compensates providers

• Field Service Manual

• Built into staff job descriptions, performance expectations (management, supervision, field services and support)


30

Integration Scale: Capacity/Resources • Competencies Identified

• Qualified Providers

• Skilled Staff (management, supervision, field services and support)

• Adequate financial resources

• Ongoing training

• Certification Process for Providers

• Mentoring

• Tools

• Technical Assistance

• Partners/ Formal Agreements


31

Integration Scale: Data Collection

• Tracking Employment outcomes Client satisfaction Longitudinal

• Planned Service

• Purchased Service


32

Integration Scale:Accountability/ Results Reporting

• Performance expectations

• Reporting requirements

• Periodic reviews

• Community of Practice with designated agency liaison


33

Integration Scale:Eval./ Continuous Improvement

• Evaluation strategy, workplan

• Periodic review of results

• Results factored into strategic plan, milestones, infrastructure, capacity-building


34

Community of Practice

• Representatives of Innovation Projects meet at least quarterly Customized Employment Asset Development

• Ongoing technical assistance

• Peer networking, exchange

• Ongoing needs assessment

• Ongoing accountability and evaluation


35

Impact MeasuresExtent to which advances in integration can be attributed to TACE strategy

•Outputs (projects, trainings, TA activities, certified or authorized practitioners)

•Systems integration indicators (policies, MOUs, curricula, expansion activities)

•Operations indicators – changes in attitudes, knowledge; use of practices; referral rates

•Client impact indicators – quantitative, qualitative


36

Observations: Highlights• Leadership willingness to take risks, devote

resources to a practice untested in their system

• Systems changes that substantially increased capacity to serve jobseekers with the most significant disabilities

• Outcomes clearly attributable to the intervention

• Costs deemed worth the return on investment

• Pilot results are basis for decisions to expand, replicate, sustain the practice

• New products, new directions (e.g., residential)


37

Observations: Challenges

• Buy-in: Reinforcing that this is about serving a population for whom VR traditionally has not had a successful strategy; that building capacity will take time/effort and will be a departure from business as usual

• Consistently engaging leadership & management commitment sufficient to support and sustain a project (even w/ personnel changes)

• Project planning to create environment for successful pilot


38

Observations: Challenges

• Articulating success measures directly connected to project activities

• Identifying terms/triggers for moving from one phase of implementation to another

• Prioritizing resources for project management and face-to-face training and technical assistance.

• Communicating project parameters, expectations

• Logistics: Scheduling training, assigning cases


39

Recommendations

• Phased approach for practices that are a significant departure from business as usual

• Success measures clearly connected to project activities and decisions to expand/ sustain the practice

• Project workplan covering all phases (pilot to evaluation/replication to full implementation)

• Project management and facilitation resources


40

Recommendations• Clear communication of expectations/

implications in advance of committing to pilot

• Supplemental resources to finance project activities and ongoing operations

• Distance learning platforms rather than conference calls

• Leadership program for administrators providing info. on promising practices

• Supports to sustain/integrate practices (Roadmap, Integration Scale, Community of Practice)


41

Questions

&

Discussion


42


43

Southeast TACE Region IVToll-free: (866) 518-7750 [voice/tty]

Fax: (404) 541-9002

Web: TACEsoutheast.org

MyTACE Portal: TACEsoutheast.org/myportal

Email: [email protected]


44

Disclaimer

This presentation was developed by the TACE Center: Region IV ©2013

with funds from the U.S. Department of Education, Rehabilitation Services

Administration (RSA) under the priority of Technical Assistance and Continuing Education

Projects (TACE) – Grant #H264A080021. However, the contents of this presentation do

not necessarily represent the policy of the RSA and you should not assume endorsement by the

Federal Government [34 CFR 75.620 (b)].

Documents

Emerging Practices in VR: Moving from Pilot Project to System Integration 6 th Annual Summit on VR Program Evaluation & Quality Assurance September 16,