Implementing evidence- informed practice · Putting theory into practice 9 ... Evidence-informed practice When implementing evidence-informed practices into the real world ... Leading

Implementing evidence-informed practice

A practical toolkit

Learning Collaboration Leadership


Table of contents

www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 2

HELPFUL TOOLS

DOWNLOAD

Use the links provided to download

examples and worksheet templates

that could prove helpful throughout

your process.

EXAMPLE

Download example

Download worksheet

Table of contents

Introduction 4

Implementing evidence-informed practice 6

Implementation science 6

Evidence-informed practice 7

Learning organizations 7

Putting theory into practice 9

Framework for implementation 10

Planning phase: Getting started 11

I. Building leadership support Organizational implementation capacity 12

Leadership support 14

Senior management charter 15

Creating a budget 16

II. Engaging stakeholders Building staff buy-in and readiness 17

Engaging youth and families 22

Communicating change to stakeholders 23

Maintaining momentum 24

III. Managing and leading change Choosing a team 25

Regular team meetings 27

Assessing the evidence 29

http://www.excellenceforchildandyouth.ca


Table of contents


Selecting an evidence-informed practice 33

Creating an implementation plan 34

Selecting a purveyor 36

Ongoing evaluation and monitoring 37

Implementation activities and timeline 38

Preparing for the next phases of implementation 39

General tips and strategies for implementing EIPs 41

Additional resources 42

References 42

Acknowledgements 44



Table of contents


HELPFUL INFORMATION

About the Centre

As a leader for child and youth

mental health in Ontario, we build

connections, share knowledge

and draw upon our expertise to

enhance the skills of front line service

providers. We passionately

promote and share the benefits

of both organizational learning

and evidence-informed practices.

Together, with our partners, we are

working to strengthen Ontario’s

mental health programs and services

for all children, youth, families and

caregivers.

Other available toolkits

“Try to take things one day at a time. If you rush the process you may overlook important things or not be truly prepared for your implementation..”

— Implementation support service

(ISS) agency

Introduction• Are you planning on introducing a new program to

address the mental health needs of children and youth in your community?

• Are you wondering what it would take to sustain a program or practice in your organization?

• Are you looking for tools and resources to kickstart the implementation process?

Welcome to the implementation toolkit

Implementation is a continuous process that includes a specified set of activities designed to put into practice an activity or program of known dimensions (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005). Knowing about implementation is one thing, but doing it is another. As with learning any new skill, only through practice will individuals and organizations become proficient in implementation processes.

As you begin on your implementation journey, be sure to consider the

following:

• Research indicates that it takes two to four years to implement a new practice/service into your organization (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005).

• Rushing or skipping steps in the implementation process may result in a false start or an inability to sustain the practice.

• Without a well thought-out plan, implementation can take longer, cost more money, have a negative impact on your people and/or become too costly to maintain.

• While the phases of implementation (planning, doing and sustaining) may overlap, each takes six to 12 months to complete.


http://www.excellenceforchildandyouth.ca/resource-hub/toolkits


Table of contents


If you are new to implementation, you might want to start with the Centre’s implementation learning modules to gain a better understanding of implementation science.

This toolkit contains practical insight, strategies and resources for the planning phase of implementation. It couples theory and research findings with practical strategies and real-life experiences from the field that may be relevant to your organization.

Implementation support services at the Ontario Centre of Excellence for Child and Youth Mental Health

The Ontario Centre of Excellence for Child and Youth Mental Health (the Centre) builds connections, shares knowledge and draws upon expertise to enhance the skills of direct service providers and agencies across the province. The Centre passionately promotes and shares the benefits of both organizational learning and evidence-informed practice. Together with our partners, the Centre is working to strengthen Ontario’s mental health programs and services for all children, youth, families and caregivers.

The Centre’s implementation support services are designed to meet the needs of Ontario’s child and youth mental health agencies and their communities. We work with agencies to achieve sustained success in the implementation of evidence-informed practices using an integrated program of education, training and ongoing consultation. We offer variable levels of support to match the needs of diverse organizations, and our team can work with agencies at any point in the change continuum.

For more information on available services please visit our website.


http://www.excellenceforchildandyouth.ca/resource-hub/learning-modules

http://www.excellenceforchildandyouth.ca/what-we-do/implementation


Table of contents


Implementing evidence-informed practiceLet’s start with the big picture. When planning a change in the services you provide, it helps to understand the concepts of implementation science, evidence-informed practice and learning organizations and consider how they can work together to help you succeed.

Implementation science

The science of implementation draws from a multidisciplinary set of theories and methodologies that bridge the gap between research and practice. It is essential to consider the processes involved when integrating new practices within your organization.

Identifying an effective practice and training staff to deliver it are not enough to achieve successful implementation. A program that has been shown to improve mental health outcomes for children and youth can be poorly implemented and fail to produce good outcomes – this is referred to as an implementation gap. Either the practice is not used as designed or isn’t applied consistently over time.

Current research and practical evidence tell us that factors called implementation drivers influence the successful implementation of EIP (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005). These drivers include leadership support, clinical supervision, performance management, policies and procedures. When implementation drivers are in place, the risk of there being a gap between research and practice is reduced.

www.excellenceforchildandyouth.ca 6




Table of contents


Evidence-informed practice

When implementing evidence-informed practices into the real world setting, it can be difficult to match scientific research findings to the specific needs of an organization and the people it serves. Where evidence exists, it may have been developed in a research environment but does not produce the same results in a less controlled evironment. Research findings may also exclude many promising practices that have not yet had a chance to be systematically evaluated. Finally, the practice may simply not make sense for a given population.

Evidence-informed practices are provided to clients in a way that combines the best available research with the experience and judgment of practitioners and the unique preferences of children, youth and their families to deliver measurable benefits. When an agency provides an evidence-informed practice for clients, it is more than finding and implementing an off-the-shelf manualized intervention. It’s about continuously gathering, sharing and using evidence from the research literature, practitioners experience and feedback from children, youth and families. By combining these elements, organizations can strengthen their services and improve outcomes for the children, youth and families they serve.

Although the terms evidence- based practice (EBP) and evidence-informed practice (EIP) are used interchangeably in the literature, the Centre favours evidence-informed practice. In order to properly reference the research, we will continue to use the same terminology (EBP or EIP) suggested by the respective author(s).

Learning organizations

Learning organizations are on a constant journey of discovery and organizational development. They are willing to take risks, embrace creativity and enjoy a culture that promotes continuous improvement and professional growth. Learning organizations provide a practical and purposeful way to embrace evidence-informed practice, share knowledge and improve outcomes for children, youth and families.



Table of contents


Gather information: Evidence is everywhere. You can find it in peer-reviewed journals, stories of successes and challenges, your daily work and the real-life experiences of the people you serve.

Use knowledge to grow: Gathering diverse and relevant information is critical but it’s what you do with it that matters most. Evaluation offers a way to arrive at your ultimate destination: better outcomes for children and youth. Implementation is the deliberate but flexible path to use evidence effectively.

Share your story: We can learn a lot from each other. Sharing research evidence is important but so is telling stories from the field. Stories are where we find meaning. They present real-life evidence that helps us find common ground. By sharing your story, you provide evidence for someone else.

The Centre asked Ontario mental health organizations what strategies they use to foster a culture of learning. Here are some of their responses:

• Be open to questions.

• Seek staff input for training ideas.

• Keep people up to date on what’s going on provincially/nationally.

• Have clear clinical goals.

• Encourage performance appraisals.

• Encourage community experience.

• Model positive leader attitudes.

• Foster an environment for people to ask questions.

• Use team meeting time as a time to learn.

• Provide learning opportunities.

• Develop partnerships.

• Use specific staff strengths and build on their knowledge and experience.

• Recognize staff as your best resource.

• Research connections with universities.

• Appreciate different skill sets.

• Acknowledge efforts.

• Conduct literature reviews relevant to each program.



Table of contents


Putting theory into practice

“The learning modules are very valuable in assisting us in finding new ways to tackle implementation, both through discussion with the facilitators and through discussion with the core team members”.

— Implementation support service (ISS) agency

Implementing an evidence-informed practice should be an organization-changing experience. The Centre is pleased to offer a collection of learning modules that can support agencies throughout their journey by building skills and strengths that last long after the implementation project is complete. These modules provide a theoretical overview of implementation science and can be used in conjunction with this toolkit.

1. Leading organizational change - an introduction to topics surrounding leadership in the change process and how to create a sense of urgency for change

2. Teamwork and collaboration - the key components involved in building a successful team and the critical factors needed for effective teamwork

3. Needs assessment - a focus on staff readiness, the need for an evidence-informed practice, critical appraisal of the literature and selecting the right evidence-informed practice

4. Implementing evidence-informed practice - a look at the factors influencing the implementation process, drivers and plan

5. Monitoring and evaluating outcomes - the importance of an outcome measurement system and how to use it at different levels within your organization

It is highly recommended that your core team members become familiar with implementation theories and establish timelines for the implementation initiative in the beginning stage.

Tips and resources

When using the modules, consider the following:• Set aside dedicated time to go through each module.

• It is ideal to complete each module as a team in order to enrich discussion and support common understanding.

• Attachments provide helpful resources which may be applied within your organization.

• Additional tips can be found in an introductory module.



http://www.excellenceforchildandyouth.ca/resource-hub/learning-modules/implementation-introductory


Table of contents


Framework for implementation

The Centre’s implementation framework was adapted from the implementation model of the National Implementation Research Network (NIRN) (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005), the clinical transformation model of Kinark Child and Family Services (Moore, Cunning & Roundwaithe, 2010), the Partnerships for Success program in the state of Ohio (Julian, Ross & Partridge, 2008; Kerns, Rivers & Enns, 2009) and the Replicating Effective Programs framework by the US Centers for Disease Control (Kilbourne, Neumann, Pincus, Bauer & Stall, 2007).

The goal of the planning phase is to consider evidence-informed practice implementation an organizational change initiative and get people and systems ready for that change.



Table of contents


During the planning phase, you may want to conduct an organizational implementation capacity assessment, engage staff readiness for change, consider youth and family feedback before practice change and select the new practice to address the current gap and/or needs. You want to focus on the core element of leadership commitment and support right from the beginning, as well as having evaluation practice in place within the organization before implementation begins.

Although the major activities of this framework are set into three stages, the process should be thought of as cyclical. That is, the framework functions like all continuous quality improvement approaches and the activities should be regularly revisited and refined.

For the timeline and activities of the planning phase of the implementation, check out the section on Implementation activitiesand timeline.

Planning phase: Getting started”There is so much to learn about implementation. The nuances and preparation are so important in the process”

— Implementation support service (ISS) agency

The following section describes each component in the planning phase and provides tools and tips from the mental health field to help you on your implementation journey. As you go through each part, you will find that success (or challenge) in one area is linked to success (or challenge) in another. While not sequential during the planning phase, the following elements should all be considered:

I. Building leadership support II. Engaging stakeholders III. Managing and leading change

Organizational implementation capacity

Leadership support

Senior management charter

Creating a budget

Building staff buy-in and readiness

Engaging youth and families

Communicating change to

stakeholders

Maintaining momentum

Choosing a team

Regular team meetings

Assessing the evidence and

selecting an EIP

Creating an implementation plan

Selecting a purveyor

Ongoing evaluation and

monitoring

Implementation activities and timeline



Table of contents


I. Building leadership support Organizational implementation capacity

Implementing an evidence-informed practice will involve change. The first step is to determine organizational capacity (ability to support) and readiness (preparedness and availability) for implementation. This process will consider things such as identifying staff attitudes and skills, assessing financial resources and competing demands (e.g. accreditation), as well as selecting the right EIP to meet organizational needs.

Implementation capacity checklistThe Centre has developed and adapted an implementation capacity checklist that can help organizations identify their capacity to implement an EIP. It assesses key characteristics, skills and processes that learning organizations use to facilitate ongoing integration of evidence into practice. Some items on the checklist were adapted from the Checklist to Assess organizational Readiness for Implementation (CARI) (Barwick, 2011). It examines the degree to which organizations and their staff have these key components:

• access and use of evidence

• youth, family and community engagement

• evaluation practices

• learning culture

• management of new or ongoing initiatives in implementation

• training and implementation plan for an evidence-informed practice

• strengths and limitations in using evidence

Who should fill out the checklist? The implementation capacity checklist should be completed by three to five people from a variety of disciplines within the organization (e.g. direct service provider, manager and senior leader).

Why is the checklist important? Organizational capacity and readiness is an under-emphasized but critical part of the implementation process. If organizations do not have the current capacity or are not ready to implement an evidence-informed practice, they will likely be unsuccessful. The checklist is also used as a measure to recognize where implementation capacity has grown over time. For example, it can help you:

• engage staff in implementation planning (start the conversation).

• set priorities and strategic directions.

• measure organizational readiness to implement a new practice.

• identify strengths and weaknesses.

• generate actionable items for planning purposes.

• lay the structure and groundwork, including developing any necessary processes, to prepare for change.



Table of contents


Tips and resources

The implementation capacity checklist:

• is completed from the perspective of the organization as a whole.

• will provide a realistic snapshot of your organization.

• will help your organization review where you are in your implementation journey and should be completed annually.

To learn more about assessing organizational capacity, view the Centre’s webinar: Assessing organizational capacity to evaluate and implement evidence-informed practices

You can access the implementation capacity checklist on the Centre’s website.


http://www.excellenceforchildandyouth.ca/webinar/assessing-organizational-capacity-evaluate-and-implement-evidence-informed-practices

http://www.excellenceforchildandyouth.ca/webinar/assessing-organizational-capacity-evaluate-and-implement-evidence-informed-practices

http://www.excellenceforchildandyouth.ca/what-we-do/implementation


Table of contents


Leadership support

“…Successful [organizational change] transformation is 70 to 90 percent leadership and only 10 to 30 percent management.” — John P. Kotter

What do we mean by leadership support?Leadership support refers to the explicit acknowledgement and buy-in from senior management within an organization, including the board of directors. This requires their understanding of the need for change and support for the organizational commitment to making it happen.

Why is leadership support important?Having leadership support is critical when a gap in service has been identified and the decision must be made to bridge it in an evidence-informed way. Senior leaders who buy in to – or drive – change efforts understand the value of evidence-informed practice, acknowledge the importance of thoughtful implemention and align organizational goals to facilitate the change process. Leadership support serves as a driving force to steer and shape the implementation, particularly in providing necessary resources.

Tips and resources

Consider the following:• Use findings from the implementation capacity checklist to create a presentation to senior management.

• Consider who your presenters are, because sometims the messenger is as important as the message. Is there a champion who can co-present to the leadership group?

• Highlight the rationale behind the change (the why) during the discussion with senior leaders. Explain the clear benefit to the organization, and by extension to staff, leadership and clients.

• Identify the critical success factors for implementation within the organization.

• Identify any potential barriers for implementation that may need to be considered.

• If possible, identify examples where your organization has successfully implemented an evidence-informed practice and how you can build upon it.

• Highlight alignment with your organization’s vision and values.




Table of contents

HELPFUL TOOLS

DOWNLOAD

Use the links povided to access and

download examples and templates


your process.

Tools and reference

Here is an example of a senior

management charter with headings

and key questions that may

be incorporated into your own

document. You may use other terms

or components to reflect your own

organization.

Download example

Example from the field

A senior management charter

from Aisling Discoveries Child and

Family Centre: Facing your Fears:

Implementing a new evidence-

informed practice to improve

outcomes for children with anxiety

(and their parents)

Download example

Senior management charter

What is a senior management charter?The senior management charter outlines the key activities that are expected during your two- to three-year implementation journey. It is sometimes referred to as a terms of reference or project proposal, and it may be viewed as the contract for the core team. The charter needs to be agreed upon and supported by senior management within the organization. The document should explain why the implementation is important at this time, identify the areas of strength within the organization and areas that may require some support.

Who should complete the charter?The senior management charter should be completed by members of the core team who will be leading the implementation process.

Why is a senior management charter important?The charter is an important communication tool between the core team and senior leadership about the implementation process. Just like a blueprint for building a house, a senior management charter is an important step for initiating the implementation process. Through the charter, the leadership group will gain a general sense of the intent and scope of the implementation plan and the responsibilities of the core team members.

How do you create a senior management charter?After you’ve identified what program(s) you want to implement:

• Bring together a core team.

• Identify someone who will write the charter.

• Brainstorm the key elements of the implementation plan.

• Circulate the document for the core team to review.

• Have someone outside of your agency provide feedback.

Tips and resources

When writing a charter:• keep your audience in mind - senior management and board members.

• consider the results of your implementation capacity checklist.

• consider the implementation drivers as they are critical to the success of your plan.

• consider key community stakeholders.

• consider gathering input from clients and families you serve.

• Remember that this document will evolve with your project and should

be reviewed on a regular basis.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/7_key_questions_for_senior_management_charter.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/8_senior_management_charter_aisling_discoveries.pdf



Table of contents

HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

Here is an example of the budget

template.

Download worksheet


In-kind time tracking sheet from Child

Community Resources,

Download example

Creating a budget

What is an implementation budget?Implementation requires resources – human and financial. When you consider allocating resources to the implementation effort, financial support is a crucial component to successful implementation. Without sufficient resources, change could cause frustration that impedes the implementation from moving forward.

Who should be involved in creating the budget?• all core team members in collaboration with senior leadership

• a financial officer could assist with estimating and tracking in-kind contributions

How do you go about creating a budget?• Start by conducting a needs assessment to determine resources required

for the implementation initiative.

• From the senior management charter, list any items that may require budget considerations (e.g. backfill, personnel costs, training costs, information and technology support, administrative costs as well as other resources).

• Consult with financial staff within the organization and discuss potential resources that can be dedicated to the implementation.

• Include estimated in-kind contributions.

• Begin discussions on how to integrate implementation expenses into annual operational costs.

• Keep accurate documentation of actual costs.

• Update and revise the projected budget at the end of the planning phase.

Tips and resources

Consider the following:• Ask staff outside the core team to review the budget to ensure it makes

sense.

• Consult with other staff/organizations that may have experience in implementing EIPs.

• Combine in-kind costs (e.g. computer or equipment) with other relevant initiatives to maximize the impact of implementation.

• Use technology for communication to minimize cost for face-face meetings.

• Anticipate and project the costs associated with on-going supervision and coaching (internal and external).

• Anticipate and project the travel costs for staff consultations and communication throughout the process.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/9_budget_worksheet.xls

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/10_tracking_sheet_ccr.pdf


Table of contents


II. Engaging stakeholders Building staff buy-in and readiness

Who should be involved?

Establishing buy-in for implementation of an EIP is necessary at all levels including practitioners, leaders, and policy-makers. Gaining commitment involves identifying the advantage of the practice (e.g. “What are the benefits for me and my client?” and “How will it make my job easier?”) and demonstrating the availability of the resources, training and leadership required to support and sustain change.

You should consider assessing staff attitudes to improve the effectiveness of implementation efforts. This will increase your success during the planning process and throughout your implementation journey.

What tools and strategies are available?

Various tools are available to understand and assess individual and organizational commitment, buy-in, readiness and attitudes related to implementing a new evidence-informed practice.

• Commitment curve

The commitment curve (Conner & Patterson, 1982) explains why someone is willing (or unwilling) to participate in daily activities. It is closely related to motivation within oneself and interaction with the environment (i.e. extrinsic and intrinsic motivation, reinforcement).

People usually go through three stages of commitment:

• Awareness and understanding: where some people may resist the idea. This is mostly due to lack of information or understanding of the situation.

• Engagement: where people see the benefits and want to get involved, and are more comfortable with the idea of change.

• Commitment: where people own the benefits of change and are actively involved and contribute to the change effort.



Table of contents


You can use this tool to assess organizational commitment throughout the implementation process. This can be done individually or as a group.



Table of contents


• Evidence-Based Practice Attitude Scale (EBPAS)

The EBPAS (Aarons, 2005) is a 15-item measure that describes an individual’s global attitude toward the adoption of evidence-based practice (EBP). It was developed by Gregory Aarons as a preliminary exploration of mental health service provider attitudes toward the adoption of EBP in community mental health settings in the U.S.

The EBPAS includes four scales:

Appeal scale: assesses the extent to which a provider would adopt an EBP if it were intuitively appealing, could be used correctly or was being used by a colleague who was happy with it.

Requirement scale: assesses the extent to which a provider would adopt an EBP if it was a requirement of their agency.

Openness scale: assesses the extent to which the provider is generally open to trying new interventions and would be willing to try or use EBPs.

Divergence scale: assesses the extent to which the provider perceives EBPs as not clinically useful and less important than clinical experience.

Sample questions (Yes/No responses):

1. I like to use new types of therapy/interventions to help my clients.

2. Clinical experience is more important than using manualized therapy interventions.

3. I would try a new therapy/intervention even if it were very different from what I am used to doing.

Information here is taken from Aarons, G. A. (2005). Measuring Provider Attitudes Toward Evidence-Based Practice:

Consideration of Organizational Context and Individual Differences. Child and Adolescent Psychiatric Clinics of North America.

14(2): 255–271.

• Implementation Capacity Checklist (ICC)

The Implementation Capacity Checklist can be used as a tool to identify organizational readiness and capacity in implementing EIPs. Please visit the section Organizational implementation capacity in this toolkit for more information.


http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564127/

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564127/



Table of contents

HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

A comparison of the Implementation

Capacity Checklist, the

Organizational Readiness Checklist

and the Readiness for Organizational

Learning and Evaluation Instrument

Download example

• Organizational Readiness to Change Scale (ORC)

The Organizational Readiness for Change (ORC) assesses an agency’s readiness for change (Lehman, Greener & Simpson, 2002). The ORC scales are good indicators of overall strengths and weaknesses within the areas assessed and are useful for highlighting potential areas for improvement.

The ORC includes 18 scales from four major domains:

1. Motivational readiness (three areas)

2. Institutional resources (five areas)

3. Staff attributes (four areas)

4. Organizational climate (six areas)

Information here is taken from the Lehman, E., K.W.; Greener, J.M.; and

Simpson, D.D (2002). Assessing organizational readiness for change. Journal

of Substance Abuse Treatment, 22, 197-209.

• Readiness for Organizational Learning and Evaluation Instrument (ROLE)

The Readiness for Organizational Learning and Evaluation Instrument (ROLE) assesses the level of readiness of an organization to implement processes and practices in learning and evaluation within the organization (Preskill, H. & Torres, R.T. 1999). The ROLE can be used to determine the level of organizational learning and its characteristics within an organization. It can also identify existing strengths and needs in terms of organizational change and evaluation processes. It can measure staff interest in participating in evaluation and provide an overall direction for improving organizational learning for both individuals and teams.

The ROLE includes 81 items that are embedded in subscales as well as some additional information including:

1. Culture (27 items)

2. Leadership (12 items)

3. Systems and structures (12 items)

4. Communication (eight items)

5. Teams (11 items)

6. Evaluation (eight items)

7. Additional information (three items)

Information here is taken from the Preskill, H. & Torres, R.T.(1999). Evaluative

Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/34_toolkit_comparison.pdf

http://www.sciencedirect.com/science/article/pii/S0740547202002337


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Go to TED Talks to access a Simon

Sinek clip on How great leaders

inspire action.


Utilizing EBPAS and sharing results

with staff - Central Toronto Youth

Services

Download example

How do we consolidate information gathered from the readiness tools?

• Identifiying barriers and strengths

Information gathered from readiness tools can help determine organizational strengths and areas to improve. It is important to identify barriers and strengths within your organization that may influence the implementation process. A change initiative can fail if the weight of the barriers is greater than the strengths or benefits. If the weight of barriers is greater than the strengths, it is important to identify actions to shift the balance.

Common examples of barriers include scepticism from staff, long waiting lists, staff shortages and poor communication. Examples of strengths include effective and functional teams, leadership support, financial security and key timing.

Key questions to consider:

• Do people consider something to be a barrier because they do not fully understand it?

• Will better communication change perceptions?

• Can the effect of a barrier be reduced? For example, could you extend the implementation period to allow people more time to adjust?

• Is an additional strength needed? For example, additional training, new equipment, improved facilities and incentives.

• What aspects of the organizational culture can you build upon?

Tips and resources

Consider the following:• Use one or more of the above tools or strategies to assess

staff readiness in the beginning, middle and the final stages of implementation.

• Staff commitment could fluctuate throughout the implementation process; you may want to revisit the commitment curve at various points.

• Work with key stakeholders to identify strengths and barriers to the proposed change.

• General tips on dealing with resistance:

• Provide information (the why, what and how) and what it means to staff.

• Reflect concerns: acknowledge and validate the person’s opinion.

• Roll with resistance: each person’s perspective is valuable. It may provide insight to the challenges associated with the change initiative.


http://www.ted.com/talks/simon_sinek_how_great_leaders_inspire_action.html

http://www.ted.com/talks/simon_sinek_how_great_leaders_inspire_action.html

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/13_utilizing_ebpas_ctys.pdf


Table of contents


Engaging youth and families

What is family engagement?

Family engagement is an active partnership between families and service providers. Family engagement in community child and youth mental health services is increasingly considered best practice for providing quality service and support (Chovil, 2009). Involvement of families in their child’s treatment can be related to successful treatment outcomes (Hoagwood, 2005). As well, engagement interventions implemented during first contact with youth and families boosts service use (McKay et al., 2004).

For service providers, this means listening to what families think, engaging them in two-way communication and involving them as essential allies in decision-making so that their involvement is meaningful and has a purpose. Effective family engagement requires the service provider to develop a relationship-building process focused on listening. Engaging families early in the process of implementing a new evidence informed practice can increase their buy in and support towards the new practice which can help in it’s success. Engagement in the form of information sessions or focus groups early in the exploration of a new practice will allow families to have input into the treatment being considered for their children.

What is youth engagement?

Youth engagement is about empowering all youth as valuable partners in addressing and making decisions about issues that affect them personally and/or they believe to be important. Engaged youth play an active role in program planning and decision-making. They can include current and former clients, or those who express a genuine interest in the issues.

An adult ally is an adult assigned by the organization that supports, advocates for and works alongside youth. Youth are meaningfully engaged when they are involved in activities that they believe to have purpose, when they show commitment to what they are doing and they demonstrate gained knowledge of the activity.

How can youth and family engagement enhance the implementation process?

The inclusion of key external stakeholders such as children, youth and their families is recommended in the implementation process. Below is a summary of comments provided by Ontario child and youth mental health agencies about why children, youth and their families should be viewed as active partners (rather than recipients) in the implementation process:

• Buy-in is better if they are active partners.

• Creating relevant services to meet client needs.

• Clients/parents are the experts on their children’s needs. They will be more motivated to participate if they feel their input has been heard.

• The client’s voice is essential in planning, implementing and evaluating EIPs.

• Client input is essential in helping us make decisions about delivery of services.

• It’s about the clients. Their perspective will be valuable and the client experiences will define our success. We need the continual voice of clients.

• You are not doing something for them, you are doing it with them.

• They have valuable points of view/expertise and can become a voice for us as advocates.

• They help keep process goals clear and real.



Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

The Centre has developed a handy

tip sheet on communicating change:

Download tip sheet

Here is one example of Centre’s

work to complete the feedback loop

with stakeholders. This provides an

opportunity for stakeholders to have

their input, indicate the next steps

and create a continuous learning

framework.

Download example


Engaging with community partners -

Children’s Centre Thunder Bay

Download example

Tips and resources

• You may not be able to include all members at once: consider incremental steps.

• You may start with existing committees. For example, a youth advisory committee

Visit the Centre’s website for more information on youth and family engagement training.

Communicating change to stakeholders

Stakeholders are individuals who have a direct or indirect interest in an organization, such as staff, clients, and community partners. It is essential to include stakeholders in the implementation process and establish ways to communicate the processes and outcomes. This involves identifying who the stakeholders are, sharing the rationale for change, and explaining how they may be impacted. Communication with stakeholders is critical, but is not always easily or effectively done.

Tips and resources

Consider the following:• Consult with clients around language to use.

• Inform via a newsletter.

• Have parent representative(s) on the implementation team.

• Hold an education session with clients/stakeholders on benefits of implementation of EIP.

• Have informal discussions with clients.

• Introduce consumer (parent/youth) into EIP committee/sub-committee


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/14_communicating_change.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/15_feedback_loop.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/16_engaging_community_partners_cctb.pdf

http://www.excellenceforchildandyouth.ca/what-we-do/youth-and-family-engagement

http://www.excellenceforchildandyouth.ca/what-we-do/youth-and-family-engagement


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tips and tricks

The Centre has captured

suggestions from provincial mental

health agencies across Ontario

at the planning phase of EIP

implementation

Download tip sheet


Implementation newsletter –

Woodview Children’s Mental Health

Download example

Maintaining momentum

Maintaining momentum within core teams and organizations is a key ingredient for effective implementation of evidence-informed practice. This requires ongoing communication and dissemination of information to ensure the implementation initiative remains a priority for all staff and ultimately becomes embedded in the organizational culture.

Why is it important to maintain momentum?Having strategies in place to build momentum will help the core team, implementation team, stakeholders and agency staff to stay motivated, energized and engaged throughout the process. Given that full implementation may take two to four years, momentum may fluctuate over that time. By keeping staff up to date with key milestones in the process, you will help to move the implementation forward.

Who should be involved in maintaining momentum? The core team should be thinking about ways to build and sustain momentum within the team and agency from the very beginning of EIP implementation.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/32_tips_maintaining_momentum.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/17_implementation_newsletter_woodview_cmh.pdf


Table of contents


III. Managing and leading change Choosing a team

“Select a team that has the widest range of representation from programs and/or employment status as possible (e.g. front line, senior management)”— Implementation support service (ISS) agency

Teamwork and collaboration are important for change and essential to the implementation process. Success depends on a cohesive and effective team that carries through the implementation from the beginning to the end.

Why is it important to establish a team?A team approach allows the work to be shared and enriches the change process. It can create a common vision to support and sustain the project, and team members act as ambassadors for the practice being implemented. They educate others, bring everyone on board and prepare the organization for change. A team approach will help to identify challenges and opportunities and pool resources to overcome them.

What teams should be established?There are two types of teams that contribute to the implementation process: the core team and the implementation team. The number of representatives on either team depends on the scope of your initiative.

A core team is established at the beginning of the planning phase and oversees the entire implementation process including the development of the implementation plan. An implementation team is assembled once the planning phase is complete. This team assists in carrying out the implementation plan and moving the selected practice or program through the stages of implementation.

Who should be on the core team?A minimum of three people should be included on the core team. These members should represent various areas within your organization including direct service providers, clinicians, senior leadership, middle management, technical support, administrative support, family and youth. These individuals utilize organizational strengths and can help eliminate barriers that may impede the implementation process.

Who should be on the implementation team?The implementation team is formed once the initial exploration and planning phase has been completed. This team is crucial in the execution of the implementation plan. Similar to the core team, members of the implementation team should be representative of the organization. Some core team members may step in and become part of the implementation team, while others may remove themselves as the initial implementation is undertaken.

Individuals may be chosen for the implementation team because of their expertise in implementation, the specific EIP, evaluation, interest, coaching and supervision. Some examples of potential implementation team members include project managers, practitioners, clinical management and leadership, family members and youth, EIP trainers/purveyors and evaluators.



Table of contents


Tips and resources

Consider the following:• Invest time in selecting and building your core and implementation teams.

• Include enough members for diversity of opinion.

• Review existing committees (i.e. best practice, evaluation). Identify opportunities for alignment.

• Identify people you may not typically include (e.g. technical support and human resources personnel).

• Involve members of your organization that are considered champions.

• Focus on building trust and relationships.

• Build on the strengths that each member can bring to the table.

• Communication is key.

• Develop terms of reference for the team.

• For the smaller agencies, seek opportunities to partner within the community.

• Survey core and implementation teams on a regular basis (i.e. six months/one year).

• If your agency has multiple sites, ensure that you have adequate representation from each of your sites.

Sample matrix for selecting core team members in a multi-site agency

Site 1 Site 2 Site 3

Youth ü

Family member ü

Clinical staff/ service provider ü

Management ü

Administrative staff (IT, communications) ü



Table of contents


Regular team meetings

“Never underestimate the strength of your selected team; they have immense knowledge about your agency and selected program, and can assist in propelling your project forward.” — Implementation support service (ISS) agency

Team meetings are critically important and are effective ways to maintain communication in the implementation process. Depending on the scope of implementation and nature of the practice, the following types of meetings may be used:

• Core team meetings

• Implementation team meetings

• Internal stakeholder meetings

• External stakeholder meetings, including community partners and families

Core team and implementation team meetings

Team meetings can occur on a regular basis with pre-determined dates and agenda items. They can also happen in an informal way that encourages dialogue and open discussion of ideas.

Internal stakeholder meetings

These meetings are opportunities to involve stakeholders within the organization in the change process. They can increase staff awareness and knowledge of the implementation initiative and create a forum that reinforces the learning organization approach.

Ideas include:

• small department/unit meetings

• lunch-and-learn

• road show Q and A hour

• annual staff meetings or retreats

• knowledge exchange and learning events

• on-line staff chat room

• ask an expert

• monthly learning circle

These types of meetings also refer to informal opportunities such as show and tell, water cooler chats, or one-on-one meetings with supervisors.



Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

Sample implementation team

meeting agenda

Download example


A sample terms of reference: CBT

community of practice - Pathways

Children and Youth

Download example

External stakeholder meetings, including community partners and families

These meetings are opportunities to involve stakeholders outside the organization in the implementation process. Ideas include:

• town hall meetings

• community of practice

• specific committees and collaborative initiatives

Tips and resources

Consider the following:• In the planning phase, core teams meet more frequently. Most

organizations meet on average 1.5 hours every two weeks in the first six months.

• Identify:

• who will attend the regular meetings.

• the frequency of the meetings.

• the objectives of the meetings.

• Determine meeting logistics in advance (i.e. dates and time, location, the meeting chair, meeting agenda, follow-up plan and note taking).

• Establish ground rules, roles and responsibilities.

• Establish meeting agendas and preview documentation prior to the meetings.

• Discover ways to maintain effective meetings and ongoing communication.

• Create opportunities to celebrate small wins.

• Share successful stories during the meetings.

• Evaluate and monitor the effectiveness of the team meetings on a regular basis.

• For organizations with multiple sites and locations:

• find a central location for easy access.

• take advantage of video capacity to cut down cost and travel time (e.g. GoToMeeting, OTN).

• Keep in mind that the initial phase of implementation planning may require more frequent meetings to build teams and establish routines.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/35_sample_meeting_agenda.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/18_cbt_cop_terms_of_reference_pathways.pdf


Table of contents


Assessing the evidence

What does it mean to assess the evidence?Before introducing a new practice into an organization, it is critical to go through a process of assessing and selecting an evidence-informed practice (EIP) that will be considered. From the initial needs assessment, the team may come to conclusion that: 1) there is a gap between current services and clients’ needs; 2) a more comprehensive approach is required to apply evidence into practice and 3) without considering organizational context, simply bringing in an off-the-shelf manualized intervention doesn’t guarantee success in implementation.

Evidence-informed practice (EIP) embraces the three sources of evidence to bring theory into practice. This includes research evidence, practitioner experience and judgment and child, youth, family and caregiver preference. When assessing the evidence and selecting an EIP to implement, it is essential to be continuously gathering, sharing and using evidence from these three sources.

Here are some key questions for the core team to consider when assessing the evidence: • How will information be generated and incorporated into the overall program practice?

• How do we capture child, youth, family and caregiver preferences into our current practices?

• Do we have inputs from the child, youth family and caregiver throughout the continuum of the client engagement?

• How do we embrace practitioner experience and judgment in our current practices?

• How do we loop back the information to these three sources?



Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

Literature review guide

Download example

Who should assess the evidence? The core team, with the input of senior leadership and other identified stakeholders, should examine the evidence. Some organizations use the help of students to complete this task, whereas some may utilize existing committees within the organization to assist with the information.

When and why should the evidence be assessed? The evidence should be assessed during the early stages of implementation to ensure the selected EIP addresses the identified needs.

How should evidence be assessed?

1. Assessing research evidenceCore and implementation teams should reference current literature and research before and during the implementation of programs and practices. This will ensure decision-makers and frontline staff are using the most current and up-to-date knowledge to inform their work. There are a number of approaches that could be used to look at research such as completing a literature review and conducting a critical appraisal.

Conducting a literature reviewWhen conducting a literature review, it typically follows four key stages:

1. Search the literature.2. Assess the findings. 3. Summarize the findings. 4. Write the review.

Conducting a critical appraisalCritical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, its value and its relevance in a particular context (Burls, 2009). Although the process of conducting a critical appraisal may be similar to that of a literature review, it usually follows a more systematic and thorough examination of research evidence; as well, the results of the appraisal are analyzed and documented in an organized matter. For example, drawing comparison of studies, categorizing or utilizing rating scales to assess against the criteria.

The National Collaborating Centre for Methods and Tools (NCCMT) has a critical appraisal module that demonstrates how to assess the quality of an intervention (National Collaborating Centre for Methods and Tools, 2010). Click to see NCCMT website.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/21_literaturereview.pdf

http://www.nccmt.ca/en/modules/cais/


Table of contents


Tips and resources

Consider the following:• Organizations that have a partnership with post-secondary institutions may have student placement programs that

include conducting a literature review as part of their placement.

• It is important to make researching the evidence a shared task among team members. This will build capacity within the organization and increase access to research information.

• Start at the Centre’s Where to find evidence page for various search sources.

• Safeguards Training has some information that is helpful for the literature search.

2. Assessing evidence from child, youth, family and caregiver preferenceThe second source of evidence to consider is the preference of children, youth and their families and caregivers - the recipients of your services. By involving youth and their families as partners in treatment planning, you improve your services by ensuring their needs are met and creating solutions that encourage their commitment. Client characteristics may also impact their preferences and responses to the interventions they received, factors such as social and culture differences of these preferences would require attention during the EIP selection. Ultimately, the reason for implementing a new EIP is to improve services and adapt to these diverse needs.

Tips and resources

Consider the following:• Discuss how to include client feedback and preferences, and discover ways to facilitate the involvement of their input

from the planning stage.

• Use pre-surveys to gather client perspectives (preference) before considering a new practice. Conduct a post survey after implementing a new EIP.

• Look for opportunities to communicate how the selected EIP will meet the needs of the client.

• Draw from examples of agencies who are using the proposed EIP and consider the outcomes they have achieved.


http://www.excellenceforchildandyouth.ca/what-we-do/evidence/where-to-find

http://safeguards-training.net/VirtualLibrary.aspx


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Examples from the field

A staff survey for CBT practice -

Pathways for Children and Youth

Download example

A staff survey on DBT - Woodview

Children’s Mental Health

Download example

3. Assessing evidence from practitioner experience and judgmentThe third source of evidence to consider is practitioner experience and judgment. Establishing evaluation practices that capture the collective practitioner clinical experience and judgment can increase implementation capacity and fidelity. In addition, this can support strategies to decrease practitioner’s feelings of isolation.

Tips and resources

Consider the following:• This is an opportunity to engage all staff in the process of

implementation. Organizations can develop surveys to solicit feedback from staff and identify emerging trends through staff feedback.

• Create opportunities for practitioners to ask questions, share experience or express concerns for the proposed practices/intervention. For example, host an open house, lunch & learn or other on-line communication tool to encourage feedback.

• Utilize existing communication strategies to promote feedback loops and share data results. Present the information through team meetings, year-end staff meetings or other forums.

• Create opportunities to draw upon the experience of champions.

• Tap into existing committees or community-of-practice groups to gather current knowledge of the subject.

The Centre provides service assisting organizations in assessing the evidence. Go to Evidence-In-Sight for more information.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/22_cbt_staff_survey_pathways.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/23_dbt_staff_survey_woodview_cmh.pdf

http://www.excellenceforchildandyouth.ca/what-we-do/evidence/evidence-in-sight


Table of contents


Selecting an evidence-informed practice

The process of selecting an EIP can bring about rich conversations that include diverse perspectives. By including the voices of multiple stakeholders, this will ensure the best EIP is selected and will increase staff commitment.

Selecting an evidence-informed practice is a multi-step process with four main steps:1. Select what approach you will take to determine the appropriate EIP. For example, using a critical appraisal or literature

review to gather evidence.2. Identify resources that can help with successful implementation.3. Identify appropriate implementation sites.4. Identify key features of the implementation that must be closely adhered to and monitored.

Similar to the implementation capacity checklist, organizations may also consider other tools, such as a hexagon tool.

An example: Hexagon toolThe Hexagon tool uses six factors to evaluate new and existing interventions and may assist organizations in systematically identifying the right EIP to implement.

Needs: how well the programs or practice might meet identified needsFit: overall fit with current initiatives, priorities, structures and supportResource availability: for training, staffing technology supports, curricula, data systems and administrationEvidence: indicating the outcomes that might be expected if the program or practices are implemented wellReadiness for replication: including expert assistance, number of replications accomplished, exemplars available for observation, and how well the program is operationalizedCapacity to implement: as intended and to sustain and improve implementation over time

Tips and resources

• A thorough process focused on the proposed program or practice will help your team have a productive discussion related to the six areas listed above.

• The information gathered can be used to communicate and egage all staff within the organization.

For more information and tips on how to use the Hexagon Tool, go to the Active Implementation Hub website.

Information from Blasé, K., Kiser, L., and Van Dyke, M. (2013). The Hexagon Tool: Exploring Context. Chapel Hill, NC: National Imple-

mentation Research Network, FPG Child Development Institute, University of North Carolina at Chapel Hill.


http://implementation.fpg.unc.edu/


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

A template designed by the Centre

to help you identify key factors within

your organization.

Download template

Creating an implementation plan

Creating an implementation plan is essential in order to address potential implementation gaps. Effective implementation planning and execution takes an average of three years, provided there is fidelity to the plan as well as to the EIP intervention (Fixsen, Blasé, Timbers and Wolf, 2007).

An implementation plan lets you anticipate challenges (e.g. cost, commitment, responsibility), create a common understanding among staff (e.g. regardless of level of involvement, everyone understands the goal of the program), and tie together all of the steps involved in moving from early planning to sustaining a new practice.

Who should be involved in creating the plan? The core team should be involved in creating the plan. The implementation team may assist with adjusting the plan along the way.

When and how should the plan be used? An implementation plan should be created early on in the implementation process to provide consistency in communication and interpretation for all staff. It is a living document that requires ongoing adjustments and revisions as necessary.

What should be considered when creating an implementation plan?Before creating a plan, it’s important to use a framework that provides an organizing structure that helps make sense of complicated processes. It provides a visual so you can more easily organize and monitor where you are at in terms of the implementation process framework. The following information can be used throughout implementation:

1. Key factors of implementationThere are several frameworks that identify key factors of implementation. At the Centre, we base our description of key factors on the Consolidated Framework for Implementation Research developed by Damschroder et al. (2009). This framework outlines five domains affecting the implementation process:

• Inner setting

• Outer setting

• Intervention characteristics

• Characteristics of individuals

• Process of implementation


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/25_key_implementation_factors.pdf


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

A template designed by the Centre

to help you identify implementation

drivers within your organization.

Download example

A sample implementation plan

template.

Download example

Examples from the field

Implementation plan from Youth

Services Bureau: Trauma Focused-

Cognitive Behaviour Therapy (TF-

CBT).

Download example

Implementation framework from

Woodview Children’s Mental Health:

Dialectic Behaviour Therapy (DBT).

Download example

2. Implementation drivers

Implementation drivers are core features that contribute to successful implementation initatives. The drivers are integrated, occurring simultaneously in order to maximize their impact on practitioner behaviour change. The drivers also compensate for one another whereby a weakness in one component can be offset by strengths in another.

There are three types of implementation drivers (adapted from The National Implementation Research Network, 2013):

• Competency drivers: They help develop, improve and sustain an agency’s ability to implement an intervention. They include staff selection, training, ongoing consultation and coaching as well as staff performance assessment.

• Organizational drivers: These are the mechanisms that create and sustain favourable organizational and system environments for effective service. They include a data system to support decision making and program evaluation, facilitative administration supports and systems interventions.

• Leadership drivers: Successful implementation requires the commitment, involvement and accountability of leaders in the implementation process. Leaders drive the use of outcome data to inform decision making and keep staff motivated.

Find more information on implementation factors and implementation drivers on the Active Implementation Hub website.

Tips and resources

Consider the following:• In addition to the Centre framework, consider the NIRN implementation

framework for further resources and tips.

• Use concrete terms and graphs to visualize implementation activities and timelines.

• Use the organizational tools that staff are familiar with to layout the timeline.

• Some implementation plans may require a more detailed layout and additional actions to carry out activities.

• Evaluation should be considered part of the implementation plan.

• When deciding on the timeline for the implementation initiative, be mindful of other concurrent initiatives and priorities within the organization.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/26_implementation_drivers.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/27_sample_implementation_plan.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/28_implementation_plan_ysb.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/29_implementation_framework_woodview_cmh.pdf

http://implementation.fpg.unc.edu/


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

Questions to ask your purveyor -

National Registry of Evidence-based

Programs and Practices

Download Example

Selecting a purveyor

What is a purveyor? A purveyor is an individual or group of individuals representing a program or practice. These individuals work to develop and implement that program or practice with fidelity and good effect (Fixsen & Blase, 2008). A purveyor, often referred to as a trainer, has knowledge and expertise in a specific area and can be someone from within or outside the organization.

When to select a purveyor?After you have selected an EIP, selecting an appropriate purveyor for the EIP intervention is the next step. A purveyor should be selected before confirming the implementation plan and training dates. The purveyor may provide input to the implementation plan during the planning phase and assist with choosing training dates that are suitable for your organization.

Who should be involved in selecting a purveyor? The core team along with other key staff is involved in selecting the right purveyor. It may be helpful to ask other organizations that have implemented the same EIP for recommendations when choosing purveyors.

Tips and resources

Consider the following:• The Centre of Excellence has an online resource hub where

organizations can access useful information on evidence-informed practices and training opportunities across the province.

Click here to see organizations that have implemented specific evidence-informed practices in Ontario.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/30_questions_to_ask_purveyor_nrebpp.pdf

http://www.excellenceforchildandyouth.ca/resource-hub

http://www.excellenceforchildandyouth.ca/what-we-do/implementation/participants


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Tools and reference

The following are a few useful

templates when completing an

evaluation.

Sample logic model

Sample evaluation framework

Sample evaluation questions

Sample fidelity resources

Ongoing evaluation and monitoring

Program evaluation is the backbone of providing evidence-informed services. It helps you assess what you’ve done, determine where you’re at and plot a course of action for the future. Monitoring your outcomes along the way gives you a real-time snapshot of how far you’ve come. Evaluation allows you to assess how well the program has been implemented in order to meet its objectives. It also shows if and how your clients are changing as a result of the intervention.

Why is evaluation important? Evaluation is a valuable strategy for measuring the success of an EIP, not only in terms of assessing child and youth mental health outcomes, but for measuring the success of the implementation process itself. An effective intervention may fail to provide expected positive outcomes due to a lack of fit within your context, or due to a lack of fidelity to the core intervention components necessary to realize the anticipated outcomes. By evaluating both the process of implementation and the outcomes for your clients, you can ensure you capture the most accurate depiction of your program.

Who should be involved in evaluation? The core team along with any other staff with knowledge or experience in evaluation or research should take part in both the development and ongoing activities of evaluating the program. While many organizations have an evaluation person or team, even those who do not can build internal capacity in this area. The Centre has a number of supports and tools that can assist in the building of such capacity. It is also recommended that organizations consider the inclusion of a student to support with a number of activities including the data collection, analyses and interpretation of results.

What should be evaluated?Evaluation activities should be focused on both the implementation process (including fidelity) as well as the child and youth mental health outcomes of the program/intervention being implemented.

A process evaluation is an important first step as it will describe how the program was developed, including the important factors of implementation as well as how it is operating once implementation is complete. It asks the “what” of the program. For example, typical process evaluation questions would include: What staff competencies were anticipated? Do program staff have those competencies? What are the characteristics of the intended clients to treat? Did you reach the intended target population?

An outcome evaluation assesses the effects that taking part in the program has on the clients you serve, or what the impact was on their mental health.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/36_sample_logic_model_template.doc

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/37_evaluation_framework_template.doc

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/38_sample_evaluation_questions.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/39_sample_fidelity_resource.pdf


Table of contents


HELPFUL TOOLS

DOWNLOAD




your process.

Sample from the field

Engaging staff in understanding a

logic model - Central Toronto Youth

Services

Download example

Tools and reference

A chart suggesting activities and

timelines for the planning phase.

Download example

It asks the “so what” of the program. This type of evaluation is important and should be conducted once your EIP has been fully implemented in order to provide important information about how the clients are improving or not based on their participation in the intervention.

Both implementation and outcome evaluations should be seen as working together to create a more complete picture of what is going on in your program.

When should we evaluate?Evaluation should begin from the early stages of implementation planning and continue through to the full implementation stage. In the early stages, the evaluation should focus more on process questions to assess the success of implementation as it is recommended that a program be fully implemented for at least six months before beginning an evaluation of the outcomes.

Tips and resources

Consider the following:• If you are implementing an EIP that is in a school, consider alignment

of training/ implementation/ evaluation with school calendars and any required ethics reviews.

• Begin thinking about evaluation early on in the implementation process.

• Create an evaluation plan and incorporate it into your implementation plan.

• A program logic model can help to understand the overall structure of a program when you are implementing the new practice/intervention.

• Identify key evaluation questions, indicators and outcomes.

• Utilize an outcome measurement database to help inform your evaluation activities.

The Centre has created an Evaluation Toolkit. Click here for more information.

Implementation activities and timeline

The activities and timeline for implementation in the planning phase will vary depending on the program itself (e.g. during or frequency of program activities), staff capacity (e.g. availability, knowledge and skills on implementation) and organizational resources (e.g. access to research literature, information technology resources). The chart in the helpful tools section presents activities and timelines as milestones that the core teams may consider.


http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/33_engaging_staff_ctys.pdf

http://www.excellenceforchildandyouth.ca/sites/default/files/docs/ISStoolkit/31_implementation_activities_timelines.pdf



Table of contents


Preparing for the next phasesof implementation

Doing phase

Things to consider when preparing for phase two of implementation:

• evaluation framework and measures

• ongoing communication with stakeholders

• ongoing regular core team and implementation team meetings

• intervention staff selection

• training on EIP

• trainer/coach follow-up (consultation, practice, and model)

• implement/adjust supervision model

• implement/adjust coaching system

• ongoing leadership support

• process and system alignment (e.g. changing clinical pathways and referral systems)

• initial process and outcome evaluation (pilot)

• link with experts and communities of practice

• fine tune budget for phase three of implementing EIP(s) (sustaining phase)

• develop a full implementation plan

• update the implementation capacity checklist

• develop a sustainability plan

• develop a knowledge exchange plan

Sustaining phase

Things to consider when preparing for phase three of implementation:

• ongoing communication with stakeholders (e.g. status reports)

• ongoing regular core team and implementation team meetings

• execute full implementation plan (scale up or refine)



Table of contents


• develop/implement knowledge exchange and dissemination plan

• ongoing linkage with experts and communities of practice

• implement a sustainability plan

• align supervision model

• align coaching system

• ongoing leadership support

• process and system alignment

• ongoing process and outcome evaluation

• program reports

• mentor other agencies in implementing EIPs



Table of contents


General tips and strategies on implementing EIPsFinally, here are some general tips and strategies suggested by agencies that reflect their experience in the implementation of EIPs during the planning phase:

“What factors contributed to your success in terms of activities in the planning phase of your implementation journey?”

• A sense that the middle management group is a key facet in the sustainability of an EIP over time

• The core team believed that training key staff early on would both motivate staff and provide useful and cautionary information that would be helpful in avoiding future problem areas

• The use of a collaborative approach / teamwork

• Input from staff who occupy different roles in the agency - all of our thoughts and ideas were heard and considered

• Provide a respectful, safe atmosphere - very understanding team

• Stability and guidance from a dedicated and committed leader

• Existing culture where team members can safely take a risk and challenge other team members

• Comprehensive, accessible and useful tools, training and resources from the Centre

• Having a valid and credible implementation model to follow was KEY

As you go through the planning phase of the implementation, keep the following things in mind:

• Recognize that implementation takes time.

• Shrink the change into manageable steps.

• Use a strength-based approach.

• Become a learning organization.

• Maintain the senior leadership commitment and buy-in throughout the entire process.

• Ensure readiness for change and implementation - create a sense of urgency.

• Celebrate the short-term wins.

• Communicate, communicate, communicate.

• Build a solid implementation plan.

• Track every step of the implementation - monitor the change.

• Embrace teamwork and collaboration.

• Be flexible.

• Have an open communication policy on the core team.

• Share the workload.

• Welcome new and energetic staff to the team.



Table of contents


Additional resourcesDisclaimer

The Centre’s toolkit contains links to sites developed and managed by others. The Centre assumes no responsibility or liability for material on

linked sites and recognizes the material may not be available in both official languages.

Centre implementation resources

Implementation online learning modules

Evaluation toolkits

Measures database

The Centre’s webinars

Child/youth and family engagement in evaluation

Harvard Family Research Project

ReferencesAarons G. A., Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., Cafri, G. (2010). Psychometric properties and U.S. national norms of the Evidence-Based Practice Attitude scale (EBPAS). Psychological Assessment, 22, 356–365.

Aarons, G. A. (2005). Measuring Provider Attitudes Toward Evidence-Based Practice: Consideration of Organizational Context and Individual Differences. Child and Adolescent Psychiatric Clinics of North America. 14(2): 255–271.

Austin, J., M., Claassen, J. (2008). Implementing Evidence-Based Practice in Human Service Organizations: Preliminary Lessons from the Frontlines. Journal of Evidence-Based Social Work. 5(1/2), 271-293.

Burls, A. (2009). What is critical appraisal? Retrieved July 10, 2012, from www.whatisseries.co.uk

Chovil, N. (2009). Engaging Families in Child and Youth Metal Health: A Review of Best, Emerging, and Promising Practices. Retrieved July 10, 2012, from http://www.forcesociety.com/sites/default/files/Engaging%20Families%20in%20Child%20%26%20Youth%20Mental%20Health.pdf.




http://www.excellenceforchildandyouth.ca/resource-hub/measures-database

http://www.excellenceforchildandyouth.ca/resource-hub/archived-learning-events

http://www.hfrp.org

http://www.whatisseries.co.uk

http://www.forcesociety.com/sites/default/files/Engaging%20Families%20in%20Child%20%26%20Youth%20Mental%20Health.pdf

http://www.forcesociety.com/sites/default/files/Engaging%20Families%20in%20Child%20%26%20Youth%20Mental%20Health.pdf


Table of contents


Conner, D. R., & Patterson, R. W. (1982). Building commitment to organizational change. Training and Development Journal, 36, 18-30.

Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., Lowery, J.(2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 4(1),50.

Fixsen, D.L., Blasé, K, Timbers, G.D. and Wolf, M.M. (2007).In Search of Program Implementation: 792 Replications of the Teaching-Family Model. The Behavior Analyst Today, 8(1), 96-114.

Fixsen, D. L., & Blase, K. A. (2008). Drivers framework. Chapel Hill, NC: The National Implementation Research Network, Frank Porter graham Child Development Institute, University of North Carolina.

Fixsen, D. L., Naoom, S.F., Blase, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research. A Synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHL Publication #231).

Godley, S. H., Garner, B. R., Smith, J. E., Meyers, R. J., & Godley, M. D. (2011). A large-scale dissemination and implementation model for evidence-based treatment and continuing care. Clinical Psychology: Science and Practice, 18, 67-83.

Hoagwood, K. E. (2005). Family-based services in children’s mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690-713.

Julian, D.A., Ross, M., & Partridge, C. (2008). Challenges in supporting community implementation of science-based programs: A critical review of local partnerships for success plans. American Journal of Community Psychology, 41(3), 351-360.

Lehman, E., K., W., Greener, M., J., Simpson, D. D.(2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment. 22(4), 197-209.

Kilbourne, A.M., Neumann, M.S., Pincus, H.A., Bauer, M.S., & Stall, R. (2007). Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implementation Science, 2:42.

McKay, M. M., Hibbert, R. Hoagwood, K. E., Rodriguez, J., Murray, L., Legerski, J., et al. (2004). Integrating evidence-based engagement interventions into ‘‘real world’’ child mental health settings. Brief Treatment and Crisis Intervention, 4, 177-186.

Moore, P., Cunning, S., & Roundwaithe, J. (November 2010). Implementation Evidence-Informed Practice – Kinark’s Clinical Transformation Journey. Presentation paper at Children’s Mental Health Ontario Conference, Toronto, ON.

National Collaborating Centre for Methods and Tools. (2010). Critical Appraisal of Intervention Studies. Retrieved May 13, 2005, from http://www.nccmt.ca/en/modules/cais/

Preskill, H. & Torres, R.T.(1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.

Sackett, D., L., Straus, S., E., Richardson, W., S., Rosenberg, W., & Haynes, R., B. (2000). Evidence based medicine: How to practice and teach EBM. (2nd ed.). London: Churchill Livingstone.

The National Implementation Research Network (2013). Implementation Drivers. Retrieved June 19th, 2013 from http://nirn.fpg.unc.edu/learn-implementation/implementation-drivers


http://www.nccmt.ca/en/modules/cais/

http://nirn.fpg.unc.edu/learn-implementation/implementation-drivers

http://nirn.fpg.unc.edu/learn-implementation/implementation-drivers


Table of contents


AcknowledgementsThis toolkit has been developed with the support and expertise of

many people. The contributions of staff from the Centre as well as the

valuable perspectives of the many reviewers from our networks are

greatly appreciated.

Ontario Centre of Excellence for Child and Youth Mental Health 2013 Implementation toolkit. Ottawa, Ontario. www.excellenceforchildandyouth.ca

© Copyright: This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. To view a copy of this license, visit http://creativecommons.org or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.

Centre implementation support team

Ann BarberLaura ConroySandra Huang Del FrariJana Kocourek Mark MacAulayVanessa Pleasance

Internal review team

Charles CarterEvangeline DansecoNatasha McBrearty

External review team

Amber McCartLorraine JeffreyNathalie JohnstonRenee ParsonsKatherine Sdao-Jarvie

Child and youth mental health organizations

Aisling Discoveries Children and Family CentreCentral Toronto Youth ServicesChild and Community ResourcesChildren’s Centre Thunder BayPathways for Children and YouthWoodview Children’s Mental HealthYouth Services Bureau

Suggested citationOntario Centre of Excellence for Child and Youth Mental Health (2013). Implementing evidence-informed practice: A practical toolkit. Ottawa, Ontario.



http://creativecommons.org/licenses/by-nc-nd/2.5/

Documents

Implementing evidence- informed practice · Putting theory into practice 9 ... Evidence-informed practice When implementing evidence-informed practices into the real world ... Leading