What is the Centre?KnowledgeBringing people and knowledge together to promote the best mental health and well-being for every child and youthCapacityPartnershipsFull continuum of effective and accessible mental health services for children and youth.
Webinar overviewThis webinar will be recorded, so that it can be available on the Centres website as an educational resourceThe slides and any supplementary materials will be sent to participants at the end of the sessionThe session will take approximately 45 minutes including time for questionsYou may ask questions or make comments at the end of the session by:Typing your question or comment
Engaging stakeholdersFacilitatorsIlana Smyth, Research Associate, Centre of ExcellencePaula Cloutier, Research Associate, CHEOWho are the stakeholders of an evaluation and how do they influence evaluation? This webinar will discuss the importance of identifying stakeholders and how to engage them in planning and conducting the evaluation.
OutlineStages in engaging stakeholders:Identifying stakeholders and their needsTechniques to identify stakeholdersManaging expectations and taking actionShared experience from recent evaluation activitiesDiscussions and questions
Communication with StakeholdersCommunicating with stakeholders is important throughout an evaluationIntended users of the evaluation are more likely to use the results if they have been involved in the process
Identify stakeholdersStakeholders are those who can affect or are affected by an evaluation process and/or its findingsIndividualsGroupsOrganizations
Stakeholders are individuals Individuals within organizations Individuals within groups
Identify stakeholdersConsider stakeholders who may have a negative influence on the evaluationIncluding those who oppose using resources for evaluation purposesIncluding those who may feel threatened or experience anxiety around certain aspects of the evaluation
Identify stakeholdersKeep the definition of stakeholders broad at first, however its difficult for an evaluation to answer all questions for all of those interested.
Necessary to then narrow down the evaluation questions and focus on a more manageable list of potential stakeholdersPrimary intended users or key evaluation stakeholders
TechniquesList evaluation stakeholdersBrainstorm a list of individuals or groups
TechniquesStakeholder influence diagram
Stakeholder:Stake or Interest in the Program: What do they want to get out of the program, and/orWhat do they want the program to produce?Stake or Interest in the Evaluation What do they want to get out of the evaluation, and/orWhat do they want the evaluation to produce?
TechniquesPower versus interest gridPowerInterest
Subjects Have a significant interest but little powerPlayers Have a significant interest and substantial powerCrowd Have little interest and not much powerContext Setters Have substantial power, but little direct interest
Document stakeholder needsAdvance planning is importantEstablish as early as possible the role and function of stakeholders Establish ground rules and to what extent stakeholders are involved in the evaluationEstablish timelines for involvement
Manage stakeholder expectationsStakeholder management ways of working with positive as well as negative stakeholdersBetter referred to as stakeholder engagementBegins in the early stages and continues through development and implementation phases
Engaging stakeholders after the evaluationTell your stakeholders how things went, both with the program and with the evaluation Follow up an evaluation report with a phone call or an in-person meeting, especially with primary stakeholdersFocus discussion on what the program has learned and how it intends to grow and improve, rather than just reporting on outcomes
Engaging stakeholders after the evaluationInvite stakeholders to brainstorm ideas on how to strengthen your program based on the evaluation findings Send out a final evaluation report to all levels of stakeholders upon the program's completion
Specialized Psychiatric & Mental Health Outreach Services for Children and Youth in Eastern Ontario
SPMHS Outreach ServicesServices designed for children and youth who have complex mental health disorders which result in serious psychiatric, emotional and/or behavioural disturbanceCollaborative care approachRole is to reinforce, support, provide consultation and training to community based MH servicesDeliver services on a planned basis or by request either through travel to the community or by way of telehealth and telepsychiatry
Who are the Stakeholders? Funder: MCYS Service Providers: CHEO (age 0-15), ROMHC (age 16-18)-Clinicians (psychologist, psychiatrist, psychological associate, psychiatric nurse, occupational therapist) Rural Community Agencies: Providing MH services Rural Physicians: referring Consumers: Children, Youth, Caregivers
Discharge to Referring Physicians and/or Community
SPMHS Centralized IntakeAssess and refer to Appropriate Service/Program:OutreachOutpatientInpatientEmergency DepartmentAppropriate Community Services
ConsultationAssessmentClinical Care PlanTreatmentRecommendationsRenfrewPrescott/RussellSD&GNorth LanarkOttawaReferring Physicians
Not appropriate for SPMHS ServicesOutreach TeamsNorth Grenville
Engaging the StakeholdersConsult with funders, service providers and consumers regarding outreach services (prior to program development)Clarify needs, expectations and capacity on part of both specialized and community mental health care partners (prior to program development)Develop services in response to identified local needs and local culture (be adaptive, enhance existing community services, address issues of role definition, responsibility.)Evaluate outcomes leading to ongoing refinement and improvement of service (ideally build evaluation in right from the start)
Getting Service Providers On boardStaff meeting(s) to discuss program and evaluation with plenty of time for Q & AIt is essential that those involved see the evaluation as meaningful and purposefulAppropriate training be offered as necessaryFocus groups to get feedback from cliniciansWillingness to make changes along the way (evaluation is not research)
Getting Service Providers On board with the Evaluation: Success?
CountyAccepted to OutreachCompliance with evaluation Stormont, Dundas & Glengarry 14622 (15%)North Lanark1328 (1%)Renfrew7451 (69%)Prescott-Russell471 (2%)North Grenville1912 (63%)Ottawa55 (100%)
Getting Clients On board with the Evaluation: Success? Satisfaction Questionnaires (17% completion) Consider more than one way to get feedback (e.g., consumer focus groups)Getting Agencies On board with the Evaluation: Success? Share evaluation outcomes with them (2/5 who wanted to hear anything)
Keeping your Stakeholders EngagedCommunication is keyContinually demonstrate the relevance of the evaluation (one size doesnt fit all)Keeping the Evaluators Engaged Dont have an all or none attitude (view it as a process) Identify and celebrate the successes along the way (the smaller achievements add up) Learn from the failures (be willing to modify and try something else) Plan your next steps (focus groups with referring physicians)
SummaryIdentify key stakeholders at the beginning of evaluation activities as much as possibleEngage stakeholders in evaluation activities from the beginning Document stakeholders involvementOngoing communication is keyFlexibility is a necessity
For more informationIlana SmythResearch Associate, Centre of Excellence613.737.7600 Ext. email@example.comPaula CloutierResearch Associate, CHEO613.737.7600 Ext. firstname.lastname@example.org
**Introduce everyone present at the Centre.****Whether you are planning an evaluation, piloting data or collecting results, communicating with stakeholders is essential.*This is a broad definition so that the full spectrum of stakeholders can be identified before you start to break down the primary intended users of the organization.
Examples of individuals are: past, current and future grant recipientsExamples of groups are: advisory groups, BoardExamples of organizations are: those who offer similar or complementary programs
Stakeholders are individuals: For example, when identifying stakeholders some groups will identify the government as a stakeholder. This could be the case if a certain government policy affects the program, for example. While this may be true it is more accurate to name individuals within whatever level of government (eg. Municipal) that youre interested in. The same may hold true for identifying the public or taxpayers as stakeholders. It is more manageable to name individuals within these groups.
Bryson, J.M., Patton, M.Q. & Bowman, R.A. (2011). Working with evaluation stakeholders: A rationale, step-wise approach and toolkit. Evaluation and Program Planning, 34, 1-12.
*Discuss how evaluation may seem threatening or intimidating to some stakeholders (such as program staff) if it is not introduced and explained at the beginning of the evaluation process.Explore the fears certain stakeholders have and explain the intention of the evaluation.Gain an understanding of what experience staff have with evaluation.*Patton refers to this group as primary intended users however we can use the terminology key evaluation stakeholders as well*These slide outline techniques in identifying stakeholders.List evaluation stakeholders this involves the evaluation gro