Community-based Evaluation Methods and Practice

Community based evaluation Community based evaluation methods & practicemethods & practiceProgram Science in PracticeProgram Science in Practice

CAHR 2015 Ancillary EventCAHR 2015 Ancillary Event

April 30, 2015April 30, 2015

Javier MignoneJavier MignoneJavier.mignone@umanitoba.caJavier.mignone@umanitoba.ca

Outline of the presentationOutline of the presentation

- Clarifying the terminologyClarifying the terminology- Issues that must be addressedIssues that must be addressed- Program evaluation: a seven Program evaluation: a seven

step processstep process- Examples of collaboration Examples of collaboration

J Mignone - University of Manitoba - 2015 2

What is program What is program evaluation?evaluation?

““Program evaluation is the Program evaluation is the systematic collection of systematic collection of informationinformation about the activities, about the activities, characteristics, and outcomes of characteristics, and outcomes of programs to make judgments programs to make judgments about the program, improve about the program, improve program effectiveness, and/or program effectiveness, and/or inform decisions about future inform decisions about future programming.” (Patton, 1997)programming.” (Patton, 1997)

J Mignone - University of Manitoba - 2015

Why?Why?“…“…..for use by specific peoplefor use by specific people to to

reduce uncertainties, improve reduce uncertainties, improve effectiveness, and make effectiveness, and make decisions with regard to what decisions with regard to what those programs are doing and those programs are doing and affecting.” It means, “gathering affecting.” It means, “gathering data that are meant to be, and data that are meant to be, and actually are, used for program actually are, used for program improvement and decision improvement and decision making.” (Patton, 1997)making.” (Patton, 1997)

J Mignone - University of Manitoba - 2015

Difference between Difference between evaluation and researchevaluation and researchBasic scientific research is Basic scientific research is

undertaken to discover new undertaken to discover new knowledge, test theories, establish knowledge, test theories, establish truth, and generalize across time and truth, and generalize across time and space.space.

Program evaluation is undertaken to Program evaluation is undertaken to inform decisions, clarify options, inform decisions, clarify options, reduce uncertainties, and provide reduce uncertainties, and provide information about programs and information about programs and policies within contextual boundaries policies within contextual boundaries of time, place, values, and politics.of time, place, values, and politics.

J Mignone - University of Manitoba - 2015

Difference between Difference between evaluation and research evaluation and research (cont’d)(cont’d)

The difference between research The difference between research and evaluation has been and evaluation has been described by Cronbach and described by Cronbach and Suppes as: Research is aimed at Suppes as: Research is aimed at truth; Evaluation is aimed at truth; Evaluation is aimed at action. action.

J Mignone - University of Manitoba - 2015

Core questionsCore questions

Who uses the systematically Who uses the systematically produced information?produced information?

How is the systematically produced How is the systematically produced information actually used?information actually used?

J Mignone - University of Manitoba - 2015

Community-based Community-based evaluationevaluationWhat community? What community?

◦ PlacePlace◦ IdentityIdentity◦ InterestInterest

What does community-based mean?What does community-based mean?◦ It depends on the purpose and the processIt depends on the purpose and the process

J Mignone - University of Manitoba - 2015 8

Community-based evaluation Community-based evaluation (cont’d)(cont’d)

Purpose of evaluationPurpose of evaluationProcess of evaluationProcess of evaluation

J Mignone - University of Manitoba - 2015 9

Some specific issues to Some specific issues to addressaddressExpectations about evaluationExpectations about evaluationPlural or conflicting agendasPlural or conflicting agendasTrustTrustFunding requirements and agenciesFunding requirements and agenciesEvaluation plan developmentEvaluation plan developmentCultural interpretations/knowledgeCultural interpretations/knowledge

J Mignone - University of Manitoba - 2015 10

Issues to address (cont’d)Issues to address (cont’d)

Appropriate methodology Appropriate methodology TimelinesTimelinesOwnership of data (e.g., OCAP)Ownership of data (e.g., OCAP)PoliticsPoliticsParticipationParticipationEthicsEthicsUse/dissemination of findingsUse/dissemination of findings

J Mignone - University of Manitoba - 2015 11

Program evaluation: A seven Program evaluation: A seven step processstep process

1.1. Identify the primary intended users of Identify the primary intended users of the evaluationthe evaluation

2.2. Identify and focus the relevant Identify and focus the relevant evaluation questionsevaluation questions

3.3. Make design methods and Make design methods and measurement decisionsmeasurement decisions

4.4. Collect dataCollect data5.5. Organize data for stakeholder analysisOrganize data for stakeholder analysis6.6. Involve users in interpretation of Involve users in interpretation of

findingsfindings7.7. Facilitate intended use by intended Facilitate intended use by intended

usersusersJ Mignone - University of Manitoba -

2015 12

Examples of collaborationExamples of collaboration

Summer Institute on Program Summer Institute on Program Evaluation (University of Manitoba)Evaluation (University of Manitoba)

Community Health Information and Community Health Information and Research Partnerships (CHIRP) Research Partnerships (CHIRP) (University of Manitoba students)(University of Manitoba students)

J Mignone - University of Manitoba - 2015 13

??????

J Mignone - University of Manitoba - 2015 14

ReferencesReferences Chomsky N (2003) Chomsky N (2003) Necessary illusions: Thought control in democratic societiesNecessary illusions: Thought control in democratic societies . .

Toronto: House of Anansi Press.Toronto: House of Anansi Press. First Nations Centre at NAHO (2007) OCAP: Ownership, control, access and First Nations Centre at NAHO (2007) OCAP: Ownership, control, access and

possession. NAHO: Ottawapossession. NAHO: Ottawa Horne T (1995) Horne T (1995) Making a difference: Program evaluation for health promotionMaking a difference: Program evaluation for health promotion. .

WellQuest Consulting: Edmonton.WellQuest Consulting: Edmonton. Kaufman Kantor G & Kendall-Tackett K (2000) Kaufman Kantor G & Kendall-Tackett K (2000) A guide to family intervention and A guide to family intervention and

prevention program evaluationprevention program evaluation. University of New Hampshire. Electronic version.. University of New Hampshire. Electronic version. King JA, Stevahn L, Ghere G & Minnema J (2001) Toward a taxonomiy of essential King JA, Stevahn L, Ghere G & Minnema J (2001) Toward a taxonomiy of essential

program evaluator competencies. program evaluator competencies. American Journal of Evaluation, 22American Journal of Evaluation, 22(2), 229-(2), 229-247.247.

Morra Imas LC & Rist RC (2009) The road to results: Designing and conducting Morra Imas LC & Rist RC (2009) The road to results: Designing and conducting effective development evaluations. The World Bank: Washington DC.effective development evaluations. The World Bank: Washington DC.

Patton MQ (1997) Patton MQ (1997) Utilization-focused evaluation: The new century text.Utilization-focused evaluation: The new century text. 3 3rdrd edition, Thousand Oaks: Sage.edition, Thousand Oaks: Sage.

Patton MQ (2008) Patton MQ (2008) Utilization-focused evaluationUtilization-focused evaluation, 4th Edition, Thousand Oaks: , 4th Edition, Thousand Oaks: Sage. Sage.

Patton MQ (2012) Essentials of utilization-focused evaluation. Thousand Oaks: Patton MQ (2012) Essentials of utilization-focused evaluation. Thousand Oaks: Sage. Sage.

The Provincial Centre for Excellence for Child and Youth Mental Health at CHEO The Provincial Centre for Excellence for Child and Youth Mental Health at CHEO (2007) (2007) Doing more with program evaluation: A toolkit for conducting program Doing more with program evaluation: A toolkit for conducting program evaluationevaluation. Ottawa, Ontario . Ottawa, Ontario www.onthepoint.ca

Shadish WR, Cook TD, Campbell DT (2002) Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental Experimental and quasi-experimental designs for generalized causal inferencedesigns for generalized causal inference. Boston: Houghton Mifflin.. Boston: Houghton Mifflin.

Suchman EA (1967) Evaluative research: Principles and practice in public service Suchman EA (1967) Evaluative research: Principles and practice in public service and social action programs. New York: Russell Sage.and social action programs. New York: Russell Sage.

J Mignone - University of Manitoba - 2015 15