Outcome Measurement 101web.mnstate.edu/ginther/499_OA/499_OA_Slides/UWCC_WS_D2.pdfActivities Activities 33--kk Mission Why you exist? Overarching philosophy Gross organizational direction

1

Outcome Measurement 101Outcome Measurement 101

47

Thinking differentlyThinking differently

48

Critical ThinkingCritical Thinking

49

…and during Step 1 we …and during Step 1 we practicedpracticed thinking differently!thinking differently!

50

2

...now we will apply actual ...now we will apply actual QuasiQuasi--sciencescience to our programto our program

51

Step 2Step 2ChCh

52

Choose your Choose your outcome outcome measuresmeasures

“Agency”“Agency”An organization that hosts one An organization that hosts one or more individual programsor more individual programs

53

“Program”“Program”••Occurs in the context of an Occurs in the context of an agency or larger agency or larger organizationorganization••A program will likely produce A program will likely produce

54

p g y pp g y pmultiple outputsmultiple outputs

3

“Program”“Program”A set of related activities and A set of related activities and outputs directed at common outputs directed at common or closely related purposes that a or closely related purposes that a meaningful portion of themeaningful portion of the

55

meaningful portion of the meaningful portion of the agency’s resources agency’s resources is is dedicated dedicated to achieveto achieve

““Output”Output”The The direct direct productproduct of program activities!of program activities!

(E i t )(E i t )

56

(E.g., services, etc.)(E.g., services, etc.)

For this exercise, For this exercise, Choose one Choose one

57

Choose one Choose one organizing program organizing program

outputoutput58

4

SASA

EsteemEsteem

Determine 3 unmet needs it addressesDetermine 3 unmet needs it addresses

BelongingBelonging

SafetySafety

PhysiologicalPhysiological59 60

Construct its logic model Construct its logic model

61 62

5

““InputInput””A A dedicated resource the program dedicated resource the program consumesconsumes!!

63

“Activity“Activity””“Behaviors” “Behaviors” to to createcreate the thing(s) the thing(s) a program provides to fulfill its a program provides to fulfill its mission. mission.

64

((E.gE.g., talking, observing, assessing, ., talking, observing, assessing, demonstratingdemonstrating, reasoning, teaching, , reasoning, teaching, transporting, etctransporting, etc.).)

ALSOALSO:: How the program turns How the program turns inputs into productsinputs into products!!

Logic ModelLogic Model

65

Program Logic ModelProgram Logic ModelTemporal-outcome Hypothesis

66

IfIf XX ThenThen

OutcomeOutcome--PPOutcomeOutcome--II

OutcomeOutcome--DD

InputsInputs ActivitiesActivities OutputsOutputs

YY ss

6

67 68

PerformancePerformance--Based ManagementBased Management

Thinking differentlyThinking differently

69

Critical ThinkingCritical Thinking

70

7

JJ

71

Go Go FishinFishin’’

“Client outcome fishing”“Client outcome fishing”A process whereby organizational A process whereby organizational members analyze a program’s members analyze a program’s components, stakeholders, and components, stakeholders, and strategic statements to determinestrategic statements to determine

72

strategic statements to determine strategic statements to determine valuable client outcomes worth valuable client outcomes worth assessingassessing

“Client outcome fishing”“Client outcome fishing”This process is crucial because scarce This process is crucial because scarce resources are spent… resources are spent…

11 evaluating the worth of various clientevaluating the worth of various client

73

1.1. evaluating the worth of various client evaluating the worth of various client outcomesoutcomes

2.2. choosing those with the most utility, then choosing those with the most utility, then 3.3. eventually measuring and tracking the eventually measuring and tracking the

selected client outcomes over timeselected client outcomes over time

Brainstorm possible client outcomesBrainstorm possible client outcomes••Proximal (4)Proximal (4)••Intermediate (4)Intermediate (4)••Distal (4)Distal (4)

74

8

Client outcome ListClient outcome List

75

Sources for client outcome ideasSources for client outcome ideas••Internal key informantsInternal key informants

••Folks involved in target output transactionFolks involved in target output transaction••ClientsClients••StaffStaff

76

••VolunteersVolunteers••Board membersBoard members

77 78

9

79 80

State GovernmentState Government

L l G tL l G tFederalFederal

CourtsCourts

External sources for client outcome ideasExternal sources for client outcome ideas

81

Local GovernmentLocal Government

NonNon--governmentalgovernmentalagenciesagencies

AdvocatesAdvocatesClientsClients(current/former)(current/former)

CorporationsCorporations

Federal Federal GovernmentGovernment

External key informantsExternal key informants••StudentsStudents••Neighborhood/Community membersNeighborhood/Community members••Volunteers Volunteers (other program outputs, community)(other program outputs, community)••Staff Staff (from other program outputs)(from other program outputs)

82

••Clients Clients (from other program outputs)(from other program outputs)••Other organizationsOther organizations••Government representativesGovernment representatives••FundersFunders

10

83 84

85

Sources for outcome ideasSources for outcome ideas••Program documentsProgram documents

86

11

Client outcomes must be Client outcomes must be grounded in grounded in “Program values”“Program values”

87

““Program values”Program values”Organizational beliefs that guide Organizational beliefs that guide how programs invest resources how programs invest resources to achieve its mission, goals, to achieve its mission, goals, objectives & client outcomesobjectives & client outcomes

88

objectives, & client outcomesobjectives, & client outcomes

Actually written, or based on your impressionActually written, or based on your impression

Do they suggest any unique program values?Do they suggest any unique program values?

89

Geographic regions embrace certain valuesGeographic regions embrace certain values

Populations uniquely embody certain valuesPopulations uniquely embody certain values

Certain values align with some social problemsCertain values align with some social problems

“Valued client outcomes”“Valued client outcomes”Client outcomes the organization Client outcomes the organization embrace as evidenced in its embrace as evidenced in its strategic statements, resource strategic statements, resource allocation patterns andallocation patterns and

90

allocation patterns, and allocation patterns, and evaluation focievaluation foci

12

Hierarchy of PurposeHierarchy of Purpose––MissionMission

––GoalsGoals••““OutcomeOutcome” Objective” Objective

••““ProcessProcess”” ObjectiveObjective

91

•• ProcessProcess ObjectiveObjectiveActivity 1Activity 1Activity 2Activity 2Activities Activities 33--kk

MissionMissionWhy you exist?Why you exist?Overarching philosophyOverarching philosophyGross organizational directionGross organizational direction

92

“The “The SynerjexSynerjex Treatment Treatment Center is dedicated to Center is dedicated to improving the learning improving the learning

experiences of troubled teensexperiences of troubled teens

A sample “A sample “MissionMission””

93

experiences of troubled teens experiences of troubled teens and helping them move and helping them move toward living fulfilling, toward living fulfilling,

independent lives.”independent lives.”

MissionMissionDo you see any key concepts Do you see any key concepts worth measuring in worth measuring in youryourprogram's mission statement?program's mission statement?

94

13




95


GoalsGoalsWill have severalWill have severalLongLong––Range Range BenefitsBenefitsGeneral General Statements Statements of Program intentof Program intentBenchmarksBenchmarks to Judge Accomplishmentsto Judge Accomplishments

96

Benchmarks Benchmarks to Judge Accomplishments to Judge Accomplishments

GoalsGoalsReflects mission Reflects mission Provide directionProvide directionGuides outcomes objectives soughtGuides outcomes objectives soughtReflectsReflects

97

Reflects… Reflects… Client’s, applicant's, staff’s Client’s, applicant's, staff’s inputsinputsProgram Program activitiesactivities

A sample “A sample “GoalGoal””

“The “The SynerjexSynerjex Treatment Treatment Center will collaborate with Center will collaborate with Moorhead Public Schools in Moorhead Public Schools in

98

developing after school developing after school mentoring programs for truant mentoring programs for truant

teens.”teens.”

14

GoalsGoalsDo you see any key concepts Do you see any key concepts worth measuring among worth measuring among youryourprogram’s goals?program’s goals?

99




100


Objectives Objectives -- are always….are always….••Specific Specific –– specifies specifies

––1 result to be accomplished1 result to be accomplished––Tells Tells whatwhat & & whenwhen

••Measurable / evaluable / verifiableMeasurable / evaluable / verifiable

101

Measurable / evaluable / verifiableMeasurable / evaluable / verifiable••Practical / feasible / logicalPractical / feasible / logical••Time bound Time bound (more immediate)(more immediate)

““OutcomeOutcome” objectives” objectives––Describe clients targeted & served Describe clients targeted & served ––Are Are clearclear , , conciseconcise, , measurablemeasurable––Unique to your approachUnique to your approach

WhoWho will be targetedwill be targeted

102

Who Who will be targetedwill be targetedHowHow (in what way) they will change(in what way) they will changeWhatWhat will be donewill be doneWhen When it will occurit will occurWhere Where it will happenit will happen

15

““OutcomeOutcome” objectives” objectives•• Program impact on unmet client needProgram impact on unmet client need•• Includes time frameIncludes time frame•• Targeted client systemTargeted client system•• Expected outcomeExpected outcome

103

•• Expected outcomeExpected outcome•• Outcome indicatorsOutcome indicators

A sample “A sample “OutcomeOutcome” objective” objective

“Seventy percent (70%) of “Seventy percent (70%) of teen students participating in teen students participating in SynerjexSynerjex onon--site aftersite after--school school

104

mentoring groups will reduce mentoring groups will reduce their truancy rate by 100% by their truancy rate by 100% by

June 1, 2010”June 1, 2010”




105


““ProcessProcess” objectives” objectivesPertains to… Pertains to…

Program startProgram start--upupOngoing program operationOngoing program operationProgram growthProgram growth

106

Program growthProgram growth

16

““ProcessProcess” objectives” objectivesFocuses on Focuses on

Program implementationProgram implementationStaff developmentStaff developmentService delivery etcService delivery etc

107

Service delivery, etc.Service delivery, etc.

A sample “A sample “ProcessProcess” objective” objective

“The “The SynerjexSynerjex Treatment Center Treatment Center will implement fifteen teen afterwill implement fifteen teen after--

school mentoring groups perschool mentoring groups per

108

school mentoring groups per school mentoring groups per school by March, 2010 with 2 school by March, 2010 with 2

mentors and 15 students each”mentors and 15 students each”

““ProcessProcess” Objectives” ObjectivesDo you see any key concepts Do you see any key concepts worth measuring among worth measuring among youryourprogram’s process objectives?program’s process objectives?

109 110

17

111 112

Environmental Environmental FactorsFactors

113

••Factors Factors such as the client’s …such as the client’s …••Cultural contextCultural context••Family lifeFamily life••Peer influencePeer influence••EnvironmentEnvironment

114

••SchoolSchool••WorkplaceWorkplace

18

••We use “E” to show environmental factorsWe use “E” to show environmental factors

••Environmental factors are forces that Environmental factors are forces that influence client outcomes (Y)influence client outcomes (Y)

••We consider these forces to understand We consider these forces to understand our true outcome effectour true outcome effect

W h i i h iW h i i h i

115

••We may want to enhance or mitigate their We may want to enhance or mitigate their cumulative impact on clientscumulative impact on clients

••Factors may be proximal, intermediate, & Factors may be proximal, intermediate, & distaldistal

YY

••We can even model these factorsWe can even model these factors

EEpp

YY

116EEiiEEdd

QuestionQuestion::••Is their effect on client Is their effect on client outcome random or outcome random or patterned with patterned with program program efforts and/or client efforts and/or client characteristics?characteristics?

117

••To answer this question we To answer this question we must collect data on must collect data on environmental concernsenvironmental concerns

••Identify environmental factors Identify environmental factors {+,{+,--} affecting participant } affecting participant outcomeoutcome

••Place them on a “factors list”Place them on a “factors list”••Then complete the worksheet: Then complete the worksheet: Cake AssessmentCake Assessment

118

Cake AssessmentCake Assessment

19

119

[If X Then Y ]

ADVICE!ADVICE!Have management supportHave management support

Board, DirectorBoard, DirectorW itt b l + t ibl t!W itt b l + t ibl t!

120

Written, verbal + tangible support!Written, verbal + tangible support!

ADVICE!ADVICE!Keep management advisedKeep management advised

Board, DirectorBoard, DirectorBrief updates reports!Brief updates reports!

121

Brief updates, reports!Brief updates, reports!Promotes Promotes

ownershipownershipknowledgeknowledgecommitmentcommitment

Ask yourselfAsk yourself--Does the outcome reflect the program logic?Does the outcome reflect the program logic?Can the program reasonably and significantly Can the program reasonably and significantly

influence the outcome even though it cannot influence the outcome even though it cannot control it?control it?

Would outcome measurement help identify Would outcome measurement help identify program success and pinpoint problems?program success and pinpoint problems?

122

p g p p pp g p p pWill “others” accept this as a “valid” and “reliable” Will “others” accept this as a “valid” and “reliable”

program outcome?program outcome?Do “intermediate” and “distal” outcomes Do “intermediate” and “distal” outcomes

represent meaningful benefits or changes in represent meaningful benefits or changes in client’s quality of life?client’s quality of life?

20

Get stakeholder feedback on your logic Get stakeholder feedback on your logic model and the outcomes model and the outcomes selected for measurementselected for measurement

123

Your Outcome Assessment Committee?Your Outcome Assessment Committee?••ChairChair••RecorderRecorder••Data personData personC tC t

124

••Computer personComputer person••Client representativeClient representative

Step 3Step 3Specify indicatorsSpecify indicators

125

Specify indicators Specify indicators for your outcomesfor your outcomes

Recall that Recall that conceptsconceptsare not directly are not directly measurablemeasurable

126

21

IndicatorIndicator exerciseexercise““Your dream HouseYour dream House””

127

IndicatorIndicator” ” exerciseexercise••““Great ShowGreat Show””

128

List List indicatorsindicators for these for these conceptual outcomesconceptual outcomes

••“Healthy lifestyle”“Healthy lifestyle”••“Increased social integration”“Increased social integration”••“Safe neighborhood”“Safe neighborhood”

129

•• Safe neighborhoodSafe neighborhood••“Good conflict resolution skills”“Good conflict resolution skills”

••““IndicatorsIndicators” signal target ” signal target outcomesoutcomes••““IndicatorsIndicators” are ” are measurable measurable thingsthings••““IndicatorsIndicators” signal ” signal outcome achievementoutcome achievement••““IndicatorsIndicators” are also “variables”” are also “variables”

130

22

Thus…Thus…••We assess We assess proxyproxy indicatorsindicators to quantify concepts to quantify concepts ••ProxyProxy indicatorsindicators are concrete and measurableare concrete and measurable

••Easy:Easy: behavioral indicatorsbehavioral indicators

131

Easy: Easy: behavioral indicators behavioral indicators ••Hard: Hard: knowledge, feelingsknowledge, feelings, emotions, , emotions, attitudesattitudes

••And “proxy IndicatorsAnd “proxy Indicators” are” are……………………….……………………….

“Variables”!“Variables”!

132

““Indicator variablesIndicator variables” ” signify target signify target outcomeoutcome

133

““Indicator variablesIndicator variables” ” are measurableare measurable

134

23

••““Indicator variablesIndicator variables” signal outcome ” signal outcome achievementachievement

135

••ThusThus, indicators measure , indicators measure ““program successprogram success””••““SuccessSuccess” targets are hypothesized in ” targets are hypothesized in outcome objectivesoutcome objectives••““SuccessSuccess” is usually expressed as a ” is usually expressed as a number, percent, statistic, etc. of persons number, percent, statistic, etc. of persons achieving some preachieving some pre--specified targetspecified target

136

achieving some preachieving some pre--specified targetspecified target

••““SuccessSuccess” = ” = worthworth

Measuring Measuring outcome outcome indicatorsindicators

137

indicatorsindicators

What to collect?What to collect?A single conceptual A single conceptual outcome will likely have outcome will likely have

138

yyseveral indicatorsseveral indicators

24

139

II2 2 Supports big governmentSupports big governmentII11 Voted for Gore?Voted for Gore?

140

II1 1 HerosHeros??II3 3 Support s high taxes?Support s high taxes?

We will measure 4 proxy We will measure 4 proxy indicators for each of 4 indicators for each of 4

What to collect?What to collect?

141

focus client outcomes to focus client outcomes to ensure confidence we are ensure confidence we are validly and reliably validly and reliably indicating the conceptindicating the concept

142

25

143 144

Documents

Outcome Measurement 101web.mnstate.edu/ginther/499_OA/499_OA_Slides/UWCC_WS_D2.pdfActivities Activities 33--kk Mission Why you exist? Overarching philosophy Gross organizational direction