DIY Evalua*on AnnualEvalua)onLabWorkshop

MelissaBinder,Director,NMEvalua)onLab@UNMApril21,2017

2of26

You may already have a really great evalua*on system in place

4principles

8bestprac)ces

3of26

Principle #1: Evalua*on should serve the interests of your organiza*on

• Addresswhatismostimportanttoorganiza)on:yourcorevalues,yourguidingmission,whatyoureallycareabout

• Maybedifferentfromtheinterestsofyourfunders

4of26

BEST PRACTICE: measure what you care about

Goalsandobjec)vesfrommissionstatements,logicmodels,andtheoriesofchange

5of26

Principle #2: Evalua*on should be integrated into your day-to-day opera*ons

• Datacollec)onshouldbepartofyourrou)neopera)ons–aSendance,clientsa)sfac)on,dischargesummaries• Withanevalua)onsprocessinplace,youwon’tbegoingonadatahuntevery)meyouneedtowriteagrantreport

6of26

BEST PRACTICE: data collec*on should not be a burden

•  Yourdatacollec)onshouldn’tposeanundueburdenonstafforpar)cipants•  Youreallydon’tneedmorethantwopointsin)meforeachpar)cipant.IntermediatedataisoXenTMI.•  Ifyourweeklypar)cipantshavetofilloutthesamesa)sfac)onsurveyeveryweek,theyaregoingtoprovidelessthough[ulfeedback

7of26

Principle #3: Have a feedback loop

Collectdata

Reviewdata

Adjustprogram

Ifyouarecollec)ngdata,reviewingitandadjus)ngyourprogram,youaredoingevalua)onright.

8of26

BEST PRACTICE: integrate data into decision making

• Reportandreviewdataonaregularcycle•  Staffmee)ngs•  Annualreport•  BoardofDirectors

9of26

Principle #4: Evalua*on should feel sa*sfying and valida*ng Andmaybeevenfun.

10of26

BEST PRACTICE: evalua*on data is useful.

Evalua)onprovidessuppor)ngdocumenta)onforwhatyouknowyouaredoingright...

...andilluminatesthewayforward

DATA COLLECTION & ANALYSIS

12of26

Measure what you care about

13of26

Mission statement

ArtWorksseekstofulfillthefollowingoutcomes:

•  Teachersbecomeengagedwiththeartsandlearntointegrateartintoclassroomlessonsandpedagogy.

•  Usingencounterswiththearts,students:o  Prac)ceanalysis,discussionandapprecia)onforawidevarietyofartformsandculturaltradi)ons,

o  Expandtheirvocabularyandprac)celiteracyskillssuchasself-expressionbothverballyandinwri)ng,

o  Developnewskillsinreasoning,boosttheirconfidenceands)mulatetheirimagina)on.

14of26

Logic model

Inputs

HumanCapitalPB&JFamilies

LeadershipteamDirectservicestaff

SupportstaffBoardofDirectorsVolunteers&Interns

Professionaldevelopment

FinancialCapitalBraided&leveragedapproachtofinancial

sourcesPrivatedonations

Government&privategrants

SocialCapitalReputationincommunity,

Communitypartners&funders,Evaluation

Accreditation

PhysicalCapitalPhysicalsitesembeddedincommunity,Furniture,

Medicalsupplies,Communicationequipment

Transportation,FacilitiesandMaintenance

Interactionswithothersaregenuinelyhopeful,welcoming,respectful,

nonjudgmental,positiveandvalidating

Responsiveness&adaptabilityallowsustomeetpeoplewheretheyare,helpthem

buildontheirstrengthsandsupportthe

capacitytocopeintheirjourneytohealpast

trauma

Heart-CenteredApproach

Program

ParentandchildrelationshipChilddevelopment

assessmentsChilddevelopment

curriculumEarlyinterventionservices

Pre-kindergartenHomevisiting

CasemanagementSocialworkCounseling

Speech,occupational,andphysicaltherapy

NursingTransportation

Meals

Activities

CoalitionbuildingPubliceducationPoliticalaction

Advocacy

IntermediateBenchmarks

MutualtrustPrograminvestment

Participation Family

INTACTHealthy

attachmentNurturing&SAFEENVIRONMENT

Hassocialnetwork

FamilyresilienceSocialnetwork

Accesstoconcretesupports(food,housingemployment,childcare,

healthcare&education)

ParentPositiveparenting

skills

Socially&emotionallycompetent

Advocatesforself,childandfamily

Understandschilddevelopment

Outcomes

ChildOPTIMAL

DEVELOPMENTHealthy,resilient,&

socially&emotionallycompetent

Self-advocacyREADYFORSCHOOL

Policy

Statewidechildabusepreventionprogram

Publichealthapproachtoinfantandchildhoodhealth

Adequatesocialsupportforbasicfood,housing,

employment,healthcareandeducationneeds

Transparentandaccessibledata

Assumptions: Regardlessofbackgroundorhistoryeveryfamilyisworthyanddeservessupport,andthepreservationoffamilyserveschildrenandcommunity.Everyfamilyandculturehasstrengths,identifyingthemthroughearlysupportandcollaborativeeffortsmakesallthedifferenceandcreatesthegreatestchangeandimpact.Weallcanchange,everyindividualisworthwhileandeverychilddeservessupportwithoutlabelingorstigma,ourjusticecouldberestorativeandchildabusecanbeprevented.

Assets: Diversity,culture,climate,naturalbeauty,intergenerationalfamilies,creativeandprogressivecommunity,UNM,Medicaid.

External Factors: Poverty,intergenerationalviolence,inadequatefundingforsocialservices,isolation,inadequatetransportation,perceptionsofsingleparentfamilies,nocomprehensivestatewidechildabusepreventionplan,fragmentedhealthmentalhealthsystem,lackofaddictionservices,lackofaffordablehousing,oppressiontowardsmarginalizedgroupsbecauseofsystemicracism.Noaccesstostatedata.

Staffwithfamilies

Parentswithchildren

Allwithcommunity

ReflectiveParallelProcess

Perceptions

Publicawarenessabouttrauma,resilienceand

protectivefactorsandtheimportance

ofprevention

Growth

AwarenessofstrengthsFeelingprotected&

supportedIncreasedcapacityto

copeOwnershipforactions,

accountabilityReducedImpulsivity

Parent&childresilience

IdentifyingrealisticgoalsandstepstoreachthemConnectiontosupportsCreatingowncommunity

SafetyPlanningProvidingprotection&supportforchildrenFewertriggers,more

optionsImprovednurturingandsignsofattachment

Action

PB&JLOGICMODEL

StrategyStartearly

MultigenerationalParentingskills&

knowledgeConcretesupportReferraltoservices

Engagesocialsupportnetwork

Children grow to full potential in

nurturing families and a supportive

community

15of26

Another logic model

16of26

BEST PRACTICE: Collect both quan*ta*ve & qualita*ve data ArtWorksClassroomTeacherSurvey,administeredviaSurveyMonkey

ü Askdirectly(canbequan)fied)ü Includeopen-endedques)ons,whichelicitpar)cipantexperience(getusefulqualita)vedata)

EnlaceComunitario:knowledgequiz&par)cipantleSers

17of26

Direct ques*ons – easy to quan*fy

18of26

BEST PRACTICE: less is more for quan*ta*ve data

74% 70% 72%77%

84%

Enhancedabilitytoar)culatetheirfeelingsandopinions

Improvedvocabulary

Improvedlisteningskills

Increasedabilityandwillingness

toreflect

Greateruseofcolorful

languageandimagery

PercentofClassroomTeachersWhoReportedThatStudentsGained“ALot”or“AGreatDeal”ofLanguageArtsSkills

Principle#1:EvaluaEonshouldservetheinterestsofyourorganizaEon

19of26

BEST PRACTICE: more is less for quan*ta*ve data

PercentofClassroomTeachersWhoReportedThatStudentsGained“Limited,”“Some,”“ALot”or“AGreatDeal”ofLanguageArtsSkills

2% 3% 3% 2% 4%

24%27% 25%

21%

12%

30%

36% 35%

46% 44%44%

34%37%

31%

40%

Enhancedabilitytoar)culatefeelingsandopinions

Improvedvocabulary

Improvedlisteningskills

Increasedabilityandwillingnesstoreflect

Greateruseofcolorfullanguageandimagery

Limited Some ALot AGreatDeal

Thisistoomuchinfo!

20of26

Combining quan*ta*ve and qualita*ve ques*ons will make both more credible

Quan)fiable

Wasitreallyintegra)on?

21of26

BEST PRACTICE: iden*fy themes and provide examples for qualita*ve data

OneofArtWorks’goalsistobuildconfidence.TeachersoXenseethisinstudentswhoarequiet,shyorstruggling.Afewexamples:

ReachingQuiet,ShyandStrugglingStudents Grade Ihaveastudentwhoisatleast1gradelevelbehindclassinreadingandcanfeelfarbehindothers.ButaOerthepost-performanceworkshopwithGregory,hewasSOproudofhisartworkandsawhowhecouldkeepupwithhispeers.

1

OneofmystudentswhostruggleswithreadingwasabletomakewonderfulobservaEonsaboutthepainEngsandbackthemupwithreasonsandevidence,truly"reading"thepainEng.

3

IsabelaisashystudentwhoreallylovedtheicebreakeracEvityofthrowinganimaginarystonetoothers.Atfirstshedidn'twanttogetthestonethrowntoher,butwhenshedid,itmadeherfeeloktobeafoolonstage.Shesaidit"changedher".

6

22of26

Measuring what you care about can be tricky

Manyoutcomesaremuchhardertomeasure,andrarelydoyouhaveclassroomteacherswhocanreportonyourclients!Whatareyoualreadycollec)ng?

•  Assignments•  Screenings•  Needsassessments•  Intakeanddischargeforms•  Par)cipantsa)sfac)onforms

23of26

Useful comparisons

• Pre-Post• Comparisongroup(butbewareofsampleselec)on*)• Par)cipant’sownassessmentofchangeortransforma)on(evenmoreconvincingwhencoupledwithquan)ta)veoutcomes)

*Sampleselec)oniswhenpar)cipantsaredifferentfromnon-par)cipants.Par)cipantsmaybemoremo)vatedormoredesperatethannon-par)cipants.Inthefirstcase,youwilloverstateyourprogrameffects.Inthesecondcase,youwillunderstateyourprogrameffects

24of26

PB&J families compared at intake and discharge

25of26

BEST PRACTICE: collect representa*ve data

• Doyouhavedataforalloralmostallpar)cipants?-  80%ormoreisgood

• Doyouhavedatafromarandomsample?-  Youdon’tneedallpar)cipantsifyouarecollectarepresenta)vesample.Youcouldsurveyallpar)cipantstwoweeksperyear.

26of26

Great evalua*on at Enlace Comunitario