PROVING ‘EFFECTIVENESS’ IN RESETTLEMENT...3 | Proving 'Effectiveness' in Resettlement Glossary • INPUTS – The resources invested, staff, time, buildings etc • OUTPUTS –

1 | Proving 'Effectiveness' in Resettlement

PROVING ‘EFFECTIVENESS’ IN RESETTLEMENT


Contents

1 Introduction 3

2 Glossary 4

3 Abbreviations 4

4 What counts as evidence of effectiveness? 5

5 Measuring service effectiveness 8

6 Why prove effectiveness 9

7 Assessing effectiveness - key approaches 10

Evaluation 10

Theory of change 11

Outcome frameworks 12

Thecost-benefitapproach14

Complimentarymethods15

8 The challenges in measuring effectiveness 15

9 What has been learnt from Phase 1 activities? 16

10 Conclusion 18


Glossary• INPUTS–Theresourcesinvested,staff,time,buildingsetc

• OUTPUTS–Products,activitiesetc,asaresultoftheinputs

• OUTCOMES–Changesthatoccurduringaperson’sengagementinaprogrammeorreceiptofanintervention

- PROXIMALandDISTAL outcomes – These terms have emergedfromthemedicalcommunitywith‘proximal’referringtoshort-termconsequencesand‘distal’tolong-term consequences

- INTERIM or INTERMEDIATEoutcomes–thoseachievedonthejourneytofullresettlemente.g.improvedfamilyrelationships,reduceduseofsubstances,engagementinactivities/ETE,maintainingstableaccommodation.

• IMPACTS – Longer-term changes which are a result of the intervention

• INDICATOR – A measurable outcome i.e. how long a tenancyhasbeenmaintained

• STAKEHOLDER-Anygrouporindividualwhocanaffectorisaffectedbytheintentionsorachievementoftheorganization'sobjectives'(Freeman1984).

NOTE: The terms ‘impact’ and ‘outcome’ are often used interchangeably. Both refer to the consequences or after-effects of the inputs, however outcomes are normally seen to be the more immediate consequences whilst impact has a longer-term dimension.

Abbreviations• BYC BeyondYouthCustody• CQC Care Quality Commission• CRC CommunityRehabilitationCompany• EBP Evidence-basedpractice• FTE First time entrants• HMIP HerMajesty'sInspectorateofPrisons• IOMI IntermediateOffenderMeasurementInstrument• MoJ Ministry of Justice• NAYJ NationalAssociationofYouthJustice• NOMS NationalOffenderManagementService• OBPM Outcomes-basedperformancemanagement• PYD Positiveyouthdevelopment• RTC RandomisedControlTrial• ToC Theory of Change


IntroductionBeyondYouthCustody(BYC)haspublishedanumberofresearchreportsandpractitioner’sguidesthatdrawuponthebestevidenceavailabletodescribe,supportandencouragegoodpracticeinresettlementservicesforyoungpeopleinEnglandandWales.Anumberofconsistentmessagesderivedfromthisworkoverthelastthreeyearsledtothefollowingdescriptionofwhatconstitutes‘effective’resettlementforyoungpeople:

Effective resettlement is a process that enables a shift in a young person’s identity, moving them away from crime towards a positive future. (p.8)

Inordertoachievethis,threeunderpinningandwidelyacceptedprincipleswereidentifiedasessentialinBYC’s‘Effectiveresettlementofyoungpeople’report(Goodfellowetal,2015):

1. Thecoordinationofservices

2. Engagingtheyoungpersonforpositivechange

3. Continuousservicefocussedonresettlement

AsBYCentersitsfinalphaseitisimportanttocapturehoworganisationsprovide‘effective’resettlementservicesand,moreimportantly,describetheirsuccess.Itishopedthatbyexploringtheseissuesaclearerpicturewillemergewhichwillsupport,extendandsharegoodpractice,generatepractice-basedevidenceofimpact,andhelpthedevelopmentofcommissioningframeworks;allofwhich,itishoped,willsupporttheyoungperson’sshiftintheiridentityandawayfrom crime.

Thisreportaimstoexplorethedifferentways‘effectiveness’ismeasuredamongstakeholdersinvolvedintheresettlementofyoungpeople.Specifically,howthese‘indicators’or‘outcomes’shapetheprioritiesforresettlementservicesandprovideevidenceofthevalueandimpactoftheworkthattheyundertakeforcommissionersandpolicymakers,aswellassocietymoregenerally.

ItisdrawnfromthecombinedknowledgeoftheBYCteam,whohavespentmanyyearsresearchingandwritingaboutissuesassociatedwitheffectiveoutcomesgeneratedbyinterventionswithoffenders.Additionally,toaugmentourunderstandingandfocusmorespecificallyupontheissuesofmeasuringeffectiveness,anumberoffact-findingactivitieswereundertakenintheautumnof2015.

Theseactivitieswere:

1. Abriefliteraturereviewthatconsidersthevariouswaysinwhichsocialimpactisbeingmeasuredacrosssocialpolicymorewidelyand,inparticular,theyouthjusticesector’sresponsetoevidencingtheirsuccessinservicedelivery.

2. AworkshopwithpractitionersattheNationalAssociationofYouthJustice(NAYJ)conference(October2015),co-facilitatedwithtwoyoungmenfromSafeHands,Liverpoolwhichconsidereda)thedifficultiesinmeasuringsuccess,b)whatsuccesslookslike/whatfactorsleadtosuccessful resettlement.

3. AnenquiryviaYouthinFocus(YiF)*projectstoexplorethedifferentwayssuccessfulresettlementisdescribedandreported.

*YouthinFocus(YIF)isaBigLotteryFundedprogramme,whichaimstosupportvulnerableyoungpeoplethroughchangesintheirlives.

AlongsideBYC,YIFfunded15servicedeliveryprojectsacrossthecountrytoworkwithyoungpeopleleavingcustody.


What counts as evidence of effectiveness?Thegovernment’semphasison‘bestevidence’of‘whatworks’hasbecomeembeddedacrossvariousprofessionsinrecentyears,andthecriminaljusticesystemisnoexception.The‘evidence-based’movementgatheredsignificantmomentuminthehealthcareprofessionintheearly1990s;asadisciplineofmedicine,evidence-basedpractice(EBP)wascoinedasatermbyagroupatMcMasterUniversityinCanadain1992andquicklyacquiredinternationalrecognition,becomingthenewmedicalorthodoxy(McIntosh2010).However,theoriginsofthe‘whatworks’paradigminsocialpolicyismoregenerallytracedbacktotheworkofDonaldCampbellinthe1960sand1970sandinparticularhisseminalpaper,TheSocialScientistasMethodologicalServantofTheExperimentingSociety(1973).

Thepredominantviewwithinthe‘whatworks’paradigmisthatthebestqualityevidenceofeffectivenessisgeneratedbyundertakingrandomisedcontrolledtrials(RCTs),whichareregardedasbeingthe‘goldstandard’notonlyformedicalresearch,butforsocialresearchaswell.Indeed,theso-calledMarylandScale,usuallyusedinmeta-analysesandsystematicresearchreviews,ranksevaluationresearchstudiesbytheextenttowhichtheyapproximateafullRCT.However,whileRCTsdohavetheadvantageofbeingabletodemonstratethatparticularchangesarenotsimplyduetochance,theyhavealsobeencriticisedfornotgivingsufficientattentiontocomplexcontextualfactors,andthereisacontinuingdebateaboutwhetheraprioritisationofRCTsreallycanadvanceunderstandingofcomplicatedsocialandindividualchange(McIntosh2010).

Cartwright(2007:4),forexample,concludes:

“… to draw causal inferences about a target population, which method is best depends case-by-case on what background knowledge we have or come to obtain. There is no gold standard.”

Althoughgainingmomentum,theuseofRCTsinsocialresearchishighlyproblematic;particularlyasdebatesaboutwhatconstitutesgoodqualityevidenceaboutimpactsandeffectivenessareon-going.Theattractionofrandomisationisthatitappearstobeasimplemethodofdeducingreplicableconclusionsbut,forGoldstein(2002),itleadstoerroneousconclusionsthatdonottaketheunderlyingcomplexityofmodellingintoaccount.

Intheyouthjusticesystemasawhole,thereiscleardirectiononwhatcountsassuccessandhowthisevidencecanbecaptured.Section37ofTheCrimeandDisorderAct(1998)madeclearthatthestatutoryaimoftheyouthjusticesystemistopreventoffending(andreoffending)bychildrenandyoungpeople.Additionally,thesystemisjudgedupontheprioritiesof(i)reducingfirsttimeentrants(FTEs)intothesystem,establishedin2008and(ii)reducingthenumberofchildrenincustody,introducedin2010.Both(i)and(ii)haveachievedaconsiderablelevelofsuccessbut,whileoffendinghasreduced,reoffendingratesremaindisappointing.

Whiletheever-reducingnumberofbothfirsttimeentrantsandchildrenandyoungpeopleincustodyiswarmlywelcomedacrossthesector,areductioninreoffendingratesremainsatoughernuttocrack.MinistryofJustice(MoJ)statisticsconfirmthat,intheyearending2013,37.4%ofchildrenwhoreceivedasubstantivedisposalreoffendedwithin12months,anincreasefrom33.4%in2002.1 Mostsignificantly,morethantwo-thirdsofchildrenreoffendwithin12monthsofreleasefromsecureinstitutions.Reoffendingratesarealsosubstantiallyhigheramongstyoungadultsinthecriminaljusticesystemthanolderadultoffenders(BatemanandHazel,2013).Thisviewissupportedbythelatestfiguresbelow(Bateman2015).

1MinistryofJustice(2015)NationalAnalysisofReoffendingData,forthoseaged10-17YouthJusticeBoard


Table 1: Proven rates of reoffending by type of disposal - 12 months ending September 2008 and 2013

Disposal Percentage reoffending within 12 months

Year to Sept 2008 Year to Sept 2013

Pre-court disposal 24.8% 28.3%

First tier sentence 45.5% 44%

Community sentence/YRO 65.9% 64.5%

Custody 72.8% 66.5%

All 32.9% 37.4%

Thetwotargets–reducingFTEsandfallingratesofrecidivism–areaccordinglyintension.ThemoresuccessfulanindividualYouthOffendingTeam(YOT)isinreducingFTEs,themoredifficultitwillbeforthatservicetodemonstratefallingratesofrecidivism,andviceversa.AccordingtoBateman(2015),itisalmostcertainlythistensionthataccountsfortheslightriseintheoverallrateofyouthreoffending. Recidivismisaneasyandcost-effectivemeasurewhichresonateswiththosemembersofthepublicandpolicymakersconcernedwithcrimerates.Althoughsuchapproachesfrequentlyreflectadeficit-basedmodel,thevalueofincludingmeasuresofreoffendingasanoutcomeremainspertinentwhentryingtounderstandmoreabouttheeffectivenessofinterventionsmade.Itmayalsoofferaninsightintowhetherthosewholeavecustodygoontoleadcrimefreelives,butitdoesnotinvestigatewhethertheselivesareindeedproductiveordemonstratepro-socialbehaviours(PetersandMyrick,2011;McNielletal,2012).

Dataonrecidivismisonlycollectedoveraone-yearperiod.Thisisproblematicinitself,asBYCshowsthatrelationshipsoftrust,andlengthyandconsistentengagement,areoftenrequiredtoproduceandprovesuccessfuloutcomes,particularlywithyoungpeoplethathavemultipleneeds.However,short-terminterventionscanbeextremelyeffectiveandshouldnotbeexcluded.Thisdataalsocannotaccountforthemanyyoungpeopletransitioningintoadulthood,movingonwiththeirlivesandtherebynotremaininginregularcontactwithprojects.Fortunately,recognitionofthevalueofalonger-termperspectivewhenconsideringtheeffectivenessofresettlementservicesappearstobeemerging-forexample,seeNOMSCommissioningIntentionsfrom2014.2

Professionalsintheyouthjusticesystemconsiderthebinaryfocusonshort-termreoffendingratestobeanunhelpfulindicator.Indeed,toviewreoffendingmeasuresasjustoneaspectoffarmorecomplexliveswhichneedtobesupportedisdeemedmorehelpful.Forspecificexamplesofthiscomplexity,seeBYC’sreport‘Resettlementworkwithyoungpeople:Usingindividualcasestudiestoassesscostsandbenefits’.Additionally,thereportbyClinks(2013)‘Whatdoesgoodrehabilitationlooklike?’citedthefollowingexample,extractedfromtheMoJ’sOffenderManagementCommunityCohortStudy(OMCCS)toillustratethe‘highlycomplexanddauntingsetofdisadvantages’and‘directandindirectdiscrimination’:

‘Sixty-five to seventy percent of young people in youth custody have experienced a traumatic brain injury; 51% have come from unsuitable accommodation; 43-57% have dyslexia; 44% of young women and 30% of young men have been in care at some point; 34% are from a Black, Asian or minority ethnic (BAME) group; 23-32% have a learning disability.’

(Frazeretal,2013:4)

2https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/280922/commissioning-intentions-2014.pdf


Formanyworkingwithinthesystem,thereisincreasingpressuretofocusenergyonmeetingthetargetsmentionedonpage3;thecompellingevidenceofthenegativeconsequencesofayoungperson’scontactwiththecriminaljusticesystemmeansthatformanypractitionerstherewillbeaninclinationtoprioritisetheFTEmeasureoverrecidivism.

Consequently,reoffendingcannotaccordinglybeconsideredareliableindicatorofYOTperformance.Indeed,ithasbeensuggestedthatduetothemoreentrenchedpatternsofoffendingamongstthoseincustodynowcomparedtotheircounterpartspriorto2008,ariseinreoffendingratesisaninevitableoutcome(Bateman2015).

Also,suchafocusonreoffendingdoesnotallowforthe‘zig-zag’processofdesistance.Increasinglydesistancetheoriststalkofa‘primary’and‘secondary’desistance.‘Primary’beingwherethereisagapinreoffendingand‘secondary’beingmoreakintoashiftinidentityandfullerintegrationbackintosociety.Forsecondarydesistancetobeachieved,ahighlyindividualisedresponseisrequiredandanumberoffactorsneedtobepresent,whichinclude:

• Anacknowledgementoftheindividual’smaturationasopposedtochronologicalage

• Positivesupportiverelationships

• Developmentofapro-socialidentity

• Maintenanceofhopeandmotivation

• Strength-basedapproaches–bothpersonalandsocialcapital

• Celebrationandrecognitionofpositivechange

(McNielletal,2012)

TheYouthJusticeBoard’s(YJB)ReducingReoffendingProgramme3isnowinitsthirdyear.Itwasestablishedto:

1. HelpdriveandsupportYOTeffortstoreducelocalreoffendingrates

2. Developabetterunderstandingofthenatureofreoffendingandthedriversbehindit

Seventy-fiveYOTpartnershipsnowusetheYJBtoolkit.Preliminaryfindingssuggestthatthedriversbehindprogressinreoffendingperformancehingeonthefollowingcharacteristics:

• HighlymotivatedandengagedstaffgroupswhoareawareoflocalreoffendingperformancedataandacceptthatthepreventionofoffendingandreoffendingisthekeybusinessoftheYOT

• Thedeploymentofhighqualityassessment,planningandinterventionprocesses.Inparticular,thequalityoftherelationshipbetweenpractitionerandyoungpersonisdeemedcrucial,aswellastheprogrammesofinterventionbeingindividualisedandculturallyanddemographicallysensitive

• Theexistenceofwiderpartnershipsupportandtheavailabilityofresourceswhichenableaccesstoarangeofservices,ratherthanmaintainingYOTstandaloneprojectse.g.parentingsupport.Furtherintegrationwithchildren’sservicesalsoallowsfortheopportunitytooffer‘step-down’supporttothosewhohadcometotheendoftheirformalYOTsupervisionbutremainedlikelytoreoffend

• Sufficientresourcestomeetminimumstandards,ascaseloadshadreducedsubstantiallyatafasterratethanthereductioninfunding.Thismeantthatpractitionershadtheopportunitytoaddressanindividual’scomplexneed.ThiswascoupledwithsophisticatedinformationsystemsinthehigherperformingYOTswhichenabledanidentificationoftrendsandthetimelydeploymentofresourcestoaddressneed

MOJ/YJB(2015)

3https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/479708/Reoffending_Furthering_Our_Understanding.pdf


Thesecharacteristicsarereflectiveoftheunderpinningprinciplesforeffectiveresettlement,identifiedbyBYCatthestartofthisreport.AsBYChavediscovered,inordertoensureresettlementis‘successful’,anumberofconditionsneedtobeinplacewhichcharacteriseandfacilitatethesmoothtransitionfromcustodytothecommunity.Atanoperationalleveltheseinclude:

1. Jointsentenceplanningthatinvolvesbothcustodialstaffaswellassupervisorystaffinthecommunity.Thisshouldstartatthepointofsentencing

2. Theestablishment,whileincustody,ofconsistentandtrustingrelationshipsbetweentheyoungpersonandtheprofessionalstaskedtosupporthim/her,bothincustodyanduponrelease

3. Theappropriateinvolvementoftheyoungperson’sfamily/supportnetworkinthesentenceplan

4. A‘seamless’transitionfromcustodyinvolvingappropriateinformationsharingbetweenservices,throughthegatesupportandaccesstoservicesuponimmediaterelease,inordertoavoidsomeofthemorestressfulaspectsofthetransitionprocess

Thereareotherstrategicobstacleswhichconspireagainsttheseoperationalconditionsbeinginplace–forexample,theplacingofyoungpeopleatsignificantdistancefromthecommunitytowhichtheywillreturn,andRoTLnotbeingusedsufficientlytoassistinthepreparationfortheyoungperson’srelease,amongstothers.

Thesechallengesarealsoreflectedinthe‘JointthematicinspectionofresettlementservicestochildrenbyYouthOffendingTeamsandpartneragencies’,conductedbyHMIP(HerMajesty’sInspectorateofPrisons),CQC(CareQualityCommission)andOfsted,(March2015)whichoffersagloomypictureofthecurrentstateofresettlementforyoungpeople.Itdoes,however,provideusefulinsights,particularlyontheoperationalandstrategicbarriersfacedbyresettlementproviders.Thereportidentifiestheneedforgreaterfocusonthespecificneedsandoutcomesofchildrenleavingcustodyandmorescrutinyofdataandimpactevaluation.Itconcludeswiththefollowingstatement:

“It is disappointing that after a decade of initiatives, resettlement work shows little improvement in outcomes for many children despite the contributory factors being well known.”

(2015:36)

Measuring service effectivenessHowdoresettlementservicesmeasuretheireffectiveness?Isitpossibletomeasurea‘shiftinidentity’and,ifso,howdoservicesdescribetheirsuccess?Formany,itappearsthatthereisadisconnectbetweentheoutcomeswhichareimportanttoarangeofstakeholdersandwhatisactuallymeasured.Whilemostresettlementorganisationsagreethattheycontributetoareductioninreoffending,itisoftennottheprimarymeasureofsuccessandstakeholderswillhavedifferingpriorities.

“There are so many different indicators of success and different ways of measuring success.”

ParticipantatNAYJworkshop2015

Thedifficultyforyouthjusticeservingagencieshasbeendefiningandmeasuringsuchsuccessinwaysthatspeaktotheyoungpersonaswellascommissioners,policymakersandthewiderpublic,andthatarealsocognisantofbothnationallyimposedtargetsandtheirpossibletensionswithmorelocallydeterminedorcompetingpriorities.

Formanycommissioners,serviceeffectivenessisoftenseenassuccessacrossanumberofhardormeasurableindicators,forexampleinsecuringemploymentormaintainingatenancy.Furtherexplorationintowhysomeonenowhastheskillsandmotivationtosustainthetenancy–asaresultofthereductioninnegativefactorswhichhavepreviouslycreatedchaoticlifestyles(forexample,druguse)–israrelyrecorded.


Forfamilies,‘success’isoftenmeasuredbyyoungpersonstayingoutoftroubleandnotreturningtocustody.Youngpeoplethemselveswillofcoursehavetheirownideas,althoughgettingthemheardcansometimesprovechallenging.ThiswasaviewechoedbythepractitionersattheNAYJworkshop:

“People sometimes forget there’s a young person in the middle – no one’s interested in our young people.”

“What do young people say/want/need – are they involved in defining their success?”

(July2015)

Whenaskedhowtheywoulddescribesuccessfulresettlement,practitionersresponded:

“Building self-esteem and inner belief/ self-reflection.”

“Achieve goals (whatever they may be, different for each individual).”

“Engagement is success – sometimes a young person just ‘showing up’ can be seen as success.”

“Securing suitable accommodation, better mental health, ETE.”

AlloftheabovearelegitimateandreflecttheYJBpathways(YJBOctober2014)althoughtheyalsoindicatethelackofconsensusintermsofprioritisinginterventionsinordertoachieve‘effectiveness’.Thekeyquestionthisraisesis,doesthislackofconsensusmattermovingforward?

Why prove effectiveness? Therearemultiplereasonswhyidentifyingeffectivenessinresettlementpracticemakessense.Inthecurrenteconomicclimate,reductionsintherelianceonthepublicpursearegatheringpace,andinterventionswhichcanreducetheharmfulimpactsofcrime,bothsociallyandeconomically,needtobeprioritised.Whereeffectivenesshasbeenprovenlocallytherecanalsobeextremelypositiveimpactsonsentencingdecisionswhichcanonlyfurtherbenefitthoseyoungpeoplewithinthesystem.

InasurveybytheNationalAuditOfficein2010,three-quartersofYOTmanagerssaidthat,whenidentifyingwhatworksinreducingreoffendingratesamongstyoungpeople,evidenceisthinontheground.Additionally,accordingtoNewPhilanthropyCapital(2011),charitiesoftenrelyonanecdotalevidencewhenmakingclaimsabouttheirsuccessratherthanmorerobustmeasuresofimpact.Aftermappingexpenditureacrosschildren’sservicesinmorethan20localauthoritiesacrosstheUKin2013,theDartingtonSocialResearchUnitestimatesthattypicallylessthan1%ofatotalbudget–includingeducation–isspentonservicesthatareunderpinnedbyrobustevidenceofimpact.

Increasingly,socialimpactmeasuresarebeingrequiredbycommissionersinordertoevidencevaluefortheirmoney.Practitionersarealsokeentoevaluatetheirinterventionsinordertofurtherunderstandthechangestheywanttomake,andtoimprovetheinitiativebyre-examiningmethodsandactivitiesused.Thereareanumberofreasonswhythisishelpful:

1. Suchevidencewilladdtothebodyofknowledgeintheparticularfield,promotegoodpracticeandstrengthentheevidencebase

2. Projectsaremorelikelytosucceediftheyuseexistingevidenceofcreatingchangeandareclearabout the outcomes they are trying to achieve

3. Demonstratingimpacttocommissionersislikelytoincreaseboththechancesofinvestmentandtheirunderstandingofbothproximalanddistaloutcomes

Organisationsthatareunabletoprovideevidenceoftheirimpactareincreasinglyvulnerableinthecurrentclimatewherepaymentbyresultsisbecomingembedded.Forsmallerorganisations,resourcingmorerigorouswaystocaptureevidenceofimpactratherthangatheringcasestudiesremains challenging.


WithinTheCodeofGoodImpactPractice4producedbytheInspiringImpactconsortiumin2014,alearningcycleof‘impactpractice’isidentifiedwhichinvolvesthefollowingfivesteps:

1. Identifyexpectedoutcomes,developaTheoryofChangeandselectpriorityoutcomes

2. Designtheinterventionbaseduponevidenceofwhatworks

3. Targetmeasurementtofocusonpriorityoutcomes

4. Chooseanevaluationmethodologywhichispractical,proportionateandmeetsanachievablestandardofevidence

5. Selecttoolsanddatawhichsupportthemeasurementofthisdata

Borrowedlanguagefromacrossdifferentfieldshasbeenimportedandisnowassociatedwiththeidentificationofeffectivenessinsocialintervention.Forexample,withinphilanthropy,theconceptsofsocialreturnoninvestment(SROI),modelling,andTheoryofChangehavebeenappliedtothesectorwithouttheoreticalorpracticalknowledgetoassistintheirapplication(Proscio,2000).Accompanyingthedriveforevidence-basedpolicyandpracticewithincriminaljusticehasbeenanincreaseinPaymentbyResults(PBR),socialimpactbondsandevidence-oroutcomes-basedfunding/commissioning.

Assessing effectiveness – key approachesThegovernment’scontributiontothegenerationofevidence-basedpolicyinpublicserviceshasbeenviatheestablishmentofsevenindependentWhatWorksCentres.Togetherthesecentrescoverpolicyareaswhichreceivepublicspendingofmorethan£200billion.WhatWorksCentresenablepolicymakers,commissionersandpractitionerstomakeevidence-baseddecisionsandtoprovidecost-efficient,usefulservices.Thedifferentcentresprovidearangeofresources,briefingsandtoolkitsforuseinparticularsectors.TheWhatWorksCentreforcrimereductionisrunbytheCollegeofPolicingandcanbefoundathttp://whatworks.college.police.uk.

EvaluationTraditionally,evaluationinappliedfieldsismosteffectivewheretheprocessisinteractive,withprofessionalsandotherstakeholdersrespondingtotheemergingfindingsastheinterventionunfolds.Indeed,ithasbeenarguedthatthisformofevaluationholdsthekeytothedevelopmentofeffectivepractice.Thisapproach,describedbyPawson&Tilley(1994)asRealisticEvaluation,isrootedtheParticipatoryActionResearch,pioneeredbyKurtLewininthe1940sand1950s,whichhedescribesas‘asystemofprogressiveproblem-solving’.InhisseminalpaperActionResearchandMinorityProblems(1946)Lewindescribedtheprocessas:

“...research on the conditions and effects of various forms of social action that uses a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the results of the action.”

UtilisingLewin’scircleofplanning,action,andfact-findingenablesthoseresponsiblefortheinterventiontoidentifythekeymechanismsthatsustaintheproblemsbeingaddressed.AsPawsonandTilley(1997)suggest,byidentifyingandanalysingthesemechanisms,andthecontextualvariableswhichshapetheirimpact,itispossibletodevisealternativemechanismsdesignedtodisableorcircumventthecircumstancesresponsiblefortheoriginalproblem(s).

Accordingto‘EffectiveInterventioninGangAffectedNeighbourhoods(Pittsetal,2015),thismodelofresearchhasseveraladvantagesovermoreconventionalforms:

1. Itpromotessystematicorganisationalfeedback,essentialtoeffectiveprojectdevelopment

2. Itpromotesdialoguewithinandbetweenorganisationsandwiththeintendedbeneficiariesofaninitiativeoroperation

4http://inspiringimpact.org/listing/the-code-of-good-impact-practice/


3. Itactsasaspurtoreflexivitywithinandbetweenorganisations

4. Ithelpstobuildcapacitywithinandbetweenorganisationsandtheirstakeholders

5. Itassistsindevelopingtheskillsandknowledgeoftheworkforceandotherstakeholders

6. Itenablesacomparisonoftheeffectsofsimilarinterventionsindifferentcontexts,establishingwhatworks,forwhom,inwhatcircumstancesandwhy?

Themainobjectiveofthisprocessistoenabletheevaluators,incollaborationwiththeotherstakeholders,tobuild,refineandoperationaliseaTheoryofChangebecause,asLewinobserved,“Thereisnothingaspracticalasagoodtheory”.RealisticEvaluationisaprocessoftheorytestingandtheory(re-)building.Itproceedsfromtheassumptionthateverysocialinitiativeisatheorywaitingtobetested.Itendeavourstomaketheimplicittheoriesinformingprogrammedevelopmentexplicit,bydevelopingclearhypothesesabouthow,andforwhom,programsmight‘work’.Theimplementationoftheprogram,andtheevaluationofit,thenteststhosehypotheses.InordertodothisitinterrogatesthethreekeycomponentsofasocialprogrammeContext(C),Mechanism(M)andOutcome(O).

Theory of ChangeTheoryofChange(ToC)or‘logicmodel’isanaspectofprogrammetheory–along-standingareaofevaluationthought–developedfrom1960sonwards.ItisnowregularlymandatedasarequirementoffundingapplicationsforfundersliketheBigLotteryandtheCabinetOffice.Programmetheoryapproachesofferamoreexplicitfocusonthetheoreticalunderpinningsofprogrammes,clearerarticulationofhowprogrammeplannersviewthelinkagesbetweeninputsandoutcomes,andhowprogrammesareintendedtowork,toimproveevaluationsandprogrammeperformance(FunnellandRogers2011).

‘People are seeing theory of change as new, but it is just about good programme design, good adaptive management and understanding where you fit into the grander scheme of things.’

LydiaGaskell,WWF-UK

AToCdefinesallbuildingblocksrequiredtobringaboutagivenlong-termgoal.Thissetofconnectedbuildingblocks–interchangeablyreferredtoasoutcomes,results,accomplishments,orpreconditionsisdepictedonamapknownasapathwayofchange/changeframework,whichisagraphicrepresentationofthechangeprocess.Eachoutcomeinthepathwayofchangeistiedtoanintervention,revealingtheoftencomplexwebofactivitythatisrequiredtobringaboutchange.Likeanygoodplanningandevaluationmethodforsocialchange,itrequiresparticipantstobeclearonlong-termgoals,identifymeasurableindicatorsofsuccess,andformulateactionstoachievegoals(Stuartetal2009).

Amorenuancedlearning-baseddefinitionisofferedbyComicRelief:

TheoryofChangeisanongoingprocessofreflectiontoexplorechangeandhowithappens-andwhatthatmeansforthepartweplayinaparticularcontext,sectorand/orgroupofpeople.

• Itlocatesaprogrammeorprojectwithinawideranalysisofhowchangecomesabout

• Itdrawsonexternallearningaboutdevelopment

• Itarticulatesourunderstandingofchange-butalsochallengesustoexploreitfurther

• Itacknowledgesthecomplexityofchange:thewidersystemsandactorsthatinfluenceit

• Itisoftenpresentedindiagrammaticformwithanaccompanyingnarrativesummary.

(James2011:2)

DevelopingaToCallowsforthearticulationofhowaninterventionwilldeliverimpact.ToCrequirespractitionerstomapwhattheywanttoachieve,andwhattheyhavegotcurrently.Eachserviceorinterventionismappedintothegapinbetweentocreatethetheorythatunderpinsthechangesthat


theyseektomake.ToCisbeingusedwidelyasanexplanationofthecausallinksthattieprogrammeinputstoprogrammeoutputs.

“It helps circumnavigate the philosophical debates on methodology, especially in evaluation research, by recognising that the most important judges of validity are the stakeholders who are going to use the results.”

(Nichols&Crow2004).

AToCwouldnotbecompletewithoutanarticulationoftheassumptionsthatstakeholdersusetoexplainthechangeprocessrepresentedbythechangeframework.Assumptionsexplainboththeconnectionsbetweenearlyproximalandlong-termdistaloutcomesandtheexpectationsabouthowandwhyproposedinterventionswillbringthemabout.Often,assumptionsaresupportedbyresearch,strengtheningthecasetobemadeabouttheplausibilityoftheoryandthelikelihoodthatstatedgoalswillbeaccomplished.RickDaviesdefinesaToCsimplyas,‘thedescriptionofasequenceofeventsthatisexpectedtoleadtoaparticulardesiredoutcome’.5

PractitionersapproachToCthinkingfromdifferentstartingpointsandfordifferentpurposesthroughouttheprojectcycle;somefromatechnicalperspectiveasatoolandmethodologytomapoutthelogicalsequenceofaninitiative,fromactivitiesthroughtothechangesitseekstomake.Othersseeitasadeeperreflectiveprocess–amappingandadialogue-basedanalysisofvalues,worldviewsandphilosophiesofchangethatmakemoreexplicittheunderlyingassumptionsofhowandwhychangemighthappenasanoutcomeofaparticularintervention.6

InevitablythereisanemergingcritiqueofthegrowinginterestinanddemandforToC.Thefirstcriticismisthatitisoftenpoorlydefined,meaningthatitishardtoascertainitsquality.Inaddition,theterm‘theory’iscontentious.Othertermslike‘changepathway’or‘practicemap’mayresonatebetter.Onestrengthofthetoolisitsabilitytocapturecomplexity,butthismayalsobeaweakness,aslargeelaborateexamplescanbediscouragingfornewcomerstoToC,andcanlooklikerigidplans,whichareoverwhelminglycomplex(seeWeiss,1997).

Additionally,andimportantlyforstaffwhoarealreadyundersignificantpressureintheirday-todayworkenvironment,mandatingaToCmayalsoturnitfromaparticipatorypracticetoolwithimmensepotentialintoanunwelcomebureaucraticimposition(Mayne,2008,Hughes&Traynor,2000).Manymaybeworkingfromtheirown‘implicit’ToCbaseduponyearsofexperienceandanunderstandingofthedifferencetheirinterventionwillachieve.Suchpractitionersalsohavetheabilitytoidentifytheempiricalbasisuponwhichtheirinterventionisdesigned,andsubsequentlyarticulateitinfundingapplications(Reusga2011).Inhisarticle‘Philanthropy’sAlbatross:DebunkingTheoriesofChange’,Reusgaassertsthefollowing:

“Requiring grantees to produce explicit theories of change – beyond what they usually include in their grant proposals – does little to improve the art or science of grant-making. Highly elaborated theories of change are generally urged upon grantees by well-meaning people who have a limited understanding of how they function in the social sciences. Because theories of change are generally shrouded in the impenetrable verbiage of philanthropy, it’s also not surprising that most of us have little inkling of their theoretical and practical limits.”

(2011:3)

Asaresult,practitionersneedtoconsiderboththestrengthsandlimitationsoftheirownToCs–andtheprocessesusedtogeneratethem.WorkundertakenbyproviderstoarticulateToCscanbeofgreatvaluetoorganisationsandstaffteamsand,asnotedabove,theyarealsooftenrequiredbykeyfunders.However,theyshouldprobablynotberegardedasendproductswhichshouldsomehowbeleftaloneonceestablished.Causalprocessescanbeenormouslycomplicated,andtherearethereforegoodreasonsforregardingToCsthemselvesasorganicandsubjecttochangeorrevision.

5RickDavies,April2012:Blogpostonthecriteriaforassessingtheevaluabilityofatheoryofchangehttp://mandenews.blogspot.co.uk/2012/04/criteria-for-assessing-evaluablity-of.html

6Arangeoftoolsareavailabletohelpprofessionalsdevelopatheoryofchangeathttps://www.youthimpact.uk/resources-hub/impact-journey/developing-theory-change


Outcomes FrameworksOutcomes-basedperformancemanagement(OBPM)hascometodominatethinkingacrosssocialpolicyinthelastdecade(Perrin2006),themostrecentincarnationbeingpaymentbyresults(PBR).Itpresupposesthatanyinterventioncanbemeasuredforitsimpactinarecipient’slifebyusingarangeofoutcomesagainstwhichtomakesuchanassessment.Unsurprisingly,formanycommissioners,thedominanceofusingreoffendingratesandotherdeficit-orientatedindicatorsoroutcomesisthattheyareeasiertomeasureandthereisgreaterconsensusaboutwhatyoungpeopleshouldnotdoratherthanwhattheyshould.Asmentionedearlier,theoriesofchangeallowforanarticulationofexpectedoutcomeswhichtakesplacebyengagingwithstaff,beneficiariesoftheservice,andotherstakeholderstoidentifyarangeofexpectedoutcomesoveralongertimeframe.

Apreferredlensthroughwhichtoconsiderwhatimpactserviceshaveisbyembracingpositiveyouthoutcomeswhichrefertoindicatorsofprotectivefactors,skillsorstrengths.Itisbaseduponthepositiveyouthdevelopment(PYD)philosophyandisunderpinnedbythebeliefthateventhoseyoungpeoplewiththemostcomplexneedscanachievesuchdevelopmentifacoordinatedapproach,whichplacestheyoungpersonatthecentre,isapparent.

PYDisacomprehensiveapproachtothinkingaboutyoungpeople’sneedsastheymaketheirtransitiontoadulthood(Buttsetal2010).Thisapproachisnotnew–someofitcanbetracedbacktotheworkof19thcenturyjuvenilecourtadvocatesintheUS,suchasJaneAddams.However,itismorefrequentlyassociatedwiththeworkofKennethPolkwhodevelopedasetof‘rules’forcreatingprogrammesforyoungpeople(Polk&Kobrin,1972).Buttsetal(2010)adoptedthisapproachintheircreationofaPositiveYouthJusticeframeworkintheUS.Elementsoftheserulesareapparentacrossarangeofsettingswithyoungpeople–itisbynomeanslimitedtocriminaljusticeenvironments.ThisapproachhasbeenfurtherrefinedwithinyouthjusticeintheUKthroughthe‘PositiveYouthJustice;childrenfirst,offenderssecond’frameworkdevelopedbyHaines&Case(2015).

AdoptingaPYDapproachcouldbegintheusefulconversationabouttheexpectationswehaveofyoungpeopleandwhatsupportneedstobeinplacefromfamilies,schoolsandcommunitiestoachievethem(Schulman&Davies,2007).Withsuchanoutcomesframeworkinplace,organisationscanbegintoreclaimtheirpracticeandincreaseconsistency,internallyand,moreimportantly,externallybetweenorganisations.Havingacoordinatedsetofoutcomesandlanguagewouldallowforamorenuancedapproachfromcommissionersandaclearerideaofwhodoeswhat,whyandwhatevidenceisusedtoprovesuccess.

Thiswillbeofparticularinteresttosmallerorganisationsthatneedtoensureaproportionateapproachto‘impactpractice’.Thiswouldrequiretheselectionoftoolsanddatathatsupportthemeasurementoftheoutcomesagreedratherthanlookingforopportunitiestomonitorallthethingsthatitisfeasibletomeasure.Theresultshouldbeamoreefficientwaytocaptureeffectivenessandamorepowerfulstoryofimpact.

Inevitably,differentprojectsaddressdifferentelementsinthepathwaytowardsdesistanceandwillthereforedeliverdifferentintermediateorproximaloutcomes.Proximaloutcomesarevaried.Therearesoftoutcomessuchasimprovingself-esteem,whilehardoutcomes,forexamplegainingemployment,willbemoretangible.AccordingtoareportcommissionedbytheBigLottery(2013)toreviewtheeffectivenessoffundedprojectsinreducingrecidivisminScotland,evenwhenprojectsaresuccessfulinachievingintermediateoutcomes,therelationshipbetweentheintermediateoutcomesanddesistanceisnotalwaysclearandnotalwaysmeasureable.However,inordertoensurearobustapproachtooutcomeevaluation,anumberofprinciplesneedtobeaddressed:

1. Theprocessmusttakeaccountofallstakeholdersincludingthecommissioners

2. Itisaprocessdeterminedatprojectoutsetandresourcedappropriately.

3. Itstartswhentheprojectstartsandisnotjustleftuntiltheendoftheprocess

4. Usesacredibleevaluationteamwhoareskilledincollectingdatainmultipleformats


5. Employsacredibledesign

6. Providesathoroughaccountbyusingateamtoensureappropriateobjectivityandprovide‘believability’(Scriven1991,1994)

7. Usesreportingasastrategyandtoolforon-goinglearningratherthananendproduct

Thereareanumberofoutcomesframeworksalreadyinusewithyoungpeople7,andalsosometoolswhichattempttocaptureamorequantitativemeasureofprogress–forexample,Goodmanetal’sStrengthsandDifficultiesQuestionnaireforadolescentsandNPC’sWellbeingMeasure(2011).Afirstassessmentisusedtoprovideabaselinemeasurewhichisrepeatedattimelyintervalsduringtheintervention.

Thedangerforsomeisthatstandardisedoutcomemeasurescanoftenfocusondeficitsandcansubsequentlypathologisedifficultiesexperiencedratherthanfocussingonthemorepositiveoutcomesidentifiedearlier(Smith1994).Theyhavealsobeenaccusedofnotbeingaccessibletoyoungpeoplethatmayhaveliteracyorlearningdifficulties,andoftenemploynormativemeasures.Theterm‘normativeassessment’referstotheprocessofcomparingonetest-takertohisorherpeers.

AbetterknowntoolistheOutcomesStarwhichcapturesthemoresubjectiveindicatorsofprogressandisbasedupona‘cycleofchange’approach.Itidentifiesfivekeystageswhichmovefrom‘beingstuck’,‘acceptinghelp’,‘believing’,‘learningtoreachpotential’andfinally,‘self-reliance’.Ithasbeenreinventedbyarangeofprovidersandmorethan20versionsnowexist.8 Thetoolhasawiderangeofenthusiasticsupportersinthefieldbecause,forsomeyoungpeople,itcanprovideausefulframeworkforprioritisingandproblem-solvingaroundkeyareasofconcern.OnedifficultyinusingOutcomeStardatatoassessimpactisthatthescoresgeneratedbythetoolarearrivedatwithinatrustrelationshipwhichevolvesovertime,andchangesinscorescanthereforemeaneitherthatrealchangehastakenplaceinrelationtokeyareasofneed,orthataprojectparticipanthasreachedapointwheretheyarereadytodiscloseparticularissuestoapractitioner.ItisthisfeatureoftheOutcomeStarwhichmakesitmoredifficulttounderstandchangesinscoresovertime,thanitisforstandardpsychometricmeasures,forexample.

Anothertoolwhichisusedincreasinglyinthefieldforcapturingindividualchangeovertimeinrelationtokeydimensionsofpersonal/emotionalwellbeingisIOMI(theIntermediateOutcomesMeasurementInstrument).Ithastheadvantageofbeingaccessibleandquicktoadminister,butalsoofbeingdesignedtogeneratedatawhichcanbeusedforresearchpurposesandforassessingprogressinastandardisedmanner.IOMIisusedformeasuring‘distancetravelled’inrelationtokeydimensionsincludingresilience,wellbeing,agency/self-efficacy,impulsivity/problem-solving,motivationtochange,hope,andinterpersonaltrust.Designofthetoolwascarefullyanchoredinfeedbackfromanationalconsultationwithseveralhundredserviceproviders,andhasbeentestedforreliabilityandvalidity(althoughfurthertestingisalsoplannedorunderway).9

The cost-benefit approachFormany,itwillalsobeofparticularimportancetotryandestablishthecostsandbenefitsassociatedwithresettlementwork,giventhesubstantialcostsinvolvedinrespondingtooffending.Thefollowingnationalfiguresprovideashockingpictureofthecostsassociatedwiththeyouthjustice system.

• £3,620:estimatedaveragecostofafirsttimeentrant(under18)tothecriminaljusticesysteminthefirstyearfollowingtheoffence

• £22,995:estimatedaveragecostofafirsttimeentrant(under18)tothecriminaljusticesystem,nine years following the offence

7https://www.gov.uk/government/publications/outcomes-frameworks-a-guide-for-providers-and-commissioners-of-youth-services

8 www.outcomesstar.org.uk

9Liddle,M.,withDisley,E.,Maguire,M.,Meek,R.andRenshaw,J(forthcoming,2016)IntermediateOutcomesMeasurementInstrument(IOMI)toolkit–GuidanceNotes.MinistryofJusticeAnalyticalSeries.London:NationalOffenderManagementService.


• £113,000,000:estimatedsavingsifonein10youngoffendersweredivertedtowardeffectivesupport

• £100,000:theaveragecostofaplaceinyouthcustody.In2012/13YOIscostaround£65,000aplaceperannum,STCscost£178,000aplaceperannumandSCHs£212,000aplaceperannum

Thereisanobviouseconomicimperativethatmakesastrongcaseforgettingresettlementrightandprovingsuccess.Thecostbenefitargumentcanbeusedtoensureappropriatelevelsofresourceareavailabletosafeguardgoodqualityresettlementservices.Indeedthecorollary,gettingresettlementwrong,canprovetobeextremelyexpensivetosocietyatlarge.Projectscanusefinancialinformationtocreate‘cost-avoidance’calculationstodemonstratecosteffectiveness.AcostavoidancetoolforyouthdiversiondevelopedbytheCentreforJusticeInnovationsprovidesausefulframework.10 There areanumberoftoolkitsandsoftwarepackagesavailabletothecriminaljusticesystemthatallowforservicestodemonstratehowtheirworkissavingthepublicpursemoney.(SeeLiddle,2016):‘Usingindividualcasestudiestohighlightthecostsandbenefitsofeffectiveresettlement’.

Complimentary methods Ofcourse,theaboveexamplesarenottheonlymethodsofdatacaptureorproofofeffectiveness,andtheuseofamixedapproachcanbecomplimentary.Othersourcesofevidenceofeffectivenesscouldincludethefollowing:

• Offending history Asalreadyreferencedearlier,theMoJpublishesfiguresonratesofreoffendingbyconvictionoffenceanddisposal.Thesefigurescanhelpprovidersunderstandthelikelihoodofreoffendingandenableacomparisontobedrawnbetweenexpectedandactualreoffendingrates.

• Case studies Casestudiesareanessentialadjuncttorobustquantitativedataandprovideverypowerfulevidenceespeciallywheretheyoungperson’svoiceiscentraltothetellingofthestory.However,asqualitativedata,theyremainsubjectiveandindividualisticintoneandnormallyonlycapturesuccessstoriesforpublicrelationspurposes.Whenprojectsareabletosharethosecaseswhichdidnotworkoutasintendedtheyprovideanextremelyvaluableinsightacrossthesector.Wherecasestudiesaredrawnfromasampleinordertoberepresentative,theycanbeextremelyeffective.BYC’srecentpublicationonassessingthecostsandbenefitsofresettlementworkconsidersarangeofwaysinwhichevidencefromindividualcasestudiescanbeusedtoestimatesuchcostsandbenefitsovertime.

• Risk profile Collectingdataonriskfactorscanhelpproviderstargettheirinterventionsmoretowardsthoseyoungpeoplewithcomplexneeds.Thiscanavoidtheaccusationof‘cherrypicking’thoseyoungpeoplethatareeasiertoengage,aclaimoftenmadebycriticsofthepaymentbyresultsapproach(NPC2011).

Putsimply,formethodstobeuseful,theyneedtoprovidetheinformationrequired,beseenasreliableandreadthroughthelensofwhatcanbedoneandknownonthatoccasion.

The challenges in measuring effectivenessTherearesignificantchallengesinmeasuringtheimpactofinterventionsthatcontributetoareductioninreoffendingorindeedfacilitateashiftinayoungperson’sidentity.Thevastmajorityoffundedprojectsoperateatarelativelysmallscale.Itisdifficulttocarryoutrobustquantitativeevaluationofsmallprojectstoshowtheirimpactonreoffendingassuchapproaches,(forexample,RCTs),requirelargenumbersofserviceusersandnon-users.Additionally,andasdiscussedearlier,therelevanceandreplicabilityofsuchmethodologieswithinsocialresearchiscontested.

10http://www.justiceinnovation.org/youth-diversion/valuing-youth-diversion-toolkit


Theapplicationofoutcomesframeworkslikethosementionedabovearenot,however,unproblematic.Thereareanumberofissuesthatthosemakingclaimsneedtomanage,theseinclude:

1. Howtodefineindicatorsfrommeasuringoutcomesthataremeaningfulbutdonotdistortpractice

2. Ensuringanaccurateattributionofresponsibilitywhereoutcomesaresharedacrossproviders

3. Thedistortingeffectsofperformancemanagementandhowtoaccountforunintendedconsequenceswhichmaynotalwaysbeconsideredpositive(Lowe2013)

Theactualimpactofanyinterventionisdeterminedbythecontextwithinwhichitisreceivedandunderstoodintermsofthecomplexityofthelifeitintendstochange.Thismakesitextremelydifficulttounderstandandmeasurethatimpact(Widdershiven&Sohl1999).Suchanunderstandingwouldrequiretheuseofbothintensivequantitativeandqualitativelongitudinalresearchmethods(Shalock&Bonham2003).Instead,manyusewhatSmithetal(1997)call‘tracerconditions’orproxymeasureswhicharesimplerandcheapertocollect.Whilstsuchanapproachgathersdatawhichcanbeusedforcomparativepurposesovertime,careneedstobetakentoensurethattheydonotremoveanyunderstandingwithincontext.

Secondly,itraisedthequestionofhowwecanattributeanimpacttoaparticularintervention.Foranyinformationonoutcomestoberobust,itmustbeabletoevidencethattheinterventioncausedtheoutcome(Shalock&Bonham2003).Noservicecouldpossiblyclaimtohavecompletecontroloverwhetherornotsuchanoutcomewasachieved.Theresponsetosuchcriticismhasbeentodevelopnewmethodologies,henceToCor‘logicmodels’whichvisuallymapcausalchainsofimpact.Outcomesarenottheresultofalinearprocess;tosuggestsoistoover-simplifyanindividual’sjourney.

Additionally,formany,successisnowjudgedupontheattainmentofoutcomes.Asaresult,ifmanagersacceptthatthereisalotaboutoutcomesthattheycannotcontrol,theybecomeadeptatmanipulatingdatabymanagingthingstheycancontrol.Theseissuesarefrequentlydescribedastechnicalchallengesinmethodsofdatacollectionthatcanbeovercome(Perrin1998).Forexample,particularlyinthePBRworld,thereareaccusationsof‘cherrypicking’thosewhoareeasiertohelp.Consequently,accordingtoLowe(2013),managerswilladapttheirbehaviourtomeetthenewlyconstructedtargets.Additionally,theimpactofOBPMonfrontlinestaffhasfrequentlybeencitedasthereasonforlesstimebeingspentdevelopingrelationshipswithyoungpeople,andmoretimecollectingdatainordertoproveeffectiveness(Keeversetal2012).Consequently,OBPMoftendistortstheprioritiesandpracticesofprofessionals.

Datacollectioncanalsoprovechallengingasdifferentorganisationsuseorhaveaccesstodifferentdatasets,particularlystatutorysources.Accesstosuchinformationforthirdsectororganisationsremainsvariable.Asaresult,statisticsoftenlackcoordination,makinganykindofcomparisonproblematic.Asmentionedearliertheneedtotrackyoungpeopleforayearpostinterventioninordertodeterminesuccessinreoffendingisthereforefrequentlyunobtainable(NPC2010).

What has been learnt from these fact-finding activities?1. Thereare,ofcourse,variationsinthewaythatkeystakeholdersdefine‘success’inresettlement

work–itishighlysubjective.However,thereareanumberoffactorswhichcanaidsuccessfulresettlementprocesses.AccordingtopractitionersattheNAYJConferenceinOctober2015,thekeyelementsinensuringaneffectiveresettlementprocesswereidentifiedasfollows:

• Planningandinvolvingyoungpeoplefromthebeginning• Consideringresettlementaspartofthestartofasentence–notjustaspartoftheexit


• Securingaccessintothesecureestateiskeysothatrelationshipscanbeformedwith workers before release• Moreuseofmentors/advocateswithincustody• Buildingrelationshipsandkeepingconsistency• Professionalsworkingtogetherandsharinginformation–ledbyoneprofessionalwhoco- ordinatestheprocess• Proactiveparticipationbyallresponsibleforsentenceplanning• Bespokepackagesforeachindividual• Keepingyoungpeopleclosertohome• Useofmobility/ROTLandplacementstoaidtransition• Properresourcingandfunding• Continuationofsupportpostlicence/order

2. Thedebatesoverwhetherthereisaneedforsomekindofuniversalmechanismorframeworktodemonstrateeffectivenessinresettlementworkwillcontinue.Inordertomaintainthediversityacrossthelandscapeofserviceproviders,anykindofmeasurewouldneedtoretainaflexibilitythatcanreflecttherangeofinterventionsonoffer.

3. Anymeasurementofimpactinresettlementneedstorememberthatthereareno‘humanuniversals’thatcanbeapplieduncriticallytoallandthereforestandardisedindicatorsthatmatchuptosomenotionofwhatsuccesslookslikeforanindividualdonotalwaysconsiderthecomplexitiesofthelivesofthosetheyarehopingtosupport.Socialimpactmeasurementalsoneedstoconsiderissuesofpower,agency,choiceandfreedomandhowthesehumanneedsplayoutamongsttheusersofservices(Hudson2015).Inshort,itisaboutdemonstratingandmeasuringtherightthing,notnecessarilyeverything(Clinks2010).

4. Resettlementproviderscanuseapositiveyouthdevelopment(PYD)outcomesframeworktomeasurea‘shiftinidentity’althoughtheopportunitiesforaggregation/scalingwillbelimited.Howtomeasurewhatmadeadifference–questionsofattribution–aretrickyasitisnotusuallyonething,butacombinationoffactors.Thetimescalesformeasuringsuccessareextremelyvariable,andthereisrecognitionthatlongitudinalstudies,thoughrareandcostly,couldhelpustounderstandsomeofthesecomplexitiesoverthelongerterm

5. Impactmeasurementshouldnotbeextractedfromuserfeedbackastheseformthebasisof‘client-centred’approachestoresettlementthatretainelementsofgoodpracticehighlightedpreviously–individualised,joinedupetc.Suchfeedbackiscrucialevidenceinfurtheringourunderstandingofimpactandeffectiveness;ithelpsustounderstandandilluminatethecausalchainsweseek.However,thesetwoaspectsofprogrammeevaluationmayremainintensionandthoseresponsibleforserviceevaluationneedtofindappropriatewaystoprovideanevidentialtapestrywhenreportingonimpact.

6. Commissionershavealsobeguntorecognisethatbeingabletodemonstrateclear,attributableoutcomesisnotstraightforward.Whiletheymaydesirestatisticalevidence,qualitativeinformation,casestudiesandstoriesallhelptobuildupapictureofimpactand,iftheevidenceispresentedthroughtheuseofcreativemethods,canconveyapowerfulmessage.Proximaloutcomes,suchassustainingatenancyandmaintainingengagementinemploymentoreducation,maybeeasiertodemonstrateandcanbestronglyindicativethatthedistaloutcome–ashiftinidentity–isbeingachieved.Ofcourseresettlementservicesneedtobeabletoevidencetheirimpact,butwithoutitdivertingsignificantresourcesfromfrontlineservicedelivery.

7. Thereisanincreasinginterestinmeasuringimpactswhicharesharedacrossanumberofproviders.Sharedmeasurementaimstomakeiteasierfororganisationstolearnfromeachother,savethemthecostsofdevelopingtheirowntools,andbuildanevidencebaseofwhatworks.Itcanthereforeallowmoreconsistencyandcomparabilityinimpactmeasurement.In2013,InspiringImpact’sBlueprintforsharedmeasurement11identifiedkeystagesandsuccess

11http://inspiringimpact.org/wp-content/uploads/2013/03/blueprint-for-shared-measurement2.pdf?Downloadchecked=true&utm_source=New%20Philanthropy%20Capital&utm_medium=email&utm_campaign=6540192_shared%20measurement%20launch%202016&dm_i=UL9,3W6G0,L1WOF9,E1BPL,1


factorsindevelopingasharedmeasurementapproach.Theworktheyhaveundertakenexplorestheconditionsnecessaryforsharedmeasurement,aswellasthekeyfactorsindeveloping,designing,scalingandsustainingapproaches.

8. Commissionersarealsolookingforwaystoco-commissionservicesinordertoincreasetheirimpactacrossissuesorlargerlocalities,therebyachievingevengreatervalueformoney.Opportunitiesforservicestobidjointlywithothers,forexamplePoliceandCrimeCommissioners,isdeemedveryattractiveandaddsgreatervaluetoanapplicationespeciallyifservicescandemonstratethattheywilladdvalueorsecurematchfundingfromothercommissioners.Afterall,keyprioritiesforlocalauthorities,CRCsandpoliceandcrimecommissionersincludethepreventionofoffendingaswellreducingreoffending(Clinks2010).

9. Whatifbothcommissionersandproviderslookedforthethingsthatworked,andtriedtoreplicatethem,ratherthanafocusuponcreatingsomethingdifferentandnew?ThisistheapproachtakenbyRealisingAmbition12,a£25millionBigLotteryFundprogrammesupportingthereplicationofevidence-basedandpromisingservicesdesignedtoimproveoutcomesforchildrenandyoungpeople.ForRaisingAmbition,goodreplicationallowsforinnovativepracticeasitisasequentialdevelopmentofearlierworkandgivesprovidersopportunitiestotestinitiativesthathaveworkedelsewhereinnewcontexts.Skilfulprovidersareabletoidentifywhatneedstostaythesame,coreservices,andwhatneedstochangetomakeitrelevantwithinthenewcontext.Thefivekeyingredientsofsuccessfulreplicationare:

“A tightly defined service; effectively and faithfully delivered to those who can benefit from it; that provides confidence that outcomes have improved; that is cost-beneficial and scaleable; and that is delivered by an organisation that uses evidence to learn and adapt, as required.”

RealisingAmbition2015

ConclusionSo,doesmeasuringeffectivenessinresettlementworkwithyoungpeoplematter?Yes,itdoes.Thereisahugeamountofenergyandmoneybeingspentinassistingorganisationstomeasuretheirimpactthroughthegenerationofrelevantevidence.However,wemustrememberthatevidenceisneveranabsolutetruth,itisoftencontingentforanumberofreasonsandthereforetheapplicationofnotionsofevidence-basedpracticeorevidence-basedcommissioningwillneverdemonstratethesameoutcomeswithoutthenecessarycaveatof‘context’.Asaresult,itwouldbeappropriatetoconsideramorenuancedapproachwhichencompassesthebestofwhatweknowandlookstodevelopourunderstandingoftheinterventionswemakethrough‘practice-basedevidence’ratherthanbeingweddedtoother’sexpectationsof‘evidence-basedpractice’.

Professionalsacrosstheyouthjusticesectorwanttodemonstratethedifferencetheirinterventionsmaketothelivesoftheyoungpeopletheyworkwith.However,theresponsetothe“Whatworks?”questionisnottheonlyevidenceweneedtouse.Thereareanumberofotherquestionsweneedtoaddress,asdifferentstakeholderswillhavedifferentinterestsintheanswers,forexample:

1. Istheneedfortheserviceestablished?

2. Istheprojectreachingthoseitwasintendedtosupport?

3. Aretheyoungpeoplefullyengagedwiththeserviceanddotheycontributetoservicedevelopment?

4. Isitbeingdeliveredtoanappropriatequalityorstandard?

5. IsthereexistingevidencetosupporttheviewthatbydoingX,Yislikelytooccur?

6. Howmuchdoesitcostandisitofferingvalueformoney?

12http://www.catch-22.org.uk/wp-content/uploads/2015/06/Realising-Ambition-Programme-Insight-The-Secret-Life-of-Innovation.pdf


ForTimHobbs(2015),evidencegivesus‘confidence’,anditsstartingpointhastobeensuringthatourdatacollectionmethodsarefitforpurpose.Whileitisrecognisedthatthemorerobustsuchmethodscanbe,thegreaterthepotentialfordemonstratingimpactregardlessofmethod,itsanalysisshouldincreaseourconfidenceinwhatwearedoingandwhy.

Measurement,whetherinoutcomesorotherindicators,isneverneutral;wehavemuchtolearnfromserviceuserfeedback.Thesectorneedstoaddresswhoissettingthestandardandreclaimtheagendatoensurethattheappropriateevidenceisbeingtakenintoaccount,andthatthisispredicatedontheyoungpersonbeingcentraltothedebate.Suchadiscussionwillinevitablyconsiderwhat‘effectiveresettlement’lookslike,andhowthatunderstandingcanbecomeembeddedacrosspolicyreforms,contractspecifications,supplychains,collaborativeworkingrelationships,evidencegathering,and,inindividuallytailoredandholisticresettlementplans.

TheongoingTaylorReviewofYouthJusticepublishedaninterimreportonitspreliminaryfindingson9thFebruary2016.13 Itbeginstosignpostthefuturedirectionofwhatwecurrentlyrecogniseastheyouthjusticesectorandthechallengesinseparatingresponsibilitiesforthesectorbetweencentralandlocalgovernment.Inparticular,itidentifiesaclearintentiontofurtherdevolveprocessesandbudgetstolocalareasinordertoavoidsomeofthedifficultiesassociatedwiththetransitionforyoungpeoplefromcustody.Inthisway,theintentionwouldbeforlocalpartnerstodesignandcommissionservicesforvulnerablegroupsofyoungpeople,forexample,mentalhealthservices,whilealsosupportingasmoothertransitionfromcustodyandaidtheintegrationbackintothecommunity.

Thedirectionoftravelidentifiedintheinterimreportsuggeststhattheneedtodemonstratetheeffectivenessofresettlementworkwillonlygainmomentumasnewcommissioningopportunitiesbecomeavailablelocally.Thiswillrequiresufficientpoliticalwill,resources,andoperationalprotocolsinplacebetweenrelevantpartnerswhoarepreparedtoworktogetherinnewmodelsofdelivery.Thisshouldinitiateanew,honestandcriticaldebateaboutwhatprofessionalsshouldbemeasuringandhowtheycanevidencetheirimpactinresettlementworkwithyoungpeople,whichmayalsoincludethedevelopmentofarangeoftoolsthatarefitforthispurpose.

13https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/498736/youth-justice-review.pdf


ReferencesArrivoConsulting(2013)Learningfromprojectsworkingwithex-offenders,Glasgow:BigLottery

Bateman,T.(2015)Thestateofyouthjustice2015:Anoverviewoftrendsanddevelopments,London:NAYJ

Bateman,T.,Hazel,N.andWright,S.(2013)Resettlementofyoungpeopleleavingcustody–Lessonsfromtheliterature,London:BeyondYouthCustody/Nacro

Butts,J.A.,Bazemore,G.andMeroe,A.S.,(2010).PositiveYouthJustice:FramingJusticeInterventionsUsingtheConceptsofPositiveYouthDevelopment,WashingtonD.C.:CoalitionforJuvenileJustice.

BYC(2015)Effectiveresettlementofyoungpeople:LessonsfromBeyondYouthCustody,London:Nacro

BYC(2016)Resettlementworkwithyoungpeople:Usingindividualcasestudiestoassesscostsandbenefits,London:Nacro

Campbell,D.T.(1973)‘TheSocialScientistasMethodologicalServantoftheExperimentingSociety’inPolicyStudiesJournalVolume2,Issue1,pp.72–75

Cartwright,N.(2007)‘AreRCTstheGoldStandard?’inBioSocieties2,pp.11–20

CentreforYouthImpact(2015)http://www.youth-impact.uk/impact-magazine/what-were-reading/setting-standard-what-can-social-impact-measurement-learn-feedback

CentreforYouthImpact(2014)OutcomesFrameworks:aguideforprovidersandcommissionersofyouthservices,London:CabinetOffice

CentreforJusticeInnovation(2015)Valuingyouthdiversion:Makingthecase,London:CJI

DartingtonSocialResearchUnit.(2013).DesignandRefine:Developingeffectiveinterventionsforchildrenandyoungpeople,Dartington,England:DartingtonSocialResearchUnit

Funnell,S.&Rogers,P.(2011).PurposefulProgrammeTheory.Effectiveuseoftheoriesofchangeandlogicmodels,SanFrancisco:Jossey-Bass

Goodfellow,P.Wilkinson,S.,Hazel,N.,Bateman,T.,Liddle,M.,Wright,S.,Factor,F.(2015)Effectiveresettlementofyoungpeople,London:BeyondYouthCustody

Goldstein.(2002),DesigningSocialresearchforthe21stCentury.ProfessorialaddressUniversityofBristol(14.10.2002)

Goodman,R.,Meltzer,H.,Bailey,V.,(1998)‘TheStrengthsandDifficultiesQuestionnaire:Apilotstudyonthevalidityoftheself-reportversion’inEuropeanChildandAdolescentPsychiatry,No7,pp.125-130

Haines,K.andCase,S.(2015)PositiveYouthJustice:ChildrenFirst,Offenderssecond,PolicyPress:Bristol

Hatry,H.P.,1996.Measuringprogramoutcomes:Apracticalapproach,Alexandria,VA:UnitedWayofAmerica

Hughes,M.andTraynor,T.,2000.‘Reconcilingprocessandoutcomeinevaluatingcommunityinitiatives’,Evaluation,6(1),pp.37-49

Hudson,M.(2015)Settingthestandard:Whatcansocialimpactmeasurementlearnfromfeedback?Availableat:http://www.youth-impact.uk/search/node/Setting%20the%20standard(accessed05.01.16)

Hurley,M.&Pitts,J.(2015)EffectiveInterventioninGangAffectedNeighbourhoods,Unpublished:UniversityofBedfordshire

James,C.(2011)TheoryofChangeReview.AreportcommissionedbyComicRelief,ComicRelief:availableonlineat:http://mande.co.uk/2012/uncategorized/comic-relief-theory-of-change-review/(accessed05.01.16)

Keevers,L.etal.(2012),‘Madetomeasure:tamingpracticeswithresults-basedaccountability’.OrganizationStudies,33,pp.97–120

Lewin,K.(1946)‘Actionresearchandminorityproblems’inJournalofSocialIssuesVolume2(4):pp.34–46

Lewin,K.(1951)Fieldtheoryinsocialscience:selectedtheoreticalpapers(editedbyDorwinCartwright)

Lowe,T.(2013)‘Newdevelopment:Theparadoxofoutcomes–themorewemeasure,thelessweunderstand’inPublicMoney&Management,33(3)pp.213-216

McIntosh,P.,(2010).Actionresearchandreflectivepractice:Creativeandvisualmethodstofacilitatereflectionandlearning,London:Routledge

Maruna,S.,Immarigeon,R.andLeBel,T.P.,(2011).Ex-offenderreintegration:Theoryandpractice.Aftercrimeandpunishment:Pathwaystooffenderreintegration,London:Routledge

Mayne,J.,(2008).Contributionanalysis:Anapproachtoexploringcauseandeffect,ILACBriefNumber,16

MinistryofJustice(2015)Youthcustodyreport:May2015Source:https://www.gov.uk/government/statistics/youth-custody-data[accessed16.07.2015

MinistryofJustice(2013).ResultsfromtheOffenderManagementCommunityCohortStudy(OMCCS):Assessmentandsentenceplanning.Online:https://www.gov.uk/government/publications/offender-management-community-cohort-study-baseline-technical-report[lastaccessed23.10.13].

McAra,L.andMcVie,S.(2007)‘Youthjustice?Theimpactofsystemcontactonpatternsofdesistancefromoffending’,inEuropeanjournalofcriminology,4(3),pp.315-345.


McNeill,F.,Farrall,S.,Lightowler,C.,Maruna,S.,2012.Howandwhypeoplestopoffending:Discoveringdesistance.IRISSInsightsNo.15.Online:http://www.iriss.org.uk/resources/how-and-why-people-stop-offending-discovering-desistance[lastaccessed23.10.13].

NewPhilanthropyCapital(2011)Children’sWellbeingMeasure.Availableathttp://www.well-beingmeasure.com

NOMSCommissioningIntentions(2012)

http://www.justice.gov.uk/downloads/about/noms/commissioning-intentions-2013-14-oct12.pdf[lastaccessed23.10.13].

Nichols,G.andCrow,I.,(2004)‘Measuringtheimpactofcrimereductioninterventionsinvolvingsportsactivitiesforyoungpeople’inTheHowardJournalofCriminalJustice,43(3),pp.267-283

Pawson,R.andTilley,N.(1997)RealisticEvaluation,ThousandOaks,California:SagePublications

Perrin,B.(2006)Movingfromoutputstooutcomes,WashingtonDC:WorldBank

Peters,C.S.andMyrick,S.,2011.‘Juvenilerecidivism–measuringsuccessorfailure:Isthereadifference’inCorrectionsToday,74(1),pp.33-35

http://project-oracle.com/uploads/files/EvidenceDigest01_v5_electron_version_151015.pdf

Proscio,T.(2000)InOtherWords:APleaforPlainSpeakinginFoundations,NewYork:TheEdnaMcConnellClarkFoundation.

Raynor,P.(2004)‘Opportunity,motivationandchange:somefindingsfromresearchonresettlement’inBurnett,RandRoberts,C(eds)WhatworksinProbationandYouthjustice:developingevidencebasedpractice,Cullompton:Willan

RaisingAmbition(2013)TheSecretLifeofInnovation:Replication,Definingsuccessinreplicatingeffectiveservicesforchildrenandyoungpeople,Dartington:DartingtonResearchUnit

RealisingAmbition(2015)‘EvidenceisConfidence.Howtocreatearicherevidentialtapestry’inProgrammeInsightsIssue2:Availableat:http://www.catch-22.org.uk/wp-content/uploads/2015/11/Realising-Ambition-Programme-Insight-Evidence-is-Confidence.pdf

TheNationalAuditOffice(2010)TheYouthJusticeSysteminEnglandandWales:Reducingoffendingbyyoungpeople,London:Ministry of Justice

YouthJusticeBoard(2015)Reducingreoffending:furtheringourunderstandinghttps://www.gov.uk/government/uploads/system/uploads/attachment_data/file/479708/Reoffending_Furthering_Our_Understanding.pdf(accessed11.12.15)

Haines,K.,andCase,S.(2015)PositiveYouthJustice:ChildrenFirst,Offenderssecond,PolicyPress:Bristol

Hatry,H.P.(1996)MeasuringProgramOutcomes:APracticalApproach,Alexandria,VA:UnitedWayofAmerica

Schulman,S.andDavies,T.,(2007).Evidenceoftheimpactofthe‘youthdevelopmentmodel’onoutcomesforyoungpeople–aliteraturereview.Availableat:http://timdavies.org.uk/files/Davies%20and%20Schulman%20%20Positive%20Youth%20Development%20Literature%20Review.pdf(accessed16.12.15)

Scriven,M.(1991)EvaluationThesaurus,NewburyPark,CA:Sage

Scriven,M.(1994).‘Thefinelinebetweenevaluationandexplanation’inEvaluationPractice,15,75-77

Shalock,R.L.&Bonham,G.S.(2003)‘Measuringoutcomesandmanagingforresults’inEvaluationandProgramPlanning,No.26pp.229-235

Smith,A.(1994)‘IntroductionandOverview’inKiresuk,T,J.,Smith,A.,andCardillo,J,E(Eds)GoalAttainmentScaling:Applications,TheoryandMeasurement.pp.1-14,NewJersey:LawrenceErlbaumAssociates,Inc

Smith,G.R.(1997)‘Implementingoutcomesmanagementsystemsinmentalhealthsettings’inPsychiatricServicesNo.48,p.3

Stein,D.andValters,C.‘Understanding'TheoryofChange'ininternationaldevelopment:Areviewofexistingknowledge’availableatwww.learningforsustainability.net/evaluation/theoryofchange.php(accessed05.01.16)

Stern,E.,Stame,N.,Mayne,J.,Forss,K.,Davies,R.andB.Befani(2012),‘Broadeningtherangeofdesignsandmethodsforimpactevaluations.ReportofastudycommissionedbytheDepartmentforInternationalDevelopment,DFIDWorkingPaper38,London

TheYoungFoundation(2012)AnOutcomesFrameworkforYoungPeople’sServices,availableathttp://www.youngfoundation.org/publications/reports/an-outcomes-framework-youngpeoples-services.

Vogel,I.(2012)Reviewoftheuseof‘TheoryofChange’ininternationaldevelopment,ReviewReport,London:DepartmentforInternationalDevelopmenthttp://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/DFID_ToC_Review_VogelV7.pdf(accessed05.01.16)

Weiss,C.(1997).‘HowCanTheory-BasedEvaluationMakeGreaterHeadway?’inEvaluationReview,Vol.21,No.4

Widdershiven,G.&Sohl,C.(1999)inAbma,T.(ed)TellingTales:onevaluationandnarrative,AdvancesinprogramEvaluation,Grennwich:JAIPress,pp.109-128

Wigboldus,S.andBrouwers,J.(2011):‘Rigidplanorvaguevision:HowprecisedoesaToCneedtobe?’,Discussionpaper,TheoryofChange,Hivosavailableat:http://www.hivos.net/Hivos-Knowledge-Programme/Themes/Theory-of-Change/E-dialogues/E-dialogue-2(accessed24.10.15)

Wright,S.,Hazel,N.&Bateman,T.(2014)EngagingYoungPeopleinResettlement:aPractitionersGuide,London:Nacro


This review has been produced by the Beyond Youth Custody partnership, consisting of Nacro, ARCS (UK) Ltd, The Centre for Social Research at the University of Salford, and the Vauxhall Centre for the Study of Crime at the University of Bedfordshire.

Authors: Fiona Factor

© Beyond Youth Custody 2016

Website: www.beyondyouthcustody.netEmail: [email protected]: @BYCustody

Documents

PROVING ‘EFFECTIVENESS’ IN RESETTLEMENT...3 | Proving 'Effectiveness' in Resettlement Glossary • INPUTS – The resources invested, staff, time, buildings etc • OUTPUTS –