Loughborough UniversityInstitutional Repository

A holistic, learning-centredapproach to buildingevaluation capacity in

development organizations

This item was submitted to Loughborough University's Institutional Repositoryby the/an author.

Citation: LENNIE, J. ... et al, 2015. A holistic, learning-centred approach tobuilding evaluation capacity in development organizations. Evaluation, 21 (3),pp. 325-343.

Additional Information:

• This paper was accepted for publication in the journal Eval-uation and the definitive published version is available athttp://dx.doi.org/10.1177/1356389015590219.

Metadata Record: https://dspace.lboro.ac.uk/2134/23582

Version: Accepted for publication

Publisher: SAGE c© The Author(s)

Rights: This work is made available according to the conditions of the Cre-ative Commons Attribution-NonCommercial-NoDerivatives 4.0 International(CC BY-NC-ND 4.0) licence. Full details of this licence are available at:https://creativecommons.org/licenses/by-nc-nd/4.0/

Please cite the published version.

1

Aholistic,learning-centredapproachtobuildingevaluation

capacityindevelopmentorganisations

JuneLennie,JoTacchi,MichaelWilmoreandBikashKoirala

PapersubmittedtoEvaluationon22-10-2014thenrevisedon17-04-2015

Introduction

Growingpressuresfromdonorstodemonstrateaccountabilityandvalueformoney

throughimpactevaluation,plusincreasedemphasison‘developmentpartnerships,

localownershipandgoodgovernance’(Dabelstein,2003:367)haveledtoincreased

interestinevaluationcapacitydevelopment(ECD;alsoknownasevaluationcapacity

building)amongstdevelopmentorganisations.1ThisinterestinECDhasbeendrivenby

anumberoffactors,especiallygrowingappreciationofthevalueofparticipatory,

systemsandcomplexity-basedapproachesforevaluatingcomplexdevelopment

interventions(Armytage,2011;Morgan,2013;Patton,2010).Theuseofexternal

evaluatorstoundertakeevaluationsofdevelopmentinitiativesisalsofrequently

questionedgiventhattheyoftenlackagoodunderstandingofthelocalcontextand

areunableto‘adequatelycapturewhatisgoingon’(Naccarellaetal.,2007:231).

Sonnichsen(1999:56)suggeststhatabenefitofusinginternalevaluatorsisthatthey

2

‘havealong-termcommitmentandcanactinthecapacityofchangeagents,increasing

organisationalperformance’.Inaddition,longitudinalresearchbyRobinsonand

Cousins(2004)hasclearlydemonstratedthepositiveeffectofinternalparticipatory

evaluationonorganisationallearning.Relatedresearchhashighlightedthepositive

roleofcriticalreflectionandevaluationindevelopinglearningorganisations,which

enablesthemtoeffectivelyrespondandadaptincomplexandrapidlychanging

contexts,suchasthoseindevelopingcountries(BehrensandKelly,2008;Hay,2010;

Pearson,2011).

However,improvingandsustainingevaluationcapacitiesoftenpresentparticularly

difficultchallengesfororganisationsbasedindevelopingcountries,becausetheyare

oftentime,capacityandresourcepoor(LennieandTacchi,2013).Effortstointroduce

ECDintoorganisations’workmaybehinderedbythesamefactorsthatdevelopment

organisationsactuallyseektoaddressintheirwork,suchasthevulnerabilityand

unpredictablenatureofinstitutionalsystemsindevelopingsocieties,highlevelsof

poverty,andlackof‘resources,opportunities,andexposuretonewideas’(Ofirand

Kumar,2013:14).TherearealsosignificantcostsassociatedwitheffectiveuseofECD

approaches(Cousinsetal.,2013),becausetheymayrequirenew‘peopleskills’

(Patton,2010:49).WhilevariousECDapproachesandmodelshavebeendeveloped,

3

theytendtoneglectthesignificantchallengesandissuesexperiencedbyorganisations

indevelopmentcontexts.

Consequently,thispaperaddressesanurgentneedtoimproveourunderstandingof

thebarriersthatstandinthewayofeffectiveECDandtoidentifyapproachesthatcan

overcomethesewithoutunderminingtheessentialaimsofECD.WearguethatECD

requireslong-termcommitmenttocapacitydevelopmentthatmaybeatoddswiththe

focusonone-offworkshopsandshort-termcapacitydevelopmentofindividualstaff

thattendtodominateintheinternationaldevelopmentfieldduetotheexigenciesof

project-basedfinancing.Kuzim(2010:240)pointsoutthat‘althoughtrainingis

importantinthe[evaluationcapacitybuilding]ECBprocess,itisnotsufficientfor

buildingasustainableevaluationcapacityinorganizations’.Thisissupportedbya

recentinternationalworkshopthatidentifiedthe‘superficial’resultsofsuch

approaches(Bayleyetal.,2012:5).Sternetal.(2012)suggestthatnewstrategiesare

neededtoenhancethecapacityofdevelopmentagenciesandevaluatorstosupport

theuptakeofabroaderrangeofdesignsandmethodsforimpactevaluationin

internationaldevelopment.

4

Ourargumentisbasedonevidencedrawnfromaproject,AssessingCommunication

forSocialChange(AC4SC),undertakenincollaborationwithEqualAccessNepal(EAN),

anNGOthatproducesradioprogramsfocussedonsocialchange.Weexplainthe

emergenceoftheholistic,learning-centredapproachtoECDinthisprojectbefore

outliningthekeyfeaturesofthisapproachandhowtheywereappliedinthiscase.The

projectdrewonapproachesfromanumberofrelatedfieldsandcombinedthemto

createanapproachtoECDthatisparticularlyappropriatefordevelopment

organisationsworkinginfieldslikecommunicationfordevelopment(C4D),thatoften

struggletoproducestrongevidentialbasesforimpactevaluationduetothe

complexityofcausalconnectionsbetweeninterventionsandindividualorcommunity

behaviours(LennieandTacchi,2013).Wealsooutlinesomeofthechallengesand

issuesthatinevitablyarisewhenusingthisapproachindevelopingcountrieslikeNepal

thathavestruggledtoimprovestandardsoflivingagainstabackdropofchronic

politicalinstabilityandcivilwar.DrawingonlearningsfromAC4SCandrelated

research,weidentifyessentialprinciplesforECDandstrategiesforthesustainable

implementationofthisapproachtoevaluation.

5

Evaluationcapacitydevelopment:definitionsandissues

MostdefinitionsofECDemphasisethatit‘isacomplexphenomenoninvolvingissues

ofindividuallearning,organizationalchange,sustainedchangeandprogramprocesses

andoutcomes’(Labinetal.2012:328).Definitionsalsoemphasisethatitinvolves

providingstaffwiththeskillstoconductrigorousevaluations,anddoingthisinaway

thatbothacknowledgesthelocalcontextand‘ensuresthatsuchevaluationsbecome

partofroutinepractice’(Naccarellaetal.2007:232).ThisrequiresECDto‘fostera

cultureoforganizationalsupportforevaluationactivities,whichincludesan

appropriatelearningenvironmentandasufficientlevelofresources’(Naccarellaetal.,

2007:232).MostwritersalsoagreethatECDisabout‘increasingthesustainabilityof

professionalevaluationpractice’(Preskill,2010:224).BaTall(2009:123)suggeststhat

ECDisacontinuinglong-termprocessoflearningandchangemanagement.Akey

lessonfrominternationalECDexperienceisthat‘buildinganeffectivecapacityfor

monitoringandevaluationisneitherquicknoreasy’andthereisaneedfor‘steady

andsustainedsupportbyinternationaldonors’(Schiavo-Campo,2005:13).

PreskillandBoyle(2008)havedevelopedauseful,comprehensiveconceptualmodelof

ECD.Takingasystemsapproach,thismodel‘illustratesanddescribesasetoffactors

thatmayinfluencetheinitiation,design,implementation,andimpactthatECB

6

activitiesandprocesseshaveonsustainableevaluationpractice’(PreskillandBoyle,

2008:444).Basedontheconceptsintheirmodel,theyprovidethefollowingdetailed

definition:

ECBinvolvesthedesignandimplementationofteachingandlearningstrategies

tohelpindividuals,groups,andorganizations,learnaboutwhatconstitutes

effective,useful,andprofessionalevaluationpractice.TheultimategoalofECB

issustainableevaluationpractice-wherememberscontinuouslyaskquestions

thatmatter,collect,analyze,andinterpretdata,anduseevaluationfindingsfor

decision-makingandaction.Forevaluationpracticetobesustained,

participantsmustbeprovidedwithleadershipsupport,incentives,resources,

andopportunitiestotransfertheirlearningaboutevaluationtotheireveryday

work.Sustainableevaluationpracticealsorequiresthedevelopmentof

systems,processes,policies,andplansthathelpembedevaluationworkinto

thewaytheorganizationaccomplishesitsmissionandstrategicgoals(Preskill

andBoyle,2008:444)

PreskillandBoyle’smodelhassomebasicsimilaritiestotheECDapproachoutlinedin

thispaperbutdoesnottaketheparticularchallengesandissuesofthedevelopment

contextintoaccount.Thefocusofthismodelonembeddingevaluationintothe

7

everydaypracticesofanorganisationandusingevaluationtohelpguideandsupport

decisionmakingandactionhasbeenlabelled‘mainstreamingevaluation’(Fitzpatrick

etal.,2012:236).ThisisakeyaspectoftheholisticapproachtoECDthatweadvocate

andisparticularlyimportanttodevelopmentorganisationsthatmustcontinually

adjusttheirpracticeinresponsetoconstantlychanginggoalsandconditions.Lennie

andTacchi(2013:94)pointoutthatdevelopingevaluationcapacitycanbeseenas

‘partoftheprocessofinstitutionalizingevaluationandcreatinganevaluationculture

withindevelopmentagenciesandtheirgovernmentandNGOimplementingpartners’.

TheyseethisprocessasvitaltowideningappreciationofthevalueofC4Dinreaching

developmentgoals.

TheECDapproachdetailedinthispaperdrawsonLennieandTacchi’s(2011)research

todevelopaUNInter-agencyResourcePackonResearch,MonitoringandEvaluationin

C4D.Thisincludedanin-depthreviewofliteratureonevaluationandECDinthe

developmentandC4Dfieldsandconsultationswithaninternationalexpertpaneland

specialistsfromsevenUnitedNations(UN)agenciesandotherbodies.Thisresearch

identifiedalackofevaluationcapacityatalllevels,especiallyinapproachesthatwere

consideredmoreeffectiveforevaluatingC4D,andlackofopportunitiesforongoing

capacitydevelopment,trainingandsupport.Theresearchalsoidentifiedmany

8

challengesandissuesforevaluatingC4Dthatapplytotheevaluationofcomplex

developmentinitiativesmorebroadly.

Acomprehensive,overarchingframeworkforevaluatingC4Dhasbeendeveloped,

whichincludesasignificantfocusonevaluationcapacitydevelopment(Lennieand

Tacchi,2013;TacchiandLennie,2014).Theframeworkrespondstothemain

challengesandissuesidentifiedinAC4SCandintheresearchfortheUNInter-agency

ResourcePack.Itcomprisesseveninter-relatedcomponents:participatory,holistic,

complex,critical,emergent,realisticandlearning-based,whichareeachunderpinned

byasetofprinciples(seeFigure1).

InsertabouthereFigure1:Keycomponentsandconceptsintheframeworkfor

evaluatingC4D(fromLennieandTacchi,2013:143)

IntroducingourcasestudyoftheAC4SCproject

WhileECDindevelopedcountriesisusuallyfocussedatorganisationalandindividual

levels,Carter(2013:3)notesthatlessonslearnedontheprocessofECDindeveloping

countrieshighlighttheimportanceofworkingatthreeinterdependentlevelsof

capacity:‘theenablingenvironment,theinstitutionalframeworkandtheindividual’.

9

Wedrawonevidencerelatedtoeachoftheselevelsderivedfromacase-study,

AssessingCommunicationforSocialChange(AC4SC)thatisparticularlyvaluable

becauseitisbasedonaseven-yearlongitudinalstudyoftheimplementationofa

participatory,holisticapproachtoECD.Originallyplannedtolastforfouryearsfrom

2007to2011,theAC4SCprojectprovidedsignificantlearningsaboutECDincomplex

andchallengingdevelopmentcontexts,especiallythroughopportunitiestodofurther

evaluationofon-goingimpactaftertheofficialcompletionofthefundedproject.

AC4SCdeveloped,implementedandevaluatedaparticipatorymethodologyfor

assessingthesocialandbehaviouralchangeimpactsofradioprogramsmadebyEqual

AccessNepal(EAN).Thisactionresearchprojectwasacollaborationbetween

researchersfromQueenslandUniversityofTechnologyandtheUniversityofAdelaide

inAustralia,EqualAccessInternational,basedinSanFrancisco,USA,EAN,local

stakeholders,andanetworkoftrainedcommunityresearchers(CRs)infivediverse

districtsinNepal.Asfaraspossible,theprojectattemptedtoimplementaholistic,

learning-basedapproachtoECDwithinEANandtofindwaystomaketheparticipatory

monitoringandevaluation(PM&E)systemsthatwereestablishedsustainable.

EANstaffhadpreviouslybeentrainedinethnographicactionresearch(EAR)in2005

and2006.ThismethodologyappealedtoEANbecauseitwasparticipatory,focussed

onimprovinginformationandcommunicationtechnology(ICT)fordevelopment

10

initiatives,andhasakeyaimofdevelopingaresearchculturethroughwhich

knowledgeandreflectionbecomesintegraltoacommunicationinitiative’songoing

development(Tacchi,forthcoming;Tacchietal.,2007).WhileEARwasconsidered

useful,EANfounditdifficulttoapplyitconsistentlyorsystematically.Theinitialidea

behindAC4SCwasthereforetodevelopEARintoamethodologythatwasembedded

inEAN.PreviousmodelshadreliedononeortwoEARresearchersworkingwithinand

attemptingtoinfluenceanorganisation.However,theaimofAC4SCwastotransform

theorganisationbyestablishingstrongerandmoreeffectiveM&Esystemsand

processeswithinitandtodevelopbetterM&Eskillsandcapacities,including

collaborativelyplanningevaluationsandmanagingandanalysingqualitativedata.

TheearlieruseofEARprovidedsomeunderstandingofEAN’sresearchandevaluation

interestsandneedsandhelpedtodevelopgoodrelationshipswithstaffatalllevels.

TheAC4SCprojectinvolvedworkingcloselywitharangeofEANstaff,followinga

participatoryactionresearch(PAR)approach,todevelopsystemsandprocessesto

assesstheimpactsoftwocommunityradioprograms.Withthecollaborationand

supportoftheAustralianresearchteam,EAN’sM&Eteamdevelopedresearchplans,

builtandtrainedanetworkofCRs,collectedandorganiseddatafromsitesacross

Nepal,developedsystemsofcodinganddataanalysis,andreportedregularlyto

contentteams(producers)andmanagement.

11

Weundertookanongoingmeta-evaluationoftheprojectthatassessedthe

effectivenessofthisapproach(Lennieetal.,2012).Ourcasestudydrawsonthe

findingsfromthemeta-evaluationaswellas:

● Adetailedanalysisofin-depthinterviewsconductedbyJuneLenniein2009with

nineEANstaffcomprisingtwoseniormanagementstaff,threeM&Estaffandfour

contentproductionorprogrammanagementstaff.

● AreportonfollowupresearchonEAN’sM&EsystemsandtheCRnetworkby

MichaelWilmoreandBikashKoiralainJanuary2012,tenmonthsaftertheproject

ended.

● NotesfromafurtherfollowupmeetingatEAN’sofficeinKathmanduthatJune

LennieandJoTacchitookpartinwith14staffinFebruary2013,atwhichan

updateonAC4SC-relatedactivitiesandimpactswasprovided.

● RecentinformationaboutM&EsystemsandpracticesinEANandthelong-term

impactsoftheprojectprovidedbyBikashKoirala,anM&EOfficeratEANwho

playedasignificantroleinAC4SCandthedevelopmentofthePM&Etoolkitthat

emergedfromtheproject.

12

Barrierstosuccessfulimplementationofevaluationcapacity

development

Aswehaveindicated,therearemanytensions,challengesandissuesinbuilding

evaluationcapacitiesindevelopmentcontexts,andinusingparticipatoryandholistic

approachestoECD,includingcontextualandculturalchallenges,organisationalculture

andpowerissues,staffturnover,negativeattitudestoM&E,resistancetochangeand

lackoftimeandresources,includingfundingforECDactivities(LennieandTacchi,

2013;LennieandTacchi,2014;Tacchietal.,2013).Wediscusssomeofthese

challengesandissuesbelow.

Contextualandculturalissuescanhindertheeffectiveimplementationofaholistic

approachtoECDwithintime,capacityandresource-poororganisationsindeveloping

countries.Akeyissueisthatdevelopmentorganisationsoftenfacepressuretomeet

theupwardaccountabilityneedsofdonorsratherthanthelearningneedsof

organisationsandcommunities(LennieandTacchi,2014).ManyC4Dinitiativesarenot

wellequippedtodealwiththechallengesandcomplexitiesofevaluatingC4D,given

thevariousconstraintsanddifficultcontextualtensionsandchallengesthattheyface.

AnimportantlearningfromPearson’s(2011)capacitybuildingeffortswitha

CambodianNGOwasthatonlyanunderstandingoflocalcultureandtherecenthistory

13

oftheCambodianpeoplecouldhelptoexplaintheresistanceofstafftonewwaysof

thinkingandlearning.WefacedsimilarchallengesduringAC4SCwhichaffectedthe

successofourECDactivities.Theyincludedlanguageandcommunicationproblems

andotherfactorsrelatedtothecomplexityofthesocialandculturalcontext,notthe

leastofwhichwastheaftermathofadecadelongcivilwarinNepal.Thisraises

questionsabouttheinfluenceofculturalcontextsonECDbecauseparticipant’s

expectationsoftheirownandeachother’srolesvarydependingontheirprevious

experiencesandbackgrounds.Forexample,therelativeinformalityofAustralian

academicinstitutionscontrastsgreatlytotheexpectationsofdeferencethattypically

characterisetherelationshipsofNepalistudentstotheirteachers(Tacchietal.,2013).

Inaddition, duetothe lackofawarenessamongsomecommunitygroupsaboutM&E

anditsbenefits,theyweresometimereluctanttotakepartinAC4SC-relatedactivities.

AchievingeffectiveandsustainableECDrequirestakingintoaccounttheorganisational

culture,dynamicsandwidercontext,andissuesofgender,powerandknowledge.

Organisationsformnetworksofpeoplewithdifferentagendasandinterestsand

varyinglevelsofpower,status,authority,experienceandexpertise(Cracknell,2000).

ThedegreeofconflictandcooperationamongthesegroupshasanimpactonECDand

evaluationactivities.GenderandpowerrelationsarelikelytoaffectECDprocesses,

14

especiallyasthoseinvolvedoftenhavedifferentlevelsofstatus,knowledgeand

experience.

AnactionresearchstudyinalargeinternationaldevelopmentorganisationbyTaut

(2007)highlightsthepoliticalnatureofevaluationandtheneedtotakethe

organisationalworkenvironmentandthepotentiallynegativeeffectsofself-

evaluationprocessesintoaccount.Organisationalandcontextualchallengesandissues

thataffectedtheoutcomesofAC4SCincludedhierarchicalstructuresthatcreated

issueswithusingPARprocesses,regularturnoverofM&Ecoordinators,andlossofkey

leadersandotherstaff.

Employeeturnoverisapersistentchallenge,especiallyindevelopingcountrieswhere

thereisoftenashortageofpeoplewithevaluationskills,experienceandcapacities.

ThiscanundermineECDeffortsduetoproblemswithmaintainingcapacityandskills

andvaryinglevelsofcommitmenttotheECDprocessfromnewstaff(Atkinsonetal.,

2005;Nappetal.,2002).InAC4SC,wefoundthattheinitialabilityofEANtorollout

complexPM&Esystemswasseriouslyconstrainedbyarangeoffactorsincludingloss

ofkeyM&Estaff,whichcreatedproblemswithcontinuity.

Ourexperiencesindicatethatitisbothimportanttoinvolveawiderangeofstaffin

ECDactivities,andtodevelopstrategiestoprovidecontinuityofleadershipandchange

15

agentroles,otherwiseprogressonECDcanberapidlylost.AsHortonetal.(2003:55)

suggest,thisrequiresstrategiessuchassettingupmechanismsthatenableknowledge,

skillsandchangedattitudestobetransferredtootherswithinanorganisation.

Aneedsanalysisforevaluationcapacitydevelopment

OurcriticalreflectionontheexperienceofattemptingtoimplementtheAC4SCproject

andevaluationoftheevidenceproducedthroughon-goingmeta-evaluationenabled

ustoidentifyarangeofneedsthatmustbemetinorderforECDtosucceedin

developmentorganisationsinthelong-term.Theseinclude,innoparticularorderof

priority,theneedfor:

● Aclearunderstandingofthecontext,includingexistingevaluationsystemsand

capacities;

● Aparticipatoryimplementationprocess;

● Acombinationofrelatedapproaches,includingparticipatoryevaluation,action

researchandcreativeapproachestoorganisationalcapacitydevelopment;

● Capacitybuildingacrossthewholeorganisation,includingstakeholders,especially

communitymembers;

16

● Afocusonthecommunicativeandrelationalaspectsoftheprocess;

● Embeddingevaluationintoeverypartoftheprogramdevelopmentcycle;

● Developmentofacultureoflearningwithinanorganisation;

● Incorporationoflocal/indigenousknowledge,creativityandideas;

● Buildingawiderangeofskills,knowledge,attitudes,valuesandawareness;

● AcriticalapproachtoECDthatacknowledgeschallengesandissues;

● Ongoingmeta-evaluationofECDactivities.

WenowdescribehowthesefeaturesoftheholisticapproachtoECDwereidentified

andappliedintheAC4SCproject.

Understandingthecontext,includingexistingevaluationsystemsandcapacities

AnunderstandingofthelocalcontextandcultureisimportantinECD.Akeystarting

pointisunderstanding‘thehistory,structure,cultureandcontextoftheorganisation’

(Fitzpatrick,2012:237),aswellasthecontextoftheinitiativesbeingevaluatedand

existingM&Esystemsandcapacities.

AC4SCbuiltonearlierresearchthathelpedtoprovidesomeunderstandingofthe

organisationanditsinterestintakinganewapproachtoevaluation.PriortoAC4SC

therewasalackofleadershipinevaluationwithinEAN.TherewasnoM&Emanager

andonlyasmallM&Eteam.Feedbacksystemswerepoorandamorecoordinatedand

17

rigorousapproachtodatamanagementandanalysiswasneeded.Indicatorsthatwere

usedtoassessprogrameffectivenessandimpactweredevelopedwithoutcommunity

inputandwereoftenseenasunrealisticandnotveryuseful.M&Ewasmainlybased

ontime-boundstudiesand‘successstories’derivedfromfeedbackthatoftencame

fromletterssentbyliteratelistenerswhowereunrepresentativeofthelistener

population.

TomoreeffectivelyplanandimplementtheAC4SCproject,adetailed‘baseline’report

wasinitiallypreparedbyMichaelWilmore.Thiscoveredthehistoryandstructureof

theorganisation,thedevelopmentofitsM&Esystems,currentM&Eactivitiesand

challenges,andvariouscontextualchallengesrelatedtoissuessuchascommunication,

travel,limitedinternetaccessandthelargevarietyoflinguisticgroupsinNepal.Initial

workshopswithEANstaffincludedidentifyingchallengesandissuesinevaluatingC4D

initiativesandstrategiesforbuildingM&EintotheireverydayC4Dactivities,anda

criticalreviewtheiruseofEAR.Analysisofquestionnairescompletedby13EANstaff

atthestartofAC4SCfoundthatmosthadamoderatelevelofexperiencewithPM&E,

aworkingknowledgeofvariousM&Emethods,andwantedmoreskillsinawiderange

ofM&Emethods.LackoftimetoundertakeM&Ewasidentifiedasakeyissue.

18

Participatoryimplementationprocesses

AkeyaimofthisECDapproachistocontributetothelong-termprocessof

encouragingcontinuousandactivecommunityandstakeholderparticipationand

engagementindevelopmentinitiativesandtheirevaluation.LennieandTacchi(2013:

3)arguethatthisprocessincreasesthelong-termsustainabilityandeffectivenessof

C4Dinitiativesandthat‘communicationandthisalternativeapproachtoevaluation

arecriticalforsustainabledevelopment’.Inthisapproach,evaluationisseenasan

ongoing,actionlearning,projectdevelopmentandimprovementandcapacity

developmentprocess.Theaimisthatthisprocessbecomesembeddedintoan

organisation’scultureanditsprojectplanningandmanagementprocesses,alongwith

regularmonitoringofandcriticalreflectionontheevaluationprocess.However,

participatoryapproachestoevaluationandECD,suchasthoseusedinAC4SC,require

greaterplanningandhigherlevelsofparticipationandengagementthanother

evaluationapproaches(Diaz-Puenteetal.,2008).

Thelearning-basedcomponentofLennieandTacchi’sframeworkforevaluatingC4Dis

particularlyrelevanttothisapproach.ThisisbasedonactionlearningandPAR

principlesandprocessesthatseektoachievegoodcommunication,cooperation,

collaborationandtrustbetweenthoseinvolved.Theaimistofacilitateandencourage

19

continuouslearning,mutualunderstanding,empowerment,creativeideasand

thinking,andresponsivenesstonewideasanddifferentattitudes,valuesand

knowledge.

AC4SCwasseenbysomeEANstaffwhowereinterviewedtwoyearsaftertheproject

beganasalreadyhelpingEANtolookatM&Einadifferentwayandtoimprovethe

qualityoftheirwork.Thisincludedunderstandingthevalueoftakingaflexible,

inclusive,‘bottom-up’approachandcontinuallyadjustingandimprovingwhattheydo.

ThisnewunderstandingisexemplifiedbythefollowingcommentfromNaresh2,a

programofficerwhoactivelyparticipatedinAC4SC:

We'velearnedthe‘learningbydoing’process.Wewouldprobablybemorerigid

ifwewerenottoimplementtheAC4SC.We'velearnedthere'salwaysaplace

foradjustmentandimprovementswithinaproject.It'snottotallyrigid.Equal

Access,thoughwealwaysvaluedourtargetaudience,butwelearnedtodo

thingsfromabottom-upapproach.Wedonotisolatepeople.Wedonotmake

decisionsontheirbehalf.Wetryandincludethosepeopleandmakethemfeel

thatit'stheirproject.It'stryingtohavethisfeelingofownership.

20

Combiningrelatedapproaches

Thisapproachdrawsonandcombinesarangeofrelatedmethodologiesand

approaches,includingactionlearning,actionresearch,participatoryevaluation,and

holisticandcreativeapproachestoorganisationalcapacitydevelopment.Itis

underpinnedbythelearningorganisationconceptandthegoalofbuildingan

evaluationcultureindevelopmentorganisations.

Participatoryevaluationmethodologieshaveclosesynergieswithcreativeandholistic

approachestoorganisationalcapacitydevelopmentsuchasthosedescribedby

Pearson(2011).Thestrengthsoftheseapproachesinclude:theyusea‘learningby

doing’approach,enablerapidfeedbackaboutthesuccessorfailureofanECD

intervention,andcanbecost-effective(Djamankulovaetal.,2010;Forssetal.,2006;

Taut,2007).FindingsfromarecentsynthesisoftheECDliterature‘confirmthe

importanceofparticipatoryprocessesinECBstrategies’(Labinetal.2012:324).

Participatoryevaluationmethodologiesareparticularlyvaluableandappropriatefor

ECDincomplexsettingswherethecontextis‘impossibletomanage’(Valeryand

Shakir,2005:87). Akeybenefitisthattheycandemystifytheevaluationprocessesand

makethemmoreaccessibletoawiderrangeofstakeholders,includingcommunity

members.

21

AC4SC’sparticipatoryapproachtoresearchandevaluationwashighlyvaluedbyEAN

staffwhowereinterviewedtwoyearsintotheproject.Severalintervieweeshad

receivedtraininginEARandunderstoodthevalueofthisapproachtounderstanding

socialchangeandlocalcommunicationnetworks,systemsandbarriers.Theyreported

valuingtheAC4SCapproachbecauseitallowedEANtounderstandthingsfroma

‘bottom-up’perspective,helpedthemtoknowtheiraudiencesbetter,understand‘the

reality’ofpeople’slives,andtoprovidea‘realpicture’ofadiversityoflistener’sviews

oftheirprograms.Theyindicatedthattheyhadbeenunabletodothesethings

effectivelythroughtheirexistingM&Emethods.Thisapproachalsohelpedthemto

identifythemostusefultoolstouseindifferentcontextsandwithdifferentgroups.

OnceEAN’sM&Estaffhaddevelopedsufficientunderstandingandcapacitiesinthe

AC4SCapproachandkeytoolsandmethods,overtimetheytrained11community

researchers3infivedistrictstouseawiderangeofparticipatoryresearchtoolsand

techniques,includingtheMostSignificantChange(MSC)technique(DaviesandDart,

2005).ThisactivitybuiltontheestablishedcommunityreportermodelthatEAN

alreadyused.CommunityreporterswerealreadyundertakingEAR-typeworkwhenthe

projectbegan.TheselectedCRswereyoungpeoplewhohadgoodnetworksinthe

community,regularlylistenedtotheradioprogramsbeingevaluated,andhadthe

capacitytoorganise,undertakeandreportonparticipatoryresearchactivitiesonan

22

ongoingbasis.TheworkoftheCRnetworkwasseenbyseveralintervieweesasan

importantmeansofdocumentingandobservingtheprocessofgradualchangeina

communityandgatheringwhattheyreferredtoas‘in-depth’,‘honest’and‘genuine’

information.

Overtime,useoftheMSCtechniqueinAC4SCprovedusefulforunderstandinga

diversityofprogramimpacts,includingpositivechangesingenderandcaste

discriminationandpoliticalempowerment,andforengagingcommunitymembersin

discussionaboutsocialchangeissues.TheMSCtechniquewasusedtocomplement

otherexistingmonitoringmechanismsbasedonletters,emails,SMSandfeedback

forms.Anin-depthreviewin2009oftheinitialtrialofMSCbytheCRsfoundthatwhile

moststoriescontainedatleastsomeusefulandinterestinginformationaboutprogram

impacts,theylackeddetailaboutthechangesexperiencedanddidnotusetheMSC

format(assetoutintheDaviesandDartmanual)verywell.However,thereview

identifiedvaluableexamplesofMSCstoriesthatwerelaterincludedindetailedMSC

manualswhichdrewonlearningsfromthereview.Theyclearlyexplainedeachstepin

thetechniqueandincludedinformationandtipsthatwererelevanttoAC4SCandthe

collectionofstoriesabouttheradioprogramsbeingevaluated.

23

Capacitybuildingacrossthewholeorganisationsandwithstakeholders

TheactiveparticipationofawiderangeofpeopleinECDisencouragedbythecreative

andflexibleuseofactionlearning-basedprocessesthataimtobuildthecapacityof

wholeorganisations,alongwiththeirpartnersandstakeholders,includingcommunity

members.Thisstrategycanhelptocushiontheimpactofstaffturnover(Gibbsetal.,

2009),akeyproblemindevelopmentorganisationsandanissuethataffectedAC4SC.

Akeyinfluenceherewastheapproachtoorganisationalcapacitydevelopment

detailedinHortonetal.(2003).Theysuggestthat,ratherthanfocussingonbuilding

thecapacitiesofindividualsandpartsofanorganisation,asintraditionalapproaches,

itismoreeffectivetofocusonbuildingthecapacityoftheorganisationasawholeand

toencouragetheactiveparticipationofabroadrangeofstaffandstakeholdersinthe

process.ThisprocessissimilartotheholisticapproachtoECDthatweadvocate.

Asfaraswecould,weusedthisapproachinAC4SC,includingadoptingapartnership

approachbyencouragingEANstafftoshareresponsibilityforfacilitationofworkshops

andmeetings,andorganisingmeetingsthatengagedvariousstakeholdersinthe

project.Wealsocollaborativelydevelopedmanualsandotherinformationmade

availableviatheprojectwebsite.ThishelpedtheM&Eteamtotransfertheir

knowledgetotheCRs,andtootherstaffofEANandlocaldevelopmentorganisations.

24

Ourmeta-evaluationofAC4SCindicatedthatinternaltrainingbytheM&Eteam

conductedaspartofAC4SChelpedtobuildM&Ecapacitiesmorebroadlyinthe

organisationandtoincreaseappreciationofthevalueandimportanceofevaluationto

theongoingprogramdevelopmentandimprovementprocess.

Focussingonthecommunicativeandrelationalaspectsoftheprocess

AsHortonetal.(2003:56)pointout,organisationalcapacitybuildingisaprocessthat

‘evolvesoveranumberofyears[and]...thedevelopmentandmaintenanceofgood

workingrelationshipsbetweenthevariouspartiesinvolvedinacapacitydevelopment

effortiscrucialtoitsoverallsuccess’.Thisstressesthecommunicativeandrelational

aspectsofevaluationand,asinC4D,effectivedialogueandinteractionisakeyfeature

oftheprocess.

AtthestartofAC4SCweidentifiedaneedtodevelopmoreeffectivecommunication,

collaborationandfeedbacksystemswithinEANandbetweenEANanditsstakeholders.

TheM&Eteamsawtheprogramproductionteamasresistanttochangingexisting

M&Esystemsandreluctanttotakeaccountofnegativefeedbackontheirprograms.

However,thischangedovertimeaftertheM&Eandprogramproductionteamsbegan

25

meetingmoreregularlyandM&Ereportsimprovedandbecamemoreusefultothe

programproductionteams,whostartedusingthisdatatoimprovetheirprograms.

AnimportantoutcomeofAC4SCwasthatcooperation,communication,dialogueand

interactionbetweentheM&Eteamandthecontentproductionteamimprovedover

time,asreportedthoughtheregularmeetingsheldbetweentheEANM&Eteamand

Australianteam4,andthroughfeedbackprovidedthroughmethodssuchas

questionnairesandinterviews.Asaresult,thecontentteamgraduallydevelopedmore

trustandconfidenceintheM&Eteamtoprovide‘reliable’data.Inaddition,

workshopsheldin2009thatbroughttheCRsandthecontentteamstogetherforthe

firsttimehelpedthemlearnfromeachotherandworkbetterasateam.

Embeddingevaluationintoeverypartoftheprogramdevelopmentcycle

OneofthemostambitiousaimsofECDistheintegrationofevaluationintothewhole

programdevelopmentandimplementationcyclefromtheconception,design,and

planningstages.Itinvolvesadiversityofstaff,stakeholdersandcommunitymembers

takingresponsibilityforresearchandevaluationactivities.Thisprocessisseenas

helpingtodevelopthewiderangeofevaluationcapacitiesthatarerequiredinthis

approach.DescribingtherealisticcomponentoftheirframeworkforevaluatingC4D,

26

LennieandTacchi(2013:36)suggeststhatthismeansthatevaluationbecomes‘a

responsiveandintegralpartoftheiterativeprocessofdeveloping,implementing,

improvingandadjustingC4Dinitiatives’.Theyadvisethatthisinvolvesusingan

approachthatis‘notrushed,andallowingdialoguetobegintheprocess’andthatin

consultationwitharangeofparticipants,theprocesswouldalsoinclude‘developing

flexibleandrealisticplansandtimeframesforthewholeevaluationprocess,usingan

organicapproachthatisresponsivetounfoldingdevelopments’(LennieandTacchi,

2013:36).

TheAC4SCmethodologyaimedtofacilitatetheparticipationofEANstaffandprimary

stakeholdersinallaspectsoftheevaluationprocessandinvolvedcarefulplanningof

M&Eworkandongoingadjustmentoftheseplansasprogramobjectiveschanged.It

includedtheuseofparticipatoryandmixedmethodsresearchandevaluationtools

andtechniques,andtriangulationofdatatoincreaserigour.Thebasicprocessesof

thismethodologyincluded‘listeningtoaudiences,learningfromthisknowledge,

systematicallyprocessingitandfeedingitbackintotheorganizationanditspractices

inanongoingcycle’(LennieandTacchi,2013:40-41).Atthe2013followupmeeting,

EANstaffreportedthattheirradioprogramswouldnotsucceediftherewasnoM&E

mechanismtoprovideregularfeedbackandthatthishadhelpedtheirprograms‘grow

evenfurtherandstronger’.

27

Developingacultureoflearningwithinanorganisation

Learningorganisationsregularlycriticallyreflectontheirsystems,processes,internal

andexternalrelationshipsandfeedbacksystems,andidentifywaystheycanbe

improved.Theseprocessesareimportanttomoreeffectiveandsustainable

developmentpractices.BehrensandKelly(2008:44)pointoutthat‘inalearning

organisationparadigm,evaluationbecomespartofthechangeeffort’.Labinetal.

(2012:307)notethatECDhasattractedtheinterestofevaluators‘committedto

increasingstakeholderunderstandingofevaluationandbuildingevaluationculture

andpracticeinorganizations’.PreskillandBoyle(2008:453)suggestthatif

organisationssupportandencourage‘organizationallearningcapacity’itismorelikely

thatECDactivitieswillbesuccessful.Thesupportofmanagersandleadersisvitalto

thedevelopmentoflearningorganisationsandtheyneedtobeseenasstrongmodels

forlearning(Forssetal.,2006;HooleandPatterson,2008;Taut,2007).

InAC4SCwefounditimportanttoobtainthesupportofallmanagementstaffinEAN,

toengagetheminkeyactivitieswheneverpossible,andtomaintainregular

communicationwiththemaboutprogresswiththeproject.Analysisofinterviewswith

EANstafffoundthattheconceptofalearningorganisationanditsassociatedpractices

wasseenasparticularlyimportanttoEqualAccessandotherC4Dorganisations.

28

SeniormanagementstaffmemberRameshindicatedhisstrongsupportforthisgoal

whenhestated:Asacommunicationorganisation,wehavetobealearning

organisation,that’sevenmoreimportant.EANintervieweesidentifiedawiderangeof

featuresofalearningorganisationwhicharesimilartothoseintheliterature

(Ortenblad,2013;Raeside,2011;Pearson,2011),includingbeingopentofeedback

fromarangeofsources,beingopentotalkingabout‘weaknesses’and‘mistakes’,

increasingtheappreciationofM&Eacrosstheorganisation,anddevelopingand

adjustingsystems,processesandknowledgethathelpsthemtocontinuouslyimprove.

RameshsawthemoreopenlycriticalapproachthatEANtookassomethingthatother

organisationsdidnotdobecauseofthepressureto‘satisfythedonor’.Devraj,a

contentproductionmanager,alsocommentedontheneedtoencouragestafftobe

‘critical’and‘analytical’aspartofthelearningprocess.Somestaffwhowere

interviewedin2009thoughtthatEANwasalreadymovingtowardsbecominga

learningorganisation.TheythoughtthiswasindicatedbyM&Ebeingconsideredasan

integralpartoftheorganisationandhavingstaffwhounderstooditsvalueandwere

activelyengagedinresearchandM&E.Inaddition,somecontentproductionstaff

indicatedthattheynowunderstoodthe‘learningbydoing’,continuousimprovement

approachofAC4SC.ParticipationinAC4SCwasseenasanimportantfactorinthese

changes.TwoM&Eteammembersthoughttheprojecthadbeeneffectiveinbuildinga

29

researchculturewithinEANwhileotherintervieweesreportedthattheM&Eteam

regularlyshareditsknowledgeandresearchfindingswithotherstaff.

Incorporatinglocal/indigenousknowledge,creativityandideas

Anotherimportantfeatureofthisapproachisthatitseekstotapintoandincorporate

local/indigenousknowledge,creativityandideas,andsocialchangeaspirationsand

needsintoECDefforts.Italsoaimstoempowerlocalstaffandcommunitiessothat

theirknowledge,ideasandlearningscanbeeffectivelyutilisedandactedupon(Hay,

2010;Raeside,2011).

Thedevelopmentofflexible,community-basedresearchandevaluationapproaches

andmethodologiesandthelonghistoryoftheuseofPARinthedevelopmentfield

highlightstheneedtolooktolocalknowledge,ideasandinnovationinorderto

developappropriate,effectiveandinnovativeevaluationapproachesandmethods.

Hay(2010:229)proposesthat‘Insteadoflookingtothenorthforcurriculumand

methods,[evaluation]fieldbuildingentailsexperimentationandindigenous

innovation,buildingonthebestideasavailablebutcreatingsomethingbetter’.

30

Carden(2007:53)makesthecasefordevelopmentevaluationbeing‘bestdoneby

locallybasedresearchersandorganizationswhoknowthecultureandcontext...and

havearesponsibilitytobuildcapacitytouseresearchindecision-makinginlocal

institutions,governmental,corporateornon-governmental’.Thisindicatesaneedto

betterappreciatelocallydevelopedlearningandevaluationmethodsandmore

appropriateandeffectiveevaluationmethodologiesthathavebeendevelopedand

testedwithpeopleinthedevelopmentcontext.Overfouryearsweworkedclosely

withEANtocollaborativelydevelop,testandrefinethemethodologyandeach

component.ThisresultedintheproductionoftheEqualAccessParticipatory

MonitoringandEvaluationToolkit(Lennieetal.,2011),basedontheexperiences,

learningsandcriticalreflectionsofEANstaff.

TheEqualAccesstoolkitincludesamoduleon‘CriticalListeningandFeedback

Sessions’(CLFS),whichwasinitiatedanddevelopedbyBikashKoiralawithinputfrom

othersinvolvedinAC4SC.Theideaofeachsessionistorandomlychooseanyepisode

ofabroadcastedradioprogramandaskallEANstafftolistencriticallyandparticipate

inafeedbacksession.Thisprocesshasencouragedprogramteammembersto

considertheirradioprogramsmorecriticallyandtocontinuallymakeimprovements.

Bikashrecentlyadvisedthat‘CLFShasbeenincorporatedintomostofEAN'sprojects

tomonitorradioprogramsatthecentreandcommunitylevels.Ithashelpedustoget

31

criticalinsightsinstantlyandhasalsobuiltthecapacityofourFMradiostation

partnerstomonitortheirradioprograms.Soit’slikeknowledgetransfer’.

Inaddition,EANhasincreasinglyincorporatedtheuseofnewICTsinitsM&Epractices,

includingdrawingonSMSandInteractiveVoiceRecordermessagesfromlistenersand

usingsmartphonestoreportoncasestudies.

Buildingawiderangeofskills,knowledge,attitudes,valuesandawareness

Awiderangeofskills,knowledge,attitudes,valuesandawarenessarerequiredforthe

effectiveevaluationofdevelopmentprogramsandinitiatives.Effectivelyusing

participatoryandsystems-orientedevaluationapproachesandfacilitatingbetteruse

ofevaluationsrequires‘peopleskills’(Patton,2010:49),includingskillsinrelationship

buildingandinterpersonalcommunication.Awiderangeofskillsareneededto

successfullyundertakeparticipatoryresearchandevaluation(Hearnetal.,2009;Taut,

2007).Aswellastechnicalskills,theyinclude:‘strongskillsinfacilitation,aswellas

humility,respectforothersandtheabilitytolisten’(Narayan,1993,citedinBoyle,

1999:143).Otherskillsinclude:‘responsivenesstouserneeds...acceptanceofdiverse

views,[andthe]abilitytoestablishrapportandtrust’(Green,1988,citedinTaut,2007:

32

49).Highlevelconflictmanagementandfacilitationskillsarealsoneededwhen

stakeholdershavecontradictoryperspectivesorunequalpower.

CapacitybuildingactivitiesconductedduringAC4SCaimedtodevelopawiderangeof

facilitation,communicationandpeopleskills.Forexample,workshopswithavarietyof

EANstaffheldinSeptember2007includedsmallgroupspracticingpowerfullistening

andothercommunication-relatedskills.Analysisoffeedbackquestionnairesfound

thatkeyoutcomesfromtheseactivitiesincludedimproved‘teambuilding’,‘team

spirit’andcommunication,andappreciationoftheneedfora‘cultureofsharing’

amongEANteammembers.Inaddition,duringfollowupresearchin2012,someof

theCRscommentedontheneedtolearnhowtobuildtrust,notingthatthiswasan

importantskill.

AcriticalapproachtoECDthatacknowledgeschallengesandissues

Thisapproachtakesacriticalperspectivethatacknowledgesthemanychallenges,

tensionsandissuesthatcanhindertheeffectivenessandsustainabilityofECDinthe

developmentcontext.ThoseidentifiedbyLennieandTacchi(2013:100-104)include:

● Contextualfactorsinpoor,politicallyunstable,developingcountries

33

● PowerrelationsinECDprojects

● ThecomplexityofevaluatingC4D

● Attitudestoevaluationamongdonors,C4DorganisationsandNGOs

● Maintainingandsustainingevaluationcapacity

● FacilitatingwideparticipationinevaluationofC4D

● ThewiderangeofskillsrequiredinevaluatingC4D.

AtregularpointsinAC4SC,EANstaffwereencouragedtocriticallyreflectonthe

projectandprovidefeedbackonwhatwasworkingwellandlesswell,andtheimpacts

oftheprojectonthemselvesandEAN.Mostofthestaffprovidedpositivefeedbackon

theinitialcapacitybuildingactivities.However,inalaterreviewofAC4SCbyanEqual

AccessInternationalmanager,feedbackwasreceivedthatthelevelofcomplexityof

themethodsandimpactassessmentframeworkwasleadingtoconfusionandfewstaff

couldclearlyarticulatetheproject’saimsandobjectives.Therewasapreferencefora

methodologythatwas‘muchsimplerandpractical’.AnM&Eteammember

commented:‘Fromthestart,everythingisnew,newmethods,newfeedback,new

ideas....wedecideacertainthingandtheacademicswillcomewithanotherissueor

ideaorthingweneedtodo’.Whattheyneededwas‘morepracticalguidancetoshow

usthebridgebetweentheacademicandthepractical’(Tacchietal.,2013:153).This

highlightsakeychallengeforprojectssuchasAC4SCwhichhavebothacademicand

34

practicalaims:‘theneedtobalancetheseaimsinwaysthatreduceconfusionand

feelingsofbeingoverwhelmedbytoomanynewideasandmethodsatonce’(Tacchiet

al.,2013:153).Thismayrequirespendingmoretimeintheinitialplanningphaseon

ensuringthattheECDobjectivesandprocessiscleartoeveryoneinvolvedandnottoo

ambitionsorunrealisticinitsscope.Therolesandresponsibilitiesofeveryoneinvolved

alsoneedtobeveryclear.

Ongoingmeta-evaluationofECDactivities

Recentstudiessuggestthatmeta-evaluationcanbevaluableindevelopingnew

approachestoevaluation,buildingevaluationcapacities,andenhancingorganisational

learning(Hanssenetal.,2008;UusikylaandVirtanen,2000).Animportantcomponent

oftheholisticECDapproachisusinganongoingmeta-evaluationprocessto

continuallyimproveECDactivitiesandincreasetheirsustainabilityandsuccess.

Ourongoing,rigorousmeta-evaluationofAC4SCassessedtheeffectivenessofthis

approach,includingforbuildinganevaluationcultureandimprovedM&E,

communicationandfeedbacksystemswithinEANasawhole.Keyobjectivesofthe

meta-evaluationincluded:continuousdevelopment,adaptationandimprovementof

theimpactassessmentmethodology,M&Esystemsandprocessandotherproject

35

activities;identificationofprojectimpacts;andcapacitybuildingincriticalreflection

andreview.Thismeta-evaluationwasessentialtounderstandingvariousconstraints

relatedtotheorganisationalcontextthataffectedthesuccessoftheprojectandthe

developmentofimprovedM&EsystemsandcapacitieswithinEAN(Lennieetal.,

2012).

Strategiesforaneffectiveandsustainableholistic,learning-based

approachtoevaluationcapacitydevelopment

Wehaveidentifiedanumberofprinciplesandstrategiesforaneffectiveand

sustainableholistic,learning-centredapproachtoECDattheorganisationaland

communitylevels:

● Assessanorganisations’readinessforlearningandchangeanditsexistingM&E

capacities,systemsandprocesses.

● Understandtheorganisationalculture,dynamicsandwidercontext;takethisinto

accountinthedesignandimplementationofECD.

● DrawonlocalinnovationandexperimentationintheECDprocessandencourage

self-organisationandcreativerisk-taking.

36

● Fosteralearningorganisationandanevaluationculturethroughthesupportand

influenceofleadersandmanagers.

● Embedevaluationintothewholeprogramdevelopmentcycle.

● DesignECDactivitiesthatareflexibleandopentocontinuousadaptationand

revision.

● PayattentiontothecommunicativeandrelationaldimensionsofECD,anddevelop

relationshipsbasedonmutualtrust,knowledgesharing,opencommunicationand

feedbacksystems.

● Empowerlocalstaffandcommunitiestoshareandactontheknowledgeand

understandingsgainedfromevaluationtobetterutilisetheresultsofparticipatory

evaluationandresearch.

● EngageinregularcriticalreflectionofECDactivitiestocontinuallyimprovethem.

Aswehaveindicated,theholisticECDapproachtakesmoretimeandresourcesthan

standardtraining-orientedapproaches.Thefunding,resourcesandsupportrequired

forlong-termECDcanbedifficultfordevelopmentorganisationstofind.Thisproblem

wasparticularlyacuteforAC4SCandEAN;theGlobalFinancialCrisis(GFC)thatswept

theworldin2008occurredduringtheprojectandcontinuedtohaveverysignificant

impactsontheabilityoftheorganisationtosustainitsactivities.Throughoutthelatter

halfoftheprojecttherewasconstantpressuretodomorewithless.However,on

37

reflectionweconsiderthistohavebeenlargelybeneficial,becausethesesignificant

financialconstraintsforcedeveryoneinvolvedtoconsiderhowtheholisticECD

approachcouldbemadesustainablefollowingtheendoftheproject’sfour-year

funding.

EANstaffwhowereinterviewedin2009werethereforeaskedtosuggestideasabout

howtomaketheirnewM&EsystemsandtheAC4SCapproachmoresustainable.They

putforwardarangeofusefulideasandstrategies,including:

● EmbeddingandinstitutionalisingthewholeAC4SCprocessintoEANbybuilding

thisinfromthestartofplanningprograms.Thisrequiresinvolvingstakeholders

moreintheprocess,continuingtobuildcapacityinM&E,andvaluingdatafrom

fieldstaff.

● IncludingdetailsoftheAC4SCmethodologyinproposalstodonorstohelpthem

obtainongoingfunding,giventhatthismethodologywasseenassomething

unique,thatgaveEANa‘competitiveadvantage’.EANhascontinuedtorefertothe

toolkitdevelopedduringAC4SCinalmosteveryfundingapplication.

● ExpandingtheCRnetworkandtheM&Eteamandcreatingmorewaystomakethe

CRnetworkuseful,suchasusingCRsasaresourceforpartnerorganisationsonce

theyreachedahigherskilllevel.

38

● OfferingtrainingintheAC4SCprocesstootherorganisationsandturningEANinto

a‘resourcecentre’forPM&EinKathmandu.

SincetheAC4SCprojectendedEANhasbeenregularlyaskedtoprovidetraininginthe

MSCtechniquebyUNICEFandotherdevelopmentorganisationsinKathmandu.This

hashelpedM&EstafftomaintainkeyevaluationcapacitiesdevelopedduringAC4SC

andtosharetheirexpertisewithothers.During2012EANfacedseriousproblemswith

obtainingfundingforitsprojects,resultinginalossofkeyM&Estaffandthe

disbandingoftheCRnetwork.However,bylate2012EANhadsuccessfullyobtained

fundingforamajorfive-yearradioproject,whichiscurrentlybeingimplementedby

PACT.ThisprojecthasitsownseparateM&Esectionandcurrentlyemploys17

communityactionresearchers(CARs),astheyarenowcalled,insixdistrictsinNepal.

TheCRmanualintheAC4SCtoolkitwasusedtotrainthenewCARs.AformerCR

helpedtoconductthistrainingandnowworksforthePACTproject.Donorsarealso

providingEANwithmorefundingforM&E.

39

Conclusion

PreskillandBoyle(2008:457)believethatECD‘representsthenextevolutionofthe

evaluationprofessionand,assuch,hasthepotentialfortransformingthefieldinways

onlyimagined’.However,inordertorealisethistransformativepotentialitisvitalthat

werecognisepotentialbarrierstosuccessfulECDandbuildeffectivecounter-measures

intoimplementationstrategiesfromtheoutset.Aflexible,emergentapproachis

requiredthatisopentochangeandcontinuousadjustmentbasedonregularfeedback

andcritiquefromthoseinvolved.ThecomplexityofthechallengesfacedintheAC4SC

projectanditslength(fouryearswithsubsequentpost-projectevaluationsoverthe

followingtwoyears)providedanunprecedentedopportunitytoexplorethesebarriers

anddevisesolutionsthatweretestedinpracticeandsubjectedtorigorousmeta-

evaluation.Drawingonthisevidence,wehaveidentifiedanumberofprinciplesand

requirementsforthesuccessfulimplementationofaholistic,learning-centred

approachtoECD.Theseareessentialifthetransformativepotentialofthisapproach

istoberealisedbydevelopmentorganisationsthatmaynothavetheluxuryof

experimentingwiththisapproachduetoseverefundingandresourceconstraints.

ThemostimportantconclusionwedrawfromthisexperienceisthatECDcannotbe

implementedasaseparate‘tool’or‘technique’,butmustbefoundedonwiderand

40

morefundamentalchangestoorganisationalculture.Inparticular,theabsencewithin

anorganisationofopennesstolearning,especiallylearningfoundedonthecritical

analysisofnorms,communicationsystemsandrelationshipsbetweenthoseinvolved,

canunderminethedevelopmentofbothholisticperspectivesonproblemsthatECDis

intendedtoovercomeandtheeffectiveparticipationofallstakeholders.Thelatterisa

keyrequirementforimplementinganypracticalactionsasanoutcomeofthe

evaluationprocess.Wesuggestthattheinclusionofrigorousmeta-evaluativeactivity

iscrucialtoECD.Thisisbecauseitenablesorganisationstoidentifywhenandhowthe

absenceofthesefoundationalcapacitiesforlearningandcriticalself-evaluationmay,

ironicallypreventthemfromusingevaluationasameanstocontinuouslyimprove

developmentprogramsandincreasetheirimpactinwaysthatbettermeetcommunity

needs.Ifthesebarriersandissuescanbeovercome,ourresearchsuggeststhatthe

holistic,learning-basedapproachtoECD,combinedwithparticipatoryformsof

developmentandevaluation,canmakeanimportantcontributiontoincreasingthe

sustainabilityofdevelopmentorganisationsand,inthelongrun,tomoreeffectiveand

sustainabledevelopment.

41

Acknowledgements

WewouldliketothankandacknowledgeeveryonewhoparticipatedintheAC4SC

projectandprovidedfeedbackandinputintoit.Wealsoacknowledgethe

contributionsofAndrewSkusefromtheUniversityofAdelaide.

Funding

TheAC4SCprojectwasfundedbytheAustralianResearchCouncil(grantnumber

LP0775252),USAID/NepalandEqualAccess.

Notes

1. Weusetheterm‘evaluationcapacitydevelopment’(ECD)sincethistermismost

oftenusedinthefieldofdevelopmentevaluation,however‘evaluationcapacity

building’(ECB)isalsousedinmuchoftheliteraturewedrawuponinthispaper.

Whiletheliteratureoftenreferstothesetermsinterchangeably,Carter(2013)

notesthatsomeexpertsdistinguishbetweenthem.InthispaperweuseECDeven

whenreferringtoworkthatusesthetermECB,exceptwhenwearedirectly

quoting,inwhichcaseECBisused.

2. AllnamesofEANstaffwhowereinterviewedhavebeenchanged.

42

3. ElevenCRsweretrainedbutonlyeightofthemprovidedsignificantamountsof

data.

4. Wemetregularlythroughphoneand/orSkypemeetings,aswellaslessregular

facetofacemeetingsinNepal.Theregularityofphone/Skypemeetingsvaried

acrosstheprojecttimespan,fromquarterly,tobi-weekly,dependingonproject

needsandactivities.

References

Armytage,L(2011)Evaluatingaid:anadolescentdomainofpractice.Evaluation17(3):

261-276.

Atkinson,D,Wilson,MandAvula,D(2005)Aparticipatoryapproachtobuilding

capacityoftreatmentprogramstoengageinevaluation.EvaluationandProgram

Planning28:329-334.

BaTall,O(2009)Theroleofnational,regionalandinternationalevaluation

organisationsinstrengtheningcountry-ledmonitoringandevaluationsystems,In:

SegoneM(ed)Country-ledMonitoringandEvaluationSystems.BetterEvidence,Better

Policies,BetterDevelopmentResults.Geneva:UNICEF,pp.119-134.

43

BayleyS,BenYounesChaouchS,BilinskyP,BrownA,CrailJ,HsuL-Netal.(2012)An

internationalworkshopagreementonevaluationcapacitydevelopment,17-21

October2011,EvaluationCapacityDevelopmentGroup,Geneva,Switzerland.URL

(consulted10September2014):http://www.ecdg.net/wp-

content/uploads/2012/03/IWA-on-ECD6.pdf

BehrensTandKellyT(2008)Payingthepiper:foundationevaluationcapacitycalls

thetune.In:CarmanJandFredericksK(eds),Nonprofitsandevaluation.New

DirectionsforEvaluation119:37–50.

BoyleR(1999)Professionalisingtheevaluationfunction:humanresourcedevelopment

andthebuildingofevaluationcapacity.In:BoyleRandLemaireD(eds)Building

EffectiveEvaluationCapacity.LessonsFromPractice.NewBrunswick:Transaction

Publishers,pp.135-151.

CardenF(2007)Therealevaluationgap.Alliance12(4):53-54.

CarterB(2013)Evaluationcapacitydevelopment.HelpdeskResearchReport,GSDRC,

17September.URL(consulted10September2014):

http://www.gsdrc.org/docs/open/HDQ996.pdf

44

CousinsJB,WhitmoreEandShulhaL(2013)Argumentsforacommonsetofprinciples

forcollaborativeinquiryinevaluation. AmericanJournalofEvaluation34(1):7-22.

CracknellBE(2000)EvaluatingDevelopmentAid:Issues,ProblemsandSolutions.

ThousandOaks:Sage.

DaviesRandDartJ(2005)The'MostSignificantChange'(MSC)Technique.AGuideto

itsUse.(consulted10September2014):

http://www.mande.co.uk/docs/MSCGuide.pdf

DabelsteinN(2003)Evaluationcapacitydevelopment:lessonslearned.Evaluation9(3):

365-369.

DjamankulovaK,TemirovaNandSobirdjonovaM(2010)Usingactionlearningsets

methodologyinanNGOcapacitybuildingprogramme.PraxisNoteNo.53,INTRAC.

FitzpatrickJL,SandersJRandWorthenBR(2012)ProgramEvaluation:Alternative

ApproachesandPracticalGuidelines,4thed.NewJersey:Pearson.

ForssK,KruseS,TautSandTendenE(2006)Chasingaghost?Anessayon

participatoryevaluationandcapacitydevelopment.Evaluation12(1):128-144.

45

Gibbs DA, Hawkins SR, Clinton-Sherrod AM and Noonan RK (2009) Empowering

programswithevaluationtechnicalassistance.Outcomesandlessonslearned.Health

PromotionPractice10(1):385-445.

Hanssen C, Lawrenz F and Dunet D (2008) Concurrent meta-evaluation. A critique.

AmericanJournalofEvaluation29(4):572-582.

HayK(2010)EvaluationfieldbuildinginSouthAsia:reflections,anecdotes,and

questions.AmericanJournalofEvaluation31(2):222-231.

HearnG,TacchiJ,FothMandLennieJ(2009)ActionResearchandNewMedia:

Concepts,MethodsandCases.Cresskill,NJ:HamptonPress.

HooleEandPattersonT(2008)Voicesfromthefield:evaluationaspartofalearning

culture.In:CarmanJandFredricksK(eds)NonprofitsandEvaluation.NewDirections

forEvaluation119:93-113.

HortonD,AlexakiA,Bennett-LarteyS,BriceKN,CampilanD,CardenFetal.(2003)

EvaluatingCapacityDevelopment:ExperiencesfromResearchandDevelopment

OrganizationsAroundtheWorld.TheHague:InternationalServiceforNational

AgriculturalResearch.

46

KuzminA(2010)Useofevaluationtraininginevaluationcapacitybuilding.In:Segone

M(ed)FromPolicytoResults.DevelopingCapacitiesforCountryMonitoringand

EvaluationSystems.NewYork:UNICEF,pp.240-251.

LabinS,DuffyJ,MeyersD,WandersmanAandLeseseneC(2012)Aresearchsynthesis

oftheevaluationcapacitybuildingliterature.AmericanJournalofEvaluation33(3):

307-338.

LennieJandTacchiJ(2011)Researching,monitoringandevaluatingcommunication

fordevelopment:Trends,challengesandapproaches,Reportonaliteraturereviewand

consultationswithExpertReferenceGroupandUNFocalPointsonC4D.Preparedfor

theUnitedNationsInter-agencyGrouponCommunicationforDevelopment,New

York:UNICEFURL(consulted10September2014):

http://www.unicef.org/cbsc/files/RME-RP-

Evaluating_C4D_Trends_Challenges__Approaches_Final-2011.pdf

Lennie, J. and Tacchi, J. (2013) Evaluating Communication for Development: A

FrameworkforSocialChange.Abingdon:Routledge.

LennieJandTacchiJ(2014)Bridgingthedividebetweenupwardaccountabilityand

learning-basedapproachestodevelopmentevaluation.Strategiesforanenabling

environment.EvaluationJournalofAustralasia,14(1):12-23.

47

LennieJ,TacchiJ,KoiralaB,WilmoreMandSkuseA(2011)EqualAccessParticipatory

MonitoringandEvaluationToolkit,QueenslandUniversityofTechnology,Universityof

Adelaide,andEqualAccessNepal.URL(consulted10September2014):

http://betterevaluation.org/toolkits/equal_access_participatory_monitoring.

LennieJ,TacchiJandWilmoreM(2012)Meta-evaluationtoimprovelearning,

evaluationcapacitydevelopmentandsustainability:findingsfromaparticipatory

evaluationprojectinNepal.SouthAsianJournalofEvaluationinPractice1(1):13-28.

MorganP(2013)Evaluatingcapacitydevelopment.In:DonaldsonS,AzzamTand

ConnerR(eds)EmergingPracticesinInternationalDevelopmentEvaluation.Charlotte,

NC:InformationAgePublishing,pp.75-104.

NaccarellaL,PirkisJ,KohnF,Morley,B,BurgessPandBlashkiG(2007)Building

evaluationcapacity:definitionalandpracticalimplicationsfromanAustraliancase

study.EvaluationandProgramPlanning30(3):231-236.

Napp D, Gibbs D, Jolly D, Westover B and Uhl G (2002) Evaluation barriers and

facilitators among community-based HIV prevention programs. AIDS Education and

Prevention14Supp.A:38–48.

48

OfirZandKumarAK(2013)Evaluationindevelopingcountries.Whatmakesit

different?’In:DonaldsonS,AzzamTandConnerR(eds),EmergingPracticesin

InternationalDevelopmentEvaluation.InformationAgePublishing,Charlotte,NC,pp.

11-24.

OrtenbladA(2013)Whatdowemeanby‘learningorganization’?In:OrtenbladA(ed)

HandbookofResearchontheLearningOrganization:AdaptationandContext.

Cheltenham:EdwardElgar,pp.22-34.

PattonMQ(2010)Futuretrendsinevaluation.In:SegoneM(ed)FromPolicyto

Results.DevelopingCapacitiesforCountryMonitoringandEvaluationSystems.New

York:UNICEF,pp.44-57.

Patton,MQ(2011),DevelopmentalEvaluation:ApplyingComplexityConceptsto

EnhanceInnovationandUse.GuilfordPress:NewYork.

PearsonJ(2011)CreativeCapacityDevelopment:LearningtoAdaptinDevelopment

Practice.SterlingVA:KumarianPress.

Preskill,H(2010)Exploringeffectivestrategiesforfacilitatingevaluationcapacity

building.In:SegoneM(ed)FromPolicytoResults.DevelopingCapacitiesforCountry

MonitoringandEvaluationSystems.NewYork:UNICEF,pp.224-238.

49

PreskillHandBoyleS(2008)Multidisciplinarymodelofevaluationcapacitybuilding.

AmericanJournalofEvaluation29(4):443-459.

RaesideA(2011)AreINGOsbraveenoughtobecomelearningorganisations?In:

AshleyH,KentonNandMilliganN(eds.)HowWidearetheRipples?FromLocal

ParticipationtoInternationalOrganisationalLearning,ParticipatoryLearningand

Action63:97-102.London:TheInternationalInstituteforEnvironmentand

Development.

RobinsonTandCousinsB(2004)Internalparticipatoryevaluationasanorganizational

learningsystem:alongitudinalcasestudy.StudiesinEducationalEvaluation30:1-22.

Schiavo-CampoS(2005)Buildingcountrycapacityformonitoringandevaluationinthe

publicsector:Selectedlessonsofinternationalexperience,EvaluationCapacity

Development,ECDWorkingPaperSeriesNo.13:TheWorldBankOperations

EvaluationDepartment.URL(consulted10September2014):

http://preval.org/files/2086.pdf

SegoneM(ed)(2009)Country-ledMonitoringandEvaluationSystems.BetterEvidence,

BetterPolicies,BetterDevelopmentResults.Geneva:UNICEF.

50

SternE,Stame,N,Mayne,J,Forss,K,Davies,RandBefani,B(2012)Broadeningthe

rangeofdesignsandmethodsforimpactevaluation.DFIDWorkingPaper38.London:

DFID.

Sonnichsen,R(1999)Buildingevaluationcapacitywithinorganisations.In:BoyleRand

LemaireD(eds)BuildingEffectiveEvaluationCapacity.LessonsfromPractice.New

BrunswickandLondon:TransactionPublishers,pp.53-73.

Tacchi,J(forthcoming)Ethnographicactionresearch:projectdevelopmentand

evaluation.In:Bradbury-Huang,H(ed.)HandbookofActionResearch:Participative

InquiryandPractice.Oxford:Sage.

TacchiJandLennieJ(2014)Aparticipatoryframeworkforresearchingandevaluating

communicationfordevelopmentandsocialchange.In:WilkinsKG,TufteTand

ObregonR(eds)TheHandbookonDevelopmentCommunicationandSocialChange.

Oxford:WileyBlackwell,pp.298-320.

TacchiJ,LennieJandWilmoreM(2013)Criticalreflectionsontheuseofparticipatory

methodologiestobuildevaluationcapacitiesininternationaldevelopment

organisations.In:GoffS(ed)FromTheoryToPractice;ContextinPraxis.Selected

Papersfromthe8thActionLearning,ActionResearchWorldCongressAustralia2010,

ActionLearningActionResearchAssociation,Toowong,Queensland,pp.150-160.

51

Tacchi J,Fildes J,MartinK,MulenahalliK,BaulchEandSkuseA (2007)Ethnographic

Action Research Training Handbook. URL (consulted 10 September 2014)

http://ear.findingavoice.org/

TautS(2007)Studyingself-evaluationcapacitybuildinginalargeinternational

developmentorganization.AmericanJournalofEvaluation28(1):45-59.

UusikylaPandVirtanenP(2000)Meta-evaluationasatoolforlearning.Acasestudyof

theEuropeanStructuralFundevaluationsinFinland.Evaluation6(1):50-65.

ValeryRandShakirS(2005)Evaluationcapacitybuildingandhumanitarian

organization.JournalofMultidisciplinaryEvaluation2(3):78-112.