Draft Monitoring and Evaluation Framework
Draft Monitoring and Evaluation Framework i
TABLE OF CONTENTSIntroduction 11.0 What does MAMPU aim to achieve and how? 12.0 How will MAMPU be monitored? 5 2.1 Partner Quarterly Progress Reporting 5 2.2 MAMPUSecretariatfieldmonitoringandverification 7 2.3 RoutinereflectionbyPartnersandtheMAMPUSecretariat 73.0 How will MAMPU be evaluated 8 3.1 KeyEvaluationQuestions(KEQs) 8 3.2 KeyEvaluationQuestion1:HowandtowhatextenthasMAMPUaffectedthe partnersandnetworks’capacitytoinfluencegovernmentreform? 9 3.3 KeyEvaluationQuestion2:Howandtowhatextenthavethepartnersand networksinfluencedgovernmentreforminrelationtotheneedsandpriorities ofpoorwomen? 12 3.4 KeyEvaluationQuestion3:HowandtowhatextenthasMAMPUcontributed toimprovedaccesstoessentialgovernmentservicesandprograms? 14 3.5 JudgingsuccessattheEnd-of-Programstage(Targets) 18 3.6 Typesofevaluation 194.0 How will the information reach decision - makers? 20
Draft Monitoring and Evaluation Framework 1
Introduction
ThisdocumentoutlinesaMonitoringandEvaluation(M&E)FrameworkfortheAustralia-IndonesiaPartnershipforGenderEqualityandWomen’sEmpowerment(MAMPUPhaseII).Thetargetaudi-enceforthisdocumentaremembersoftheMAMPUTechnicalCommittee(TC)andSteeringCom-mittee(SC).IthasbeendevelopedtoenablememberstoappraisetheproposedoverallapproachtoM&EforPhaseIIofMAMPU.ThisM&EFrameworkshouldbereadinconjunctionwiththeMAMPUPhaseIIStrategicForwardPlanandStandardOperatingProcedures(SOPs).ThefinalapprovedversionofthisM&EFrameworkwillbethebasisofarevisedM&EPlan,whichelaboratesMAMPU’sM&Esystemingreaterdetail.
Thisdocumentisstructuredinfoursections.Insection1itbrieflyexplainsthe‘changeprocess’thatneedstobemonitoredandevaluatedandsummariseswhatMAMPUexpectstoachieve.Section2outlineshowMAMPUwillroutinelymonitorimplementationprogress.Section3setsoutarrange-mentsforevaluation,includingKeyEvaluationQuestions,datatoaddressthese,andthebasisforjudgingsuccess.Finally,section4describesmechanismsforensuringthatfindingsfromtheseM&Eprocessesreacheskeydecision-makersintheMAMPUgovernancestructure.
1.0 What does MAMPU aim to achieve and how?
TheSubsidiaryAgreementbetweentheGovernmentsofIndonesiaandAustralianotesthattheul-timategoalofMAMPUistocontributeto“genderequalityandwomen’sempowermentinselectedareas in Indonesia.”Achieving thishigh levelgoalwillbe theresultof thecomplex interactionofwidersocio-cultural,politicalandeconomicforces,manyofwhichlieoutsidethedirectinfluenceofMAMPU.
Nevertheless,by2020MAMPUwillmakeacontributiontowardsthisgoalintwoways.Firstly,MAM-PU expects to have “improved access to essential government services and programs for poorwomenin target locations”.This is theEnd-of-Program-Outcome(EOPO).The ‘essentialgovern-mentservicesandprograms’referredto in thisEOPOstatementreflect thefiveMAMPUthemesdescribedintheSubsidiaryAgreement:
• Social protection programs, particularly publically-funded health insurance providedthroughtheNationalHealthInsuranceScheme(JaminanKesehatanNasional)adminis-teredbyBPJS(theme1);
• Workplaceprotections,particularlyhealthinsuranceforwomenhomeworkers(theme2);• Servicesthatimprovemigrationconditionsforwomenmigrantworkers(theme3);• Servicesthataddresswomen’ssexual,reproductivehealth,andnutritionalneeds(theme
4);and• Counsellingandsupportservicesthataddresstheneedswomenvictimsandsurvivorsof
violence(theme5).
Secondly, by 2020MAMPU expects to seepositivechange in the ‘voice’and ‘influence’ofwomenatmultiplelevels.Invillages,wom-enwill working collectively, expressing theirviewsinpublicandprivate(‘voice’),shapingdecision-making and influencing the alloca-tionofstateresources(‘influence’) forwiderbenefit, including improved access to ser-vices. By doing so they will be challengingnormsthatconstrainwhatissociallyaccept-ableforwomenandgirlstodo.Thesechangesin‘voice’and‘influence’areaprocessaswellasanexpectedoutcome.Asa‘process’theydescribeapathwaythroughwhichMAMPUimproveswomen’saccesstoservices.Theyarean ‘outcome’ofMAMPUin that theydescribeanexpectedendstatethatinitselfhasintrinsicvalue.Criticaltobothisaviewofpoorwomenasagents,notonlyusersofservicesprovidedbyothers.Thisisthe empowerment agenda that is central toMAMPU.
Draft Monitoring and Evaluation Framework2
How does capacity increase ‘voice’ at the grassroots?
In 2017 a qualitative study of eight local women’s groups established by MAMPU partners explored the effects of membership. The analytical framework centred on five types of empowerment ‘assets’. Applying this framework to women’s experiences as members shed light on how and in what circumstances individual and collective capacity leads to increased ‘voice’: “In terms of an empowerment pathway development of human assets, particularly confidence, self-belief, and as called by many interviewees, ‘courage’, appears to be a pre-requisite for other forms of empowerment. Members of each example of collective action studied reported a progression from growing confidence and knowledge (changes in human or individual assets), to speaking out and participating in or presenting to community forums (agency assets), and then some expectation that this will lead to either finance and resource assets or enabling assets, and these will reinforce each other.” These findings are consistent with a wide variety of evidence about women’s empowerment drawn from other contexts.
(MAMPU, (2017) Women’s Collective Action for Empowerment in Indonesia: A study of collective action initiated by partners of the MAMPU program. Yogyakarta: Migunani and MAMPU)
MAMPUisbasedontheideathatnetworksofselectedcivilsocietyorganisations–theMAMPUpart-ners–canplayapivotalroleinshapinggovernmentreformtobenefitpoorwomenonasignificantscale.TheprogramhaselectedtobuildontheworkoforganisationswithanestablishedtrackrecordofinfluencingreforminIndonesia.Thetheoryisthatwiththerightkindofsupportattherighttime,theseorganisationswillactmorecollectively,andincreasinglyinconcertwithalliesingovernment,parliament,andprivatesector.Atthesametime,partnerswillworkwithanddrawfromtheprioritiesandexperiencesofpoorwomeninvillagesacrossIndonesia.Bysupportingthesemultilevelpro-cesses,MAMPUexpectsmomentumforchangetogrow,influencinghowthegovernmentmakesandcarriesoutpolicies,improvingwomen’saccesstoessentialservicesonawidescale.
Figure 1: Outcomes and timeframes
Draft Monitoring and Evaluation Framework 3
By theconclusionof2017MAMPUexpects the resultsof thisprocess tobevisible in increased‘voice’ and ‘influence’. In the villages where MAMPU works, women will be more involved indecision-making and their ongoing participation will be acknowledged and enshrined in formalvillageregulations.At thedistrict,provincialandnational levels,MAMPU’scontributionwillbere-flectedinpolicydecisionsthatcanplausiblycontributetoimprovingwomen’saccesstoservicesonawidescaleby2020.Thisistheexpectedmedium-termoutcomeofMAMPU.
However, a series of preconditions are needed if these outcomes are to contribute to betterservicedelivery, improvedaccess to services, andempowerment by2020.Firstly, theymust beaccompaniedbypositivechangesintheself-belief,knowledgeandconfidenceofwomenwithwhomMAMPUworksatthevillagelevel.EvidencefromMAMPUaswellasinternationally,suggeststhattheseareimportantprerequisitesforvoiceandempowerment(seetextboxabove).Secondly,theauthorizingregulationsandpoliciesforreformsthataddresswomen’sprioritiesneedtobeinplaceatthenational,provincial,anddistrictormunicipalitylevels.Withoutthese,furtheractionsuchasal-locatingbudgetstowomen’sprioritiesisdifficult.Thirdly,resources(financialandhuman)needtobemobilizedandallocatedtowardstheparticularservice-relatedissueaddressedinthepolicy.Fourthly,governmentserviceprovidersneedsufficientcapacitytodeliverthepolicyintentattheservicelevel.Thesefourpreconditionsarenotenoughintheabsenceofafifth:evidentcommitmentandsupportfromacriticalmassofleaders–insidegovernmentandparliamentaswellasincommunities.
MAMPUiscautiousaboutgeneralizingacrossthediversityofcontextswheretheprogramaimsforchange.Theopportunitiesandconstraintswilldependtoagreatextentontheuniquesocial,politicalandculturaldynamicsineachvillage,districtandprovince.Nonetheless,togetherthesefiveprecon-ditions–invaryingconfigurations–representthe‘causalpackage’thatMAMPUbelievesisrequiredforpolicyinfluencetoleadtoimprovedaccesstoservicesandempowerment.
Increasing thecapacityand readinessofpartners forcollectiveaction– including their linkswithgovernmentandprivatesector– laysthefoundationforachievingmedium-termandEnd-Of-Pro-gram-Outcomes.MAMPUhelps toaccelerate thisbydevelopingpartnerorganisational capacity,bolstering the focus and intensity of collective action, and enabling them to expand their reachamongwomenat thegrassroots.Theprogramincentivizespartnerstowork innetworksthat linkup localandnationalorganisationsandprovides themwithgrant funds to test ideas inselectedlocationsacrossIndonesia.Alongsidethis,MAMPUlinkspartnerstotechnicalexpertise,highqual-ityevidence,andresources,toenablethemtobetterseizeopportunitiesthatemergeinthecontext.
AsMAMPUmoves intoasecondphase, itsroleasanactive ‘connector’and ‘bridgebuilder’be-tween partners, government agencies, and other strategically significant actors will increase.The new governance structure for MAMPU – which opens space for routine partner-nationalgovernmentinteraction–willbecriticaltothis.TheincreasedinvolvementofBappenasinaguidingandfacilitatingrolewillboosttheprogram’scapacitytolinkwithwiderreformsacrossthegovern-ment.
Draft Monitoring and Evaluation Framework4
Table 1: Summary program logic for MAMPU
Draft Monitoring and Evaluation Framework 5
2.0 How will MAMPU be monitored?
Thepathfrominfluencinggovernmentpolicytoimprovedaccesstoservicesisneitherstraightnorpredictable.Insuchcontextsitisespeciallyimportantthatimplementationteamshaverapidfeed-backtogaugeprogressandmaketimelyadjustments.MAMPU’smonitoringsystemwilladdressthisneed.
Broadly,therewillbethreecomplementarycomponents:(i)quarterlyprogressreportingandanaly-sis;(ii)fieldmonitoringandverification;and(iii)regularopportunitiesforstructuredreflectionandadaptation.
2.1 Partner Quarterly Progress Reporting
TheworkofpartnersisattheheartofMAMPU’schangeprocessandfrequentfeedbackontheirperformanceisessentialforresponsiveandadaptivemanagement.
Every3months,eachpartnerwillsubmitashort,structuredreporttoMAMPUthroughtheonlinereportingsystem,‘MANISKita’.ThequarterlyprogressreportwillcontaininformationanddatasothatpartnersandtheMAMPUSecretariatcananswer5keymonitoringquestions:
1. Did we do what we expected to do?Appliesa‘trafficlight’scaleandbriefnarrativetoproduceasnapshotofperformanceagainstannualworkplanovertheprevious3months.TheexistingtemplatewillbemodifiedtoensurealignmentwiththeGovernmentofIndone-sia’sBASTfinancialreportingobligations.
2. Were the costs in line with what we expected?Providesasummarizedpictureofex-penditureagainstbudgetforeachimmediateoutcomeintheannualworkplan.
3. What challenges and risks are affecting progress? Provides information to explaindivergence between planned and actual implementation, and identifies risks that haveemergedinthecontextduringtheprevious3months.
4. Are we reaching and engaging the right people and groups in sufficient numbers? Qualitativeandquantitativedata–includingsex-disaggregatedstatistics–enablingarapidassessment of whom andwhere partners are engaging, including other organisations,menandwomenatthevillagelevel,religiousandcommunityleadersandpolicymakersatmultiplelevelsofgovernment.
5. What changes and benefits are being experienced by direct participants and stake-holders?InformationonoutcomesincludingshortnarrativeofprogresstowardspartnerEnd-of-Project-Outcomes,progresstowardspolicyinfluence,andquantitativedataonse-lectedindicators.
Takentogether,thisinformationisdesignedtogenerateapictureofperformanceacrosskeydimen-sionsofastylizedprogramlogic,asillustratedinfigure1overleaf.
Draft Monitoring and Evaluation Framework6
InputsBudget
Resources
InputsBudget
Resources
InputsBudget
Resources
ImmediateOutcomes
ReachEngagement
ImmediateOutcomes
ReachEngagement
ImmediateOutcomes
ReachEngagement
IntermediateOutcomes
IntermediateOutcomes
IntermediateOutcomes
Partner Endof ProjectOutcomes
Partner Endof ProjectOutcomes
Partner Endof ProjectOutcomes
Partner 3Partner 2
Partner 1
Theme
Monitoring
4 & 5
Monitoring
1, 2 & 3 Dashboards that visualize databy partner, aggregated by
Figure 2: Using data from Quarterly Progress Reports to monitor performance by partner, theme, or portfolio
Keydata ineachreportarevisualizedautomatically inaseriesof interactive ‘dashboards’whichareaccessibletopartnersandtheMAMPUSecretariatthroughtheonlineMANISKitasystem.Thisassistsanalysisandhelpstomonitor trends inkey indicatorsat thethematic level,oracrossthewhole-of-MAMPUportfolio.
Draft Monitoring and Evaluation Framework 7
2.2 MAMPU Secretariat field monitoring and verification
Monitoring dan verifikasi lapangan akan menjadi komponen penting yang melengkapi sistempelaporankemajuan triwulanan formal ini.BagiMAMPU, ‘monitoring lapangan’diartikansebagaikegiatanditingkatdesaataukelurahan.Bersamadenganmitra,SekretariatMAMPUmelaksanakanrata-rata10kunjunganmonitoringlapanganpertriwulananuntukmengamatipelaksanaankegiatan,mendengarsecaralangsungdarilaki-lakidanperempuanyangterlibatdiakarrumput,sertaberin-teraksidenganparatokohmasyarakatsetempatdanaktorpemerintah.Salahsatutujuanpentingdarimonitoringlapanganadalahmentriangulasidanmemverifikasihasildanisudalamlaporanke-majuantriwulanan.Temuan-temuanpentingyangdidapatdariseluruhmonitoringlapanganmaupundatakuncikemudiandimasukkankedalamLaporanBack-to-OfficediSistemInformasiNasionalMAMPU(MANIS).HaliniakanmembantustafMonevuntukmelakukananalisistemuansecararu-tin,memperkuatkepercayaanterhadaphasilyangdilaporkan,sertamembantuSekretariatMAMPUuntukmelacakfrekuensidancakupanmonitoringlapangandiseluruhportofolio.
Figure 3: Field trip coverage dashboard, MAMPU National Informa-tion System
2.3 Routine reflection by Partners and the MAMPU Secretariat
OpportunitiestomakesenseofmonitoringinformationandplanfollowupactionswillbeessentialtoMAMPU’sapproach.TheMAMPUSecretariatwillfacilitatetwotypesofregularstructuredreflectionprocessduringPhaseII.
Firstly, theMAMPU Secretariat will facilitate regular 6-monthly reflection sessions (Participatory
Draft Monitoring and Evaluation Framework8
AnalysisandReflection)witheachpartnercoveringwhatworkedwell,whatwaschallenging,andwhatactionneedstobetaken–byboththepartnerandtheMAMPUSecretariat.Thesehalftofulldaysessionsaredesignedtobesimple,interactiveexercisesthatareopenandflexible.Facilitationwillmakeuseofavarietyofparticipatorytechniquesincludingranking,voting,andWorldCafétoelicit reflectionandencouragediscussion.Thisapproachacknowledgesthatwrittenreportingfol-lowingastructuredtemplatecanmissinterestingandvaluableinformation.Highlights,keyfindingsandagreedactionsfromeach6-monthlyreflectionarerecordedonMAMPU’sinternalManagementInformationSystemwheretheprocessandfollowupcanbemonitored.
Asecondtypeofroutinereflectionwill takeplaceevery3monthswithintheMAMPUSecretariat.FacilitatedbyinternalM&Estaff,thiswillbringtogetherfindingsfromarapidanalysisofquarterlyprogressreports,fieldmonitoring,andfinancialmonitoringtoconsiderprogressineachofMAMPU’sfivethematic‘hubs’.Usingamixofpresentationsandinteractivediscussion,thesewillaimtofostergreaterstrategiccoherencewithinandbetweenhubsandensureMAMPUisresponsivetoemergingdevelopments.
3.0 How will MAMPU be evaluated?
ThissectionexplainshowMAMPUwillbeevaluatedatkeypointsoverPhaseII.ItdescribestheKeyEvaluationQuestions(KEQs)thatneedtobeanswered,whichdataandinformationwillbeused,howjudgementswillbemade,andwhattypesofevaluationexercisewillbeundertaken.
3.1 Key Evaluation Questions (KEQs)
Allevaluativeactivitywilladdressasetof4KeyEvaluationQuestions(KEQs)thatlinktotheout-comesexpectedatkeytimesintheprogramlife.ProposedKEQsforMAMPUsetinthedesignare:
KEQ1. HowandtowhatextenthasMAMPUaffectedthepartnersandnetworks’ capacitytoinfluencegovernmentreform?
KEQ2. Howandtowhatextenthavethepartnersandnetworksinfluenced governmentreforminrelationtotheneedsandprioritiesofpoorwomen?
KEQ3. MAMPUEnd-of-Program-Outcome(EOPO):Howandtowhatextenthas MAMPUcontributedtoimprovedaccessforpoorwomentoessential governmentservicesandprograms?
KEQ4. WhatchangedinthecontextandhowdidMAMPUrespond?
AllfourKEQswillbeaddressedduringphaseIItogenerateacompletepictureofMAMPU’scon-tributiontotheEOPO.However,therewillaspecialfocusonKEQ3reflectingexpectedprogramachievementinthisareaby2020.
Draft Monitoring and Evaluation Framework 9
Contribution Analysis (CA): MAMPU’s approach to addressing KEQs 1, 2 and 3 will draw from ‘contribution analysis’ (CA)1. CA can be distinguished from traditional approaches to evaluation which typically attempt to attribute an outcome to a particular intervention. This often involves isolating (using statistical techniques) the role of the intervention from other factors that could be responsible for causing the outcome. By doing this it is possible to say unequivocally whether or not X intervention ‘caused’ Y outcome. Such an approach is well suited to highly defined interventions with largely predictable types of effect.
MAMPU will need a different approach. Influencing policy, fostering empowerment in a dynamic political, economic and social context is complex and non-linear and it is widely acknowledged that the outcomes of interventions in this space are more challenging to predict. Generally MAMPU will be only one of many factors that contribute to an observed change. In this situation it is more reasonable to establish a credible case that MAMPU contributed rather than attempt to tease out the effects of large numbers of variables that are often interdependently related.
CA is well suited to this task. This approach boils down to four ingredients. First, set out the ‘logic’ or ‘theory’ that shows how an intervention is expected to work. Secondly establish whether or not the expected outcomes have happened. Thirdly, map out the contribution by an intervention to that outcome using the theory or logic to structure the evidence. Fourthly, acknowledge and account for the relative contributions of other factors. This approach is reflected in the sections below addressing KEQs 1, 2 and 3.
3.2 Key Evaluation Question 1: How and to what extent has MAMPU affected the partners and networks’ capacity to influence government reform?
KEQ1addressestheshort-termoutcomeofMAMPUthatisexpectedtoemergewithinyears2to3oftheprogramlife:positivechangesinthecapacityandreadinessofpartnersandnetworks.To
Figure 4: Domains of organisational capacity assessed through the OCPAT
1 Mayne,J.(2008)ContributionAnalysis:Anapproachtoexploringcauseandeffect,ILACmethodologicalbrief,availableathttp://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf
Draft Monitoring and Evaluation Framework10
answerthisMAMPUmustfirstestablishifcapacityischangingandinwhatways.Assuch,thefirstsub-questionis:How and to what degree has the capacity of MAMPU partners changed?
Datatoaddresssub-question1willdrawfromtwosources:(i)longitudinalcapacityassessmentsofMAMPUnationalpartnerorganisations;and(ii)monitoringdataonpartnercollaborationwithotherorganisations.
Longitudinalassessmentsapplyastructuredmethodologycalledthe‘OrganisationalCapacityandPerformanceAssessmentTool’(OCPAT)developedbyIndonesianCSOYAPPIKA.TheOCPATisbasedonexistingevidenceofhowcapacitydevelopsinorganisations,includingtheexperienceofotherCSOprogramsinIndonesia2.Thetoolassessescapacityinsix‘domains’(seefigure4).Theprocesstakestwodayswitheachorganization and combines a highly participatory approach withspecificmeasurestobolstervalidityandreliability.TheassessmentisconductedbyindependentfacilitatorsandMAMPU’sroleislim-itedtoobserving,andonlywiththeapprovalofthepartner.
A ‘baseline’ OCPATwas facilitated withMAMPU partners in late2012/2013,asecondroundin2015,andathirdroundin2017.Ateachround,theassessmentmethodappliedamixtureofscoring,
Scale for tracking level of
partners and other types of
1 –
2 –
3 –
4 –
rankinganddiscussiontoidentifywhichdomainsofcapacityhadchanged,towhatextent,andwhatneedstobedonedifferently.AfinalroundofOCPATassessmentswillbecompletedin2019.
AsidefromtheOCPAT,additionaldatawillbeneededtoassessthestrengthofthenetworksamongpartnersandbetweenpartnersandothertypesoforganisations.ThiswilldrawfromMAMPU’squar-terly reportingsystem.Each threemonths,partner’sprovidedataon(i)organisationswithwhichtheyhadmostcontactduringthereportingperiod;(ii)describewhattheycollaboratedon;and(iii)usesasimplescale(seetextboxontheright)toassessthelevelofcollaboration.Thesedatawillbeanalysedtodrawoutwhichtypesoforganisationspartnersdevelopedlinkswith,whattypesofactivitytheyworkedtogetheron,andhowthischangedovertime.
Specifically,thisanalysiswilladdresstwoindicators: •Numberofinstancesofincreasedpartnercommunicationwithgovernmentorganisations orotherpolicymakingactors; •Numberofinstancesofincreasedpartnercollaborationwithotherorganisationsoutsideof government,specificallyforadvocacypurposes.Anincreaseinbothwillbeconsideredevidenceofincreasednetworkcapacitytoinfluencegovern-mentreform.
2 Inparticular,theOCPATisinfluencedbyamajorstudyoncapacitydevelopmentbytheEuropeanCentreforDevelopmentPolicyManagement(ECDPM)in2008.SeeBaser,H.andP.Morgan(2008),Capacity,ChangeandPerformanceStudyReport.(ECDPMDiscussionPaper59B).Maastricht:ECDPM.
Draft Monitoring and Evaluation Framework 11
WhilethisanalysiswillshowcapacitychangeamongpartnersandtheirnetworksitwillnotdirectlyassesstheextenttowhichMAMPUcontributedtosuchchanges.TofullyaddressKEQ1,MAMPUmustaddressasecondsubquestion:TowhatextentdidMAMPU’scontributetochangesincapacityandinwhatways?Therewillnotbeasingleanswertothisquestion.Rather,itislikelythatMAMPU’scontributionwillbemoreevidentinsomeaspectsofcapacitychangethaninothers.ItisalsolikelythatMAMPUwillmakeastrongercontributiontosomepartnersthanothers.Thiswillrequireanuancedapproach.
ThisanalysiswilluseinternallyhelddataonkeyfunctionsoftheMAMPUSecretariat:(i)grantfund-ing;(ii)technicalassistanceandadvice;(iii)conveningtheMAMPUnetwork;(iv)buildingandcom-municatingevidence;and(v)bridgingandlinkingpartners.ThesedatawillbebroughttogethertoassessthecasethatMAMPUmadeacontributiontochangesincapacityofpartnersandnetworks.Aqualitativescale(or‘rubric’)willbeusedtoenableclearjudgementsaboutMAMPU’scontributiontocapacitychange.Table2belowoutlinesadraftscale,whichwillneedtobefurtherrefinedbeforebeingapplied.
Table 2: Draft rubric to assess the strength of the case for MAMPU contri-bution to capacity change
12 Draft Monitoring and Evaluation Framework
3.3 Key Evaluation Question 2: How and to what extent have the partners and networks influ-enced government reform in relation to the needs and priorities of poor women?
WhileKEQ1addressescapacitychange,KEQ2focusesontheapplicationofthiscapacitytoinflu-encegovernmentreform.Therearetwointerrelatedfacetstothis.Firstly,partnersworkdirectlytoformnetworksandcoalitions (includingwithallies ingovernmentandprivatesector) to influencegovernmentdecision-making(formalandinformal)atmultiplelevels.Alongsidethis,partnersorgan-isewomenatlocallevel(villageanddistrict)andsupportthemtoexpresstheirviews(‘voice’)withtheexpectationthatthiswillinfluencechangethatbenefitspoorwomenandtheirfamilies.Bothofthese‘pathways’toachievinginfluencewillbeassessedunderKEQ2.
Subquestion1willfocusoninfluenceonformalgovernmentpolicies:How and what extent have MAMPU partners and networks influenced formal government policies?
DatatoassessthiswilldrawfromMAMPU’smonitoringsystem,particularlydata(qualitativeandquantitative)onengagementbetweenpartnersandpolicymakers,andpolicychanges.ThedataonMANISenablesMAMPUto:
•Tracktheprogressofdecision-makingonregulationsandformalpoliciesfromplanningto agendasetting,formaldebate,throughtoformalratification; •Monitorlevelsofengagementbetweenpartners,policymakers,themediaandother influentialstakeholdersincludingreligiousandcommunityleaders.
ThisenablesMAMPUtoidentifyinstancesofwhereMAMPUmaybeabletoclaimacontributiontodifferent typesofpolicychange, fromnational laws tovillage regulations.Minimumcriteria forinferringaclaimforcontributiontopolicyinfluenceare:
Gambar 5: Jumlah kebijakan di setiap tahap kemajuan, dashboard penelusuran kebijakan MANIS
50
01. In Prep 3. Under
Discussion4. Passed
47
22
37
Draft Monitoring and Evaluation Framework 13
(i) Therehasbeenpartnerengagementwiththepolicymakeroveratleasttwoquarterspriortothedecision,asevidencedinquantitativedataonengagement;and(ii) Priorengagementhasaddressedthesubstanceofthedecisiontakenbythepolicymakerasevidencedthroughnarrativematerialinpartnerreportingand/ormonitoringrecords.
Oncepotentialclaimsmeetthesecriteria,MAMPUfollowsuptocriticallyreviewthe‘influencesto-ry’,interviewingpartnerstoclarifydetailsandseekadditionalevidencewhererequired.UsingthisprocessMAMPUmaintainsalistofinstancesofpolicyinfluenceatmultiplelevels,backedbyverifi-ableevidence.
AsecondsetofsubquestionswilladdressthecriticalgrassrootsvoiceandinfluencethatMAMPUseekstodevelop: How and to what extent have partners contributed to the capacity of women at the village level to project ‘voice’? To what extent has this translated to influence on deci-sion-making at the household, village, and beyond?
Thesearecomplexquestions.ToaddressthemMAMPUwillsynthesizedatafromarangeofsourc-esagainstasetof‘evaluativecriteria’.Theseidentifythetypesofchangesthatwillbevaluedandtakentoindicatecapacity,readiness,voiceandinfluenceamongwomenatthevillagelevel.Table3containsadraftlist,developedwithpartners.
Table 3: Draft evaluative criteria to assess changes in capacity, readiness, voice and influence among women at the village level3
The data on these criteria will draw from several sources. Quantitative and qualitative data inQuarterlyProgressReportswillbecomplementedbyaqualitativemonitoring toolalready inuseby MAMPU– theMostSignificantChange (MSC) technique.Basedonfirst-personnarrativesaboutchanges(positiveornegative),MSCinvolvesthecollectionofstoriesfromwomenatthevil-lageacrossMAMPUwhicharethensystematicallyselectedbypartnersandMAMPU.Storiesare
3 ThesedraftcriteriawereidentifiedduringaworkshopfacilitatedbytheM&ESpecialistaspartoftheMAMPUPartner’sForuminJakartainJuly2017.
Draft Monitoring and Evaluation Framework14
Table 4: Draft rubric scale to assess capacity, readiness, voice and influence at the village level
uploadedbypartnersintoacustom-designeddatabase(‘MAMPUStorybook’)whereMAMPUcon-ductssecondaryanalysisof their content.Anopen-endedmonitoring tool,MSC iswell suited tocapturingcomplexsocialchangesthatareoftenintangibleandhardtoobservedirectly4.MAMPU’sownfieldmonitoringrecordswillbeusedtoverifyandtriangulatethesedatawithdirectobservationofvillage-levelprocesses.Inaddition,MAMPUwillneedtodrawuponarangeofresearchstudiesforfurtherevidence.ThesewillincludetheMAMPUlongitudinalsurvey,whichexamineschangesinaccesstoservicesin1500women-andmale-headedhouseholdsin15villagesover3waves:2014(baseline),2017(midline),and2019(endline).Itwillalsoincludeafurtherstudyoflocallevelvoiceandinfluence,designedtobuildonthefindingsofthe2017Women’sCollectiveAction(WCA)Study.
Thesedatawillbebroughttogetheragainstarubricscale–draftedintable4below–thatdistin-guishesbetweendifferentlevelsofachievementagainsteachoftheevaluativecriterioncontainedintable3above.Itisproposedthatthisrubricisappliedonapartner-by-partnerbasis.However,itwillbeessentialthatthisisfirstrefinedwithpartnersandadaptedtosuittheirdiversecircumstances.Nevertheless,theuseofthescalewillenablesomeconsistencyinthesynthesisofevidence.
3.4 Key Evaluation Question 3: How and to what extent has MAMPU contributed to improved ac-cess to essential government services and programs?
Putsimply,twodistinctbutlinkedpiecesofanalysiswillbeneededtoassessMAMPU’scontributiontoimprovedaccesstoessentialservices.First,itmustbeclearifmorepoorwomenhaveaccesstoessentialservicesinMAMPUlocations.Secondly,theremustbeanassessmentoftheMAMPUcontributiontosuchincreases.Theseanalyticalpointswilladdressedthroughtwosub-questions.Thefirstsub-questionconcernstheextentandscaleofchange:Has access to services increased for women who are poor and if so, where and by how much?Thiswillrequirethecollectionandanalysisofquantitativedata.Table4proposesasetof8quantitativeindicatorsforthispurpose.
3 KloostermanJ.(2012),‘Measuringtheunmeasurable’:gendermainstreamingandculturalchange,inGenderandDevelopment,Vol.20,2012,Issue3
Draft Monitoring and Evaluation Framework 15
Table 5: Quantitative indicators to measure changes in access to services
Draft Monitoring and Evaluation Framework16
Figure 6: Identifying quantitative indicators of access to services
Which theme ?
What governmentservice or program ?
What measures ofaccess are feasible ?(Direct & Indirect)
Indicatorsintable2havebeenidentifiedthroughacascadingprocessstartingwiththetheme,andthenconfirmingtheserviceorprogramaddressedbytherelevantpartners.MAMPUthenidentifiedindicatorsofaccesstotheserviceorprogram.TobefeasibleforMAMPU,these indicatorsneedtobesex-disaggregated,representativeatthedistrictormunicipality level,sufficiently linkedwithpartneractivity,andforwhichdataareavailableatbaseline(2015/2016)andendline(2019/2020).Assessingchangesintheseindicatorsacrossdifferenttargetareasshouldenableasufficientlynu-ancedpictureofchangesinaccesstoservicesacrossthediversityofMAMPUcontexts.
Quantitativeindicatorsintable2measuretwocategoriesofaccess:
Category1. DIRECTsupportbyMAMPU:wherewomenhavebeendirectlysupportedbypartnerstoaccessagovernmentserviceorprogram.Thisdatawillbecollectedandre-portedbypartnersthroughaspecificmoduleinthequarterlyprogressreportingtemplate.ThemagnitudeofincreasethatisachievableoverthelifeofMAMPUisexpectedtobeintheorderof10-15,000womenacrossallfivethemes.
Category2. INDIRECTbenefit throughpolicy implementation:Wherewomengain in-creasedaccesstoservicesthroughimplementationofgovernmentpolicydecisions.DatatomeasurethiswillcomefromselectedGovernmentofIndonesiadatasetsthatarerepre-sentativeatthedistrict/municipalitylevelandsex-disaggregated.TheseincludeSUSENAS(annual‘core’survey)aswellasadministrativedatapublishedroutinelybylineagenciesincluding theMinistryofLabourand theMinistryofWomen’sEmpowerment.Given thecoverageandscaleofMAMPU,thenumberofwomenwhocouldgainaccesstoservicesindirectlyisinthemillions.
Draft Monitoring and Evaluation Framework 17
Clearly,MAMPU’sinfluenceonpolicydevelopmentandimplementationiscriticalforwidescaleim-provementsinaccess(i.e.category2above).However,MAMPU’scontributionmustbeshownforthisoutcometobeclaimed.Thisisthefocusofthesecondsub-questionunderKEQ3:HowstrongisthecasethatMAMPUcontributedtoobservedincreasesinaccesstogovernmentservicesandprograms?
Data toanswer thiswill drawprimarily fromMAMPU’smonitoringsystem including thequarterlyprogressreportsandfieldmonitoring,supplementedwithcasestudiesandwhereavailable,otherqualitativematerialsuchasMostSignificantChangenarrativesandresearchstudies.ThisdatawillbedrawntogethertoassessthestrengthofthecaseforMAMPU’scontributionagainsta5-point‘rubric’(describedintable6).
Table 6: ‘Draft rubric to assess the strength of the case for MAMPU contribution to improved access to services
Combininganassessmentofcontribution(sub-question2)andquantitativeanalysisofchangesinaccess(sub-question1)willenableMAMPUtoanswerKEQ3clearly,butalsoinanuancedwaythatreflectsthediversityacrosstargetareas.UsingthisapproachMAMPUwilldistinguishbetweenfourpossibletypesofscenario,aslistedintable7.
Draft Monitoring and Evaluation Framework18
Table 7: Combining sub-questions to assess EOPO achievement in target districts and municipalities
Itisimportanttodifferentiatebetweenthese.Targetareaswheretypes3or4haveoccurredwillberegardedashavingachievedtheEOPO.However,onlytype4willbecountedasanactualincreaseinthenumberofwomenwithaccesstoserviceswithintheMAMPUtime-frame.Furtheranalysisofthequantitativedatawillbedonewheresuchcasesareidentified.Thiswill includeattentiontotherelativedifferencesinaccesstoservicesexperiencedbywomenandmeninthetargetareatodrawtentativeconclusionsabouteffectsongenderequalitybeyondthelifeofMAMPU.
3.5 Judging success at the End-of-Program stage (Targets)
AlthoughMAMPUaimsforwidespreadimprovementsinaccess,thelinksbetweenpolicyinfluence–evenatlocallevels–andservicedeliveryareundeniablycomplex5.Toaccountforthis,MAMPUconsiderstargetsettingatthedistrictormunicipalityleveltobeanappro-priateapproach.Thealternative–expressingtargetsintermsofthenumberofwomenwithincreasedaccess to services–will inevitably understate the instanceswhereMAMPU’scontributionis‘strong’or‘verystrong’butnotyetreflectedattheservicelevel.MAMPUwilltrackquantitativechanges inwomen’saccess toservices,butultimatelysuccesswillbejudgedonthebasisofcontributiontogovernmentdecision-makingthathasalreadyledtoorislikelytoleadtoanincreaseinaccess.ThisisconsistentwiththeintentofKeyEvaluationQuestion 3.
5 ArecentevaluationofaDFAT-fundedprogramtoimproveservicedeliveryineasternIndonesiarecommendedthatdonorstakealong-termandmultilevelapproachthatrecognisesthiscomplexity.SeeL.KellyandSakriD.(2015),IndependentCompletionReport,Australia-IndonesiaPartnershipforDecentralisa-tion,Canberra:DFAT
Draft Monitoring and Evaluation Framework 19
Table5 sets targets for thenumberof districtsormunicipalitieswhereMAMPUexpectstheEOPOtobe‘largelyachieved’or‘fullyachieved’infourofthefivethemes.Tosetthesetargets,MAMPUconsultedwithpartners,reviewedtrendsinpartnerengagement,andcon-sideredrecentchangestoprogramcoverage.
Theme2,whichhasnotbeensetaspecifictargetinTable5,isaspecialcase.Becauseofthelowvisibilityofthehomeworkerissueamongpolicymakers,MAMPUdoesnotexpecttoachieve influence thatwill increaseaccess toservicesbefore2020. Instead,MAMPUexpects toachieveaquantitative targetofdirectly increasingaccess tohealth insurance(BPJSKetenagakerjaan).
Meetingthethematictargetsintable5willbeasoundbasisonwhichtojudgethatMAMPUhasachievedtheEOPOatascalethatreflectstheintentintheprogramdesign.
3.6 Types of evaluation
TwotypesofevaluationwillbeundertakenoverPhaseIIofMAMPU:(i)twointernalevalu-ationsbasedontheCollaborativeOutcomesReporting(COR)technique6;(ii)andaninde-pendentmid-termevaluationcommissionedbyDFAT.
Internal evaluations addressingallfourKEQswillbeundertakenattwopoints:duringthefirsthalfof2018;andinlate2019/early2020.FacilitatedbytheMAMPUM&ESpecialist,
Table 8: Targets for MAMPU achievement by 2020, by theme
6 http://www.betterevaluation.org/en/plan/approach/cort
Draft Monitoring and Evaluation Framework20
thesewilldrawtogetherdatacollectedthroughtheM&EsystemtoassessthestrengthofMAMPU’scontribution toshort-,medium-termoutcomes,andexamineprogress towardstheEOPO.ThesecondinternalevaluationwillmakeasummativeassessmentofprogramachievementoftheEOPO,applyingthemethodologyoutlinedabove,alongwithdataavail-ablefromothersources.
WhiletheM&ESpecialistwillberesponsibleforleadingtheprocess,theCORtechniqueworks best as participatory exercise that involves different stakeholders in data collec-tion and analysis. Consistent with the COR methodology, all claims of contribution willtransparentlyreferencethesourceofevidencethatcanbeverifiedbyanindependentparty.MAMPUwillensureanindependentperspectivetoassessthestrengthofthecaseforcon-tributionforoutcomes.Thefindings,evidence,andrecommendationswillbedocumentedinashortreadablereportcalleda‘PerformanceStory’.
DFATinconsultationwithBappenaswillberesponsibleforcommissioninganindependent evaluationofprogressoverPhaseII.TheexactscopeandtimingofthisevaluativeexercisewillbedeterminedbyDFATbutithasbeententativelyscheduledtotakeplaceinthesecondhalfof2018.
4.0 How will the information reach decision-makers?
Thissectionbrieflydescribesthedifferentreportingandinformationproductsthatcontainfindingsandrecommendations fromtheM&Eprocessesdescribed in theprecedingsec-tions.Itexplainshowthesewillreachkeyaudiencestosupportdecision-makingabouttheprogram.
KeyaudiencesoftheinformationgatheredthroughtheM&EsystemaretheMAMPUSecre-tariat,partners,AustralianEmbassy,ThematicWorkingGroups(TWGs),TechnicalCommit-tee(TC),andSteeringCommittee(SC).ThesedifferentstakeholdersplaykeyroleswithinthegovernancestructureforMAMPUoutlinedintheSOPs.Therightinformationwillneedtoreachthesestakeholdersattherighttimeandintheappropriateformattosupportdecision-making.
Figure 7: Timeline of evaluations over Phase II
Draft Monitoring and Evaluation Framework 21
TheMAMPUSecretariatwillneedfrequentfeedbackonperformanceatthepartnerlevel,acrossathematic‘hub’,aswellasfortheprogramoverall.Ofparticularimportancewillberapidfeedback,earlywarningsigns,andrisksthatenabletheSecretariatstafftotakeac-tion.Thisneedstobealliedtomore‘strategic’andlong-termconsiderationsthatenabletheSecretariattoseehowimmediateconcernsaffectthelikelihoodofachievingtheEOPO.
Asthedonor,theAustralianEmbassyneedsinformationtoassessprogresstowardsshort-,medium-andlong-termoutcomes,aswellassummarizedinformationonimplementationoftheannualworkplanandbudgetaryperformance.ThisinformationshouldenableEmbassystafftomeetinternalaccountabilityandcompliancerequirementsbutalsomakedecisionsconcerningrisksandstrategicopportunitiesthatwouldbenefitfromofficialgovernment-to-governmentinvolvement.
The5ThematicWorkingGroups (TWGs)are venues for partner-government interactionandcoordination,notdecision-making.Tosupportthesefunctions,informationisneededonpoliciestargetedineachtheme,progressandobstacles,andhighlightsandlessonsfromwork‘ontheground’wherepartnersareattemptingtosolvepolicyissuesinpracticalways.TheTechnicalCommittee(TC)requiressummarizedinformationonprogramperformanceagainstoutcomes,particularlyprogresstowardstheEOPO.Theinformationshouldbesuf-ficientlydetailedtoenablememberstojudgetheadequacyofoverallperformance,considerrelativeprogressacrossthemes,andassesstheappropriatenessoftheAnnualWorkplanandbudgetproposedbytheMAMPUSecretariat.
TheSteeringCommittee (SC)needshigh levelsynthesized informationonperformance,sufficienttoapprovetheannualworkplanandbudget.AkeyconsiderationfortheSCwillbetheongoingrelevanceandalignmentofMAMPU’sportfoliowithstrategicprioritiesofbothgovernments.
Draft Monitoring and Evaluation Framework22
Table 9: Key reporting and information products, audience, content and timing
Draft Monitoring and Evaluation Framework 23
Annex 1 Key terms and concepts
Analysis originates from a 2001 paper by John Mayne, formerly of
the Auditor-General of Canada. ( -
).
In it, Mayne argued that a narrow focus on outcomes to an . Instead, there should
be a focus on using evidence to show contributed. This , requires six steps: (i) acknowledge the
theory of change; (iii) show that the steps in the theory of change have occurred
the
Poor women, poverty locally relevant and suited to the range of contexts where they work. Overall, MAMPU is guided by the cent’,