Really Using (Useful) Theories of Change

Really Using(Useful)

Theories of Change

IPDET 2013John Mayne, Advisor on Public Sector

[email protected]

mailto:[email protected]

mailto:[email protected]

Using Theories of Change

1. Understanding and agreeing on interventions2. Addressing equity issues3. Designing interventions4. Ex ante evaluation5. Designing monitoring systems6. Managing adaptively7. Reporting performance8. Designing evaluation questions, methods and tools9. Making causal claims10. Generalizing to other locations

Behaviour changes

Activities and Outputs with respect to beneficiaries

External Influences

Capacity changes in knowledge, attitudes

skills, opportunities & incentives

Reach &Reaction

Direct Benefits

Impacts

A Generic Results Chain

Timeline

Behaviour changes


External Influences



Reach &Reaction

Direct Benefits

Impacts

A Generic Results Chain

Timeline

Note• No labeling of

outcomes types• Focus on program

theory• Timeline• Built on capacity and

practices changes• Explicit recognition of

reach• Feedback loops

Behaviour changes


External Influences



Reach &Reaction

Direct Benefits

Impacts

Assumptions and Risks

Timeline

Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes?

Risks: Risks to the link not occurring.Other Explanatory Factors: Peer or trend pressure; other interventions

An assumption is NOT a description of the causal link

Behaviour changes


External Influences



Reach &Reaction

Direct Benefits

Direct Benefits Assumptions & Risks Other explanatory factors

Behaviour Change Assumptions & Risks

Other explanatory factors

Capacity Change Assumptions & Risks

Other explanatory factors

Reach Assumptions & Risks

Impacts

Impact Assumptions & Risks Other explanatory factors

A Generic Beneficiaries Theory of Change

Timeline

timelineAwareness and skills training provided to

teachers on girls’ education needs

External InfluencesParentsPeers

ReligionCulture

Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills

All relevant teachers get the

training

Girls learning improves

Results Chain for Enhancing Education Outcomes for Girls

Girls more engaged in and wanting an

education

Improved education

outcomes for girls

Teachers provide girls with more empathetic

and supportive teaching in schools

Intervention to improve educational outcomes for girls by providing gender sensitivity training to teachers, in a region where girls education has not been a priority.


Other Factors for Enhancing Education Outcomes for Girls

Girls more engaged in and

wanting an education

Teachers provide girls with more empathetic and

supportive teaching in schools

• Parents and community support

• Time for studying

• Access to schools• Accommodation in

schools




ReligionCulture

Education resources Capacity changes in teachers’ knowledge, attitudes and skills


training


Unintended effect: Boys’ education suffers

Girls’ Behaviour Assumptions & RisksGirls have access to schoolsSchools can adequately

accommodate girlsOther explanatory factors Internet access becomes widely

available

Theory of Change for Enhancing Education Outcomes for Girls


education

Improved education

outcomes for girls



Girls’ Direct Benefit Assumptions & RisksGirls have time to studyGirls education supported by parents and

communityFocus on girls is not disruptive to boy’s

educationOther explanatory factors Internet access becomes widely available

Other factorsSeems clear that more than training teachers may be required, such as:• Getting girls to school• Parental and community support• Schools accommodate girls• Teachers support girls education• Empathetic teaching is rewarded• Focus on and resources for girls’ education are

ongoing (outcomes take time)




ReligionCulture



training



Teacher Behaviour Change Assumptions & Risks

Training relates to the local conditions Teachers are assessed in part on their

application of the training

Capacity Change Assumptions and Risks• Most teachers want to better education

for girls

Reach Assumptions & Risks Only teachers with a predisposition

to the importance of girls’ education attend

Impact Assumptions & Risks Resources and will continue for

multi-year involvement



education

Improved education

outcomes for girls



Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available

Girls’ Direct Benefit Assumptions & Risks Girls have time to study Girls education supported by parents and

community Focus on girls is not disruptive to boy’s


Equity: Distribution of impacts• As set out so far, ‘girls’ is an general group• But among ‘girls’, there are worst-off girls, who may need

special attention (e.g., access, income, conservative parents)

• And as already noted, there may be unintended impacts on boys

• To get at these distributional affects, we can build nested theories of reach:– A ToC for a particular reach group within an overall ToC

• And can then tell reach impact stories about the effects and impacts of different reach groups




ReligionCulture



training







for girls







education

Improved education

outcomes for girls







Teachers Theory of Reach

Girls Theory of Reach

Worst-Off Girls Theory of Reach




ReligionCulture



training







for girls







education

Improved education

outcomes for girls










ReligionCulture



training







for girls







education

Improved education

outcomes for girls




Girls’ Direct Benefit Assumptions/Risks Girls have time to study Girls education supported by parents and



Supporting activities




ReligionCulture



training







for girls







education

Improved education

outcomes for girls




Girls’ Direct Benefit Assumptions/Risks Girls have time to study Girls education supported by parents and



Supporting activities

Teachers Theory of Reach

Girls Theory of Reach

Parents Theory of Reach

Worst-Off girls learning improves


Worst-Off Girls’ Behaviour & Direct Benefit Change Assumptions & Risks Worst-off girls have

access to schools Worst-off girls education

supported by parents and community

Worst-off girls have time to study

Schools can adequately accommodate girls

Focus on girls is not disruptive to boy’s education

Other explanatory factors Internet access becomes

widely available

Impact Assumptions & Risks Resources and will continue

for multi-year involvement

Theory of Reach for Enhancing Education Outcomes for Worst-Off Girls

Worst-Off girls more engaged in and wanting an

education

Improved education outcomes for worst-

off girls

Teachers provide (all) girls with more empathetic and

supportive teaching in schools

Worst-off girls are provided transportation to schools

Engaged with parents & communities with worst-off girls

Support Activities

intervention activities results

timeline

timeline

Engagement with governments

Engagement with NGOs

Partner Activities

Nested theory of reach for access by worst-off girls

Reach &Reaction

Capacity Changes

Behaviour Changes

Impact

Direct Benefits Assumptions & Risks Other Explanatory Factors

Behavioural Change Assumptions & Risks Other Explanatory Factors

Capacity Change Assumptions & Risks Other Explanatory Factors


External Influences

Supporting Activities and O

utputs

Direct Benefit Changes

Impact Assumptions & Risks Other Explanatory Factors

TimelineActivities and Outputs

Timeline

A Basic Theory of Change for Comprehensive Interventions

Reach, C

apacity changeB

ehavioural Change

Reach &Reaction

Capacity Changes

Behaviour Changes

Impact





External Influences


utputs




Timeline

A Basic Theory of Change for Comprehensive Interventions

Reach, C

apacity changeB

ehavioural Change

Different Beneficiary Nested ToC

Different Intermediary Nested ToC


5. Designing monitoring systems– likely need more than just a focus on results—the

whole ToC, including assumptions and risks6. Adaptive management– Identify what to be monitored– On a regular basis, reflect on what is being

observed and on your ToC – Revise your intervention activities and your ToC as

you learn

Using Theories of Change7. Reporting performance– A verified ToC is your performance story

8. Designing evaluation questions, methods, tools– Identifies questions– Adds power to case studies by confirming the ToC– Basis for surveys et al

Behaviour changes


External Influences



Reach &Reaction

Direct Benefits

Impacts

There are potential Evaluation Questions associated with each box and arrow

Timeline

Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes?

Risks: Risks to the link not occurring.Other Explanatory Factors: Peer or trend pressure; other interventions

Behaviour Change• What behavioural

(practice) changes are observed?

Causal Question• To what extent are the

behavioural changes the result of the capacity changes?

Capacity Change• What capacity) changes

are observed?

Assumption Questions• Did the events/conditions

occur?• Did risks materialize?• Are other explanations

plausible?


9. Making causal claims– Based on generative causality– ToC are models of causality– Approaches such as contribution analysis

demonstrates causality

Theories of Change asCausal Packages

ToC is a model of the intervention as a contributing cause– ToC identifies supporting factors (assumptions)

and confounding factors (risks)– Intervention activities plus the assumptions are

sufficient to bring about the intended impacts– Explains how and why intended impacts are

expected to occur

Reach &Reaction

Capacity Changes

Behaviour Changes

Impact





External Influences


utputs




Timeline

A Basic Theory of Change

Causal package

Contribution AnalysisUsing a generative perspective on causality, CA shows that an intervention is a contributory cause:• The expected result occurred• The ToC (causal package) is sufficient– supporting factors (assumptions) occurred and any other

supporting factors have been included in the ToC– plausible rival explanations have been accounted for

• The intervention is necessary for the ToC to be sufficient• And can explore the role the intervention played, such

as a trigger


10.Generalizing– What works where and when?– ToC explicitly includes the context and the causal

mechanisms– Confirming a ToC provides significant evidence on

if and how the intervention would work elsewhere

Messages

• ToC are worth the effort to develop• Not a throw away, but a powerful tool to use

in evaluation and in managing• Lots of fun … and lots of debate!

ReferencesMayne, J. (forthcoming). Using Theories of Reach to Enhance Equity Considerations in

Evaluation. In Evaluation and Equity. K. Forss and M. Marra, Eds: Transaction.Mayne, J. (2013). Making Causal Claims. Connections, June. Available at

http://www.europeanevaluation.org/images/file/files/ees-newsletter-2013-06-june.pdf. Mayne, J. (2012). Contribution Analysis: Coming of Age? Evaluation 18(3), 270-280.Mayne, J. (2012). Making Causal Claims, ILAC Brief No. 26: The Institutional Learning and

Change Initiative. Available at http://www.cgiar-ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf

Stern, E, Stame, N., Mayne, J., Forss, K, Davies, R. and Befani, B. (2012). Broadening the Range of Designs and Methods for Impact Evaluation, Working Paper 38, Department for International Development, London. Available at http://www.dfid.gov.uk/R4D/Output/189575/Default.aspx

Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf

http://www.europeanevaluation.org/images/file/files/ees-newsletter-2013-06-june.pdf

http://www.cgiar-ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf

http://www.cgiar-ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf

http://www.dfid.gov.uk/R4D/Output/189575/Default.aspx

http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf

http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf

Documents

Really Using (Useful) Theories of Change