DIME – FRAGILE STATESDUBAI, MAY 31 – JUNE 4

Introduction to Using Qualitative Methods in Impact Evaluation

Ryan SheelyKennedy School of GovernmentHarvard UniversityDIME Workshop, DubaiJune 1, 2010

Multiple Reasons for Using Participatory Methods

This morning’s session emphasized social justice and accountability as reasons to utilize participatory methods in impact evaluation

There are are also methodological justifications for integrating participatory assessment and other qualitative measures with randomized impact evaluation

What Are Qualitative Methods?

Qualitative methods are a diverse set of research tools that are primarily concerned with the meaning of social actions, statements, and documents

Both alone and in combination with randomized evaluation and sample surveys, these methods provide leverage with respect to: Describing and categorizing social contexts Interpreting the social behavior, actions,

and outcomes Tracing the specific processes and

mechanisms whereby programs lead to effects

The Diversity of Qualitative Methods Participatory and Action-Based Research

Interviewing and Discussion

Content Analysis of Documents and Records

The Diversity of Qualitative Methods Participatory and Action-Based Research

Systematic Social Observation Drama and Role Playing Maps, Models, and Diagrams Ranking and Scoring

Interviewing and Discussion

Content Analysis of Documents and Records

The Diversity of Qualitative Methods Participatory and Action-Based Research

Interviewing and Discussion In-Depth, Semi-Structured Interviews Focus Group Discussions Oral Histories of Individuals, Communities, or

Projects

Content Analysis of Documents and Records

The Diversity of Qualitative Methods Participatory and Action-Based Research

Interviewing and Discussion

Content Analysis of Documents and Records Diaries or Personal Time/Financial Records Archives and Government Records Newspapers and Civil Society Reports

Analyzing Qualitative Data: The Role of Narrative

Despite the differences between these methods of collecting qualitative data, the common principle is using close observation to understand the meaning of what people say and do

As a result, the writing of field notes and systematic the aggregation of notes into narrative case studies are a central skill in analyzing qualitative data

Combining Qualitative and Quantitative Methods

Qualitative methods can be linked to randomized evaluation

Before Implementation: Use interviews, participation, and observation to design

contextually relevant surveys and behavioral activities Use observation or participation to design interventions

that can test the interaction of different program components

After Implementation: Aggregating and coding the content of interviews,

behavioral observation, and newspapers to cross-check survey data

Analyzing process to understand the lack of statistically significant differences▪ Implementation failure vs. design failure?

Homework: Qualitative Methods and Your Projects

When you meet with your clinic groups this afternoon, consider this question:

What types of qualitative methods would add value to the evaluation of your project?

Focus on three things:1) Identify each question you want to answer2) Choose a set of qualitative methods to

integrate into your evaluation design for each question and how/when you would use each method

3) Why these methods add value to your research design

Qualitative Methods:Methodological Limitations

Small, Nonrepresentative samples Researcher/Facilitator Bias Possibility of Contamination of Control

Groups when combining with Randomized Evaluation

Lack of standards for comparing qualitative and quantitative findings

Qualitative Methods:Practical Limitations Time and Cost of Qualitative Research

Understanding and Documenting Context Requires Time to Develop Mutual Trust Between Researcher and Community

Increased Staffing and Training due to more diverse skill sets

Coordination and Management Problems Associated With Larger Teams

Conclusions: Mixing Qualitative and Quantitative Methods

Qualitative Methods are a diverse set of tools and skills that require training, practice, and careful research design

Qualitative methods can help to better understand why a given program did or did not have an effect Helping to develop better treatments and outcome

measures for randomized evaluations Using local knowledge and cultural context to

interpret the meaning of social behavior and process of intervention

Ideal mix of qualitative and quantiative methods depends on the questions you want to answer and the local context

REFERENCES AND SUGGESTIONS FOR FURTHER READING

Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications

Barron, Patrick, Rachael Diprose and Michael Woolcock (2010) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press (in press)

Chambers, Robert. 1997. Whose Reality Counts: Putting the First Last, ITDG Publishing.

Cronk, Lee. 2007. “The influence of cultural framing on play in the trust game: a Maasai example” Evolution and Human Behavior.

Kanbur, Ravi, ed . 2001. Q-Squared? Qualitative and Quantitative Methods of Poverty Appraisal

Malan, N. 2000. “On the Relationship between participatory research and participatory development.” Africanus 2: 64-75.

Ravallion, Martin. 2001. “How Can Qualitative Methods Help in Measuring Poverty,” Qual-Quant: Qualitative and Quantitative Poverty Appraisal- Complementarities, Tensions and the Way Forward, Permanent Black Publishers, 2002.

Rodrik, Dani. 2008. “The New Development Economics: We Shall Experiment, but How Shall we Learn?” Paper Prepared for the Brookings Development Conference, May 29-30 2008.

Sheely, Ryan. 2010. “Community Governance, Collective Action, and the Maintenance of Local Public Goods: Qualitative and Experimental Evidence from Rural Kenya.” Better Governance for Better Health Conference, Stanford University. April 26-27, 2010.

Van der Riet, Mary. 2008. “Participatory Research ant he Philosophy of Social Science.” Qualitative Inquiry 14 (4): 546-565.

White, Howard. 2002. “Combining Quantitative and Qualitative Approaches in Poverty Analysis,” World Development.

Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14