Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Participatory Research, Empowerment, and Accountability: Assessing Participant Driven Evaluation
Ryan Sheely∗
Associate Professor of Public Policy Harvard Kennedy School of Government
September 28, 2018
Abstract: Participatory research is a set of related approaches to including communities in the process of designing and implementing policy-oriented research projects. Practitioners and academics in international development have argued that participatory research can empower marginalized communities by increasing individuals’ sense of efficacy and reconfiguring accountability relationships. However, evidence of the relationship between participatory research and empowerment is limited. This paper presents the results of an exploratory field experiment designed to assess the impacts of Participant Driven Evaluation (PDE), a project implemented by a Kenyan NGO. The PDE program consists of a set of interactive workshops and community-driven projects that train participants in how to use mixed-methods social science research to design and evaluate development programs. The results of the experiment indicate that the implementation of this particular participatory research intervention has a broad impact on attitudes and behaviors towards research and more targeted effects on empowerment. With respect to the psychological aspects of empowerment, the intervention had a positive impact on individuals’ sense of their own capabilities, but no effects on collective or political efficacy. With respect to accountability, participatory research was more effective in fostering “short route” of accountability relationships between citizens and service providers than it was in strengthening the “long-route” of accountability linking citizens to politicians. Taken together, these findings highlight both the opportunities and challenges associated with using participatory research to empower communities and highlight the need for further collaboration and dialogue between the diverse sets of scholars, practitioners, and communities engaged in participatory research and evidence-based policymaking.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!∗!Acknowledgements: Ruth Carlitz, Bill Clark, Kim Yi Dionne, Joan Ricart-Huguet, Linda Stern, and Beth Wellman all provided useful feedback and comments on earlier drafts of this paper. Vanessa Zhang, Peninah Ndegwa, Chenyue Ma, Naomi Mathenge, Tara Grillos, Aletheia Donald, Akshay Dixit, Peter Fenzel, and staff members of the SAFI project and CREED contributed valuable program design, implementation and research assistance at various stages of this project. This project was made possible in part by funding from the Weatherhead Center for International Affairs, the Ash Center for Democratic Governance and Innovation, and the Milton Fund at the Harvard School of Medicine. The research reported in this study was carried out under Harvard University IRB Protocol #20883.!
! 2
Practitioners and academics in the field of international development have started to
implement participatory research as a component of their programs and projects (J. A. Ashby and
Sperling 1995; Brisolara 1998; Defoer et al. 2000; Israel et al. 2001; Dalton et al. 2011).
Participatory research refers to a broad category of approaches that directly incorporate
development project beneficiaries in the process of conducting social science research, including
asking research questions, designing interventions, measuring outcomes, and using evidence to
advocate for policy change (Cargo and Mercer 2008; Chambers 2008). A major rationale for the
use of these methods is that including citizens in the process of the research has the potential to
empower individuals (Brunner and Guzman 1989; Wetmore and Theron 1998). Despite these
normative claims about participatory research, there is relatively little quantitative evidence in
favor of this hypothesized link between participatory research and empowerment.
This paper uses a randomized-rollout field experiment in rural Kenya to provide
exploratory empirical evidence about the impact of a participatory research project on two
aspects of empowerment: efficacy and accountability. I worked with a Kenyan NGO to develop
Participant Driven Evaluation (PDE), a participatory research program focused on using mixed-
methods social science research design in service of empowering communities to solve problems
and hold service providers accountable. The NGO implemented the project in 32 villages in one
Kenyan County. I evaluate the impacts of this project by randomly assigning villages to one of two
implementation waves and measuring a set of social and political outcomes at the midpoint of the
implementation, treating the second wave of villages as a control group.
There are three main sets of findings. First, I find that the intervention has a large and
statistically significant positive impact on individuals’ attitudes, behaviors, and knowledge
regarding research. Second, I find more mixed effects of the participatory research intervention
on efficacy. Although the intervention has a significant positive effect on individuals’ perceptions
of their own personal capabilities, there is no effect on individuals’ sense of efficacy regarding
collective action or political engagement. Finally, the participatory research intervention also has
mixed effects on accountability relationships between citizens and various government and non-
governmental actors. In particular, the intervention increased the frequency of citizen
! 3
engagement with street-level bureaucrats and civil society organizations. At the same time, there
is no effect of the intervention on the likelihood of citizens sanctioning these actors for poor
performance or overall citizen satisfaction with service provision by these actors.
The paper proceeds as follows. In the next section, I provide an overview of the core
concepts explored in this paper—participatory research, efficacy, and accountability. I then
highlight three families of empirical research questions about the relationship between
participatory research and empowerment. In the next section, I provide an overview of the
randomized field experiment that I utilized in this study, describing the PDE intervention, the data,
and the empirical strategy that I use to evaluate the impact of the intervention. I then discuss key
findings from the field experiment, focusing on both average effects related to the broad research
questions and more targeted effects on specific measures. I conclude by briefly discussing the
impact of these findings on research and practice related to participatory research,
empowerment, and accountability.
Concepts and Questions
Participatory Research in Development
Over the course of the 1990s and 2000s, participation became a central preoccupation
in the practice of international development (Casey, Glennerster, and Miguel 2012; Mansuri and
Rao 2004, 2012; Sheely 2015). Concurrent with this larger trend toward incorporating
participation into the planning and implementation of development projects, practitioners and
researchers started to incorporate a similar set of approaches into their attempts to use social
science research to design and evaluate interventions (Brisolara 1998; Wetmore and Theron
1998). Broadly, participatory research is an umbrella term for a wide array of approaches,
programs, and methods that emphasize values of inclusivity and which recognize the importance
of deeply including a research project’s subjects in the broader process of asking questions,
gathering and analyzing data, and taking action based on findings (Cargo and Mercer 2008).
As with participatory development more broadly, participatory research had its roots in
Paolo Friere’s critical approach to adult education, as well as in Kurt Lewin’s “action research”
which emphasized the use of discussion and reflection to identify and address problems within
! 4
firms and organizations (Cargo and Mercer 2008; Dickens and Watkins 1999; Freire 1970; Lewin
1946; Wallerstein and Duran 2006).
The subsequent evolution of participatory research has been the result of substantial
interaction and collaboration between practitioners working for NGOs and governments,
qualitative researchers in disciplines like anthropology and sociology, and members of
communities that are themselves the beneficiaries of development projects. As a result of these
origins in a diverse community of practice that spans multiple professions, disciplines, and
regions, the actual types of programs, interventions, and methods that fall under the broader
umbrella of participatory research vary substantially in the nature and scope of participation that
they employ.
In international development, the best known participatory research approach is
Participatory Rural Appraisal (PRA), a collection of collective ranking, mapping, and scoring tools
to diagnose and solve local problems most closely associated with the writing and work of Robert
Chambers and a vast network of NGOs across the Global South (Chambers 1994b, 1994a,
2008). A related approach in development is Participatory Evaluation/Participatory Impact
Assessment, which utilizes a similar set of interactive and collaborative research methods to
evaluate the impact of programs (Brunner and Guzman 1989; Catley et al. 2013; Cullen and
Coryn 2011). Another set of approaches to participatory research in development focuses on
forging partnerships between researchers and farmers to develop, test, and scale agricultural
technologies (Asiabaka 2002; Braun, Thiele, and Fernández 2000; Martin and Sherington 1997;
Reid et al. 2009).
Several other approaches to participatory research have been primarily used outside of
the development space. Community-Based Participatory Research, which is primarily used in
public health and nursing in advanced industrialized countries incorporates community members
and nonprofits in the process of designing and evaluating health interventions, with the joint aims
of empowering communities, encouraging behavior change, and improving health outcomes
(Cargo and Mercer 2008; Israel et al. 2010; Wallerstein and Duran 2006). Citizen Science is
primarily used in environmental management and sustainability sciences to engage members of
! 5
the public in research in the natural sciences, with the aim of empowering individuals to
participate in environmental policy discussions and conservation activities (Bonney et al. 2009;
Conrad and Hilchey 2011; Irwin 1995).
Given the wide variety of approaches and methods that can be classified as
participatory research, it is helpful to disaggregate three dimensions along which research
projects can be more or less participatory: pedagogy, organizational structure, and methods.
While a full classification of participatory research approaches into an exhaustive typology is
beyond the scope of this study, identifying the possible range of participation in each dimension
helps to provide context for the wide range of variation between participatory research
approaches.
In the context of research projects, pedagogy refers to the methods that are used to train
members of the research team, community members, and other stakeholders. Participatory
modes of pedagogy involve discussions and activities that stimulate active engagement by
learners, including facilitated discussions of case studies, role-plays and simulations, and
participant-led instruction (Freire 1970; Israel et al. 2010). In contrast, less participatory modes of
pedagogy are more didactic, and involve one-way information flow from an instructor or trainer to
the learners (Israel et al. 2010).
The organizational/social dimension of a research project involves the set of
institutional structures within a research team that determine who makes decisions about the
scope and aims of research, how tasks are executed, and who gets to use the data generated
through the research project. The participatory mode of organizing a research team is associated
with a flat organizational structure with limited role differentiation and highly dispersed authority
(Flaskerud and Nyamathi 2000). In this type of research organization, the broad set of
stakeholders associated with the research project collectively share in decision-making about
what questions to study, what methods to use, and the ultimate analysis and use of the data. A
non-participatory mode of organizing a research project utilizes a hierarchical organizational
structure in which a single researcher or small research team controls decisions about research
questions, research design, and data use and analysis (Flaskerud and Nyamathi 2000).
! 6
The methodological dimension of a research project involves the way that researchers
collect and analyze data. Participatory research methods entail generating data through collective
discussion and social interaction (Mukherjee 2002). These participatory methods may include
collective ranking and scoring methods (Catley et al. 2013), dramas and role-plays (Gallacher
and Gallagher 2008), and focus groups (Cheezum et al. 2013). A common feature of all of these
data collection methods is that they entail high degrees of interaction between the research team
and the research subjects themselves.
This level of open interactivity between research participants means that content and
focus the activity can evolve as a result of the knowledge, norms, and interactions of the local
participants, allowing these methods to capture dynamics of local social interactions that may be
hard to observe otherwise (van der Riet 2008). Non-participatory methodologies involve data
collection methods typically associated with quantitative analysis, such as randomized controlled
trials and random sample surveys. These types of data collection methods typically entail low
levels of interaction between the research team and the research subjects (Chambers 2010;
Sheely 2016).
Participatory Research and Empowerment
As noted above, the primary normative and political goal of the incorporation of
participatory research into international development projects is to empower individuals who are
typically marginalized. There are two main channels through which participatory research is
hypothesized to lead to empowerment.
First, participatory research can lead to empowerment by increasing participants’ sense
of efficacy. In social psychology and related literatures, efficacy is defined as individuals’ own
sense of the ability of themselves or a group to achieve goals (Grillos 2015). As this definition
indicates, efficacy can exist on a number of different dimensions-- individual, collective, and
political. Individual efficacy refers to individual’ sense of what they are capable of—what kinds of
tasks they can undertake and their likelihood of success at those tasks (Israel et al. 2010; Kim
and Park 2008). Political efficacy refers to individuals’ perception of their ability to influence
politics, either through voting or through direct interactions with politicians and bureaucrats (Yeich
! 7
and Levine 1994). Collective efficacy refers to individuals’ sense of their community’s ability to
successfully engage in collective action (Israel et al. 2010; Bandura 2000).
All of these types of efficacy are connected to empowerment because increased efficacy
can lead individuals to engage in social and political actions that they would not have previously
even attempted (Grillos 2015; Ohmer 2007). Participatory research has been linked to efficacy
because the act of learning to participate in the systematic production of knowledge has the
ability of expanding individuals’ capabilities, as well as their awareness of those capabilities
(Berg, Coman, and Schensul 2009; Israel et al. 2010; Wallerstein and Duran 2006). Thus, if
engaging in participatory research expands individuals’ sense of what themselves and their
community can accomplish, they will become more active and engaged citizens, challenging the
control of local politics by segments of society that have higher pre-existing levels of efficacy due
to longer legacies of social and political privilege.
The second main channel through which participatory research can lead to
empowerment is by shaping the nature of accountability relationships between citizens,
politicians, and bureaucrats. In theory, citizens have the power to hold both politicians and
bureaucrats accountable for service delivery outcomes. This body of theory identifies two key
accountability relationships: the long route and the short route (Pande and Olken 2013; World
Bank 2004). In the long route of accountability, voters can monitor politicians, and can punish
them for poor service delivery during elections (De Kadt and Lieberman 2017; Lindberg 2010).
Politicians in turn monitor bureaucrats, and can punish them for poor service delivery through
performance evaluations and promotions. In the short route of accountability, citizens directly
monitor bureaucrats, and can punish them by either choosing private service providers or by
directly lobbying street-level bureaucrats or their superiors for better performance (World Bank
2004). This short-route accountability relationship can also result in “institutionalized
coproduction” arrangements in which citizens, government bureaucracies, and civil society
organizations work together to provide public services (Joshi and Moore 2004).
The ability of citizens to exercise power through both of these accountability relationships
require that citizens have information about service delivery outcomes, that they know which
! 8
politicians and bureaucrats have responsibility for service delivery, and that they are easily able to
take advantage of opportunities to exert influence over politicians and bureaucrats, both through
elections and through one-on-one interactions. As a result, participatory research can help to
empower marginalized citizens by transforming accountability relationships in a number of
different ways. When participatory research projects focus on recording citizen experiences with
service delivery outcomes, this can create common knowledge of government failures, increasing
the ability of citizens to hold politicians and bureaucrats accountable for poor outcomes (Birner
and Sekher 2018). When participatory research projects are focused on mapping the
stakeholders connected to a given problem or project, citizens become better informed about
which government officials are responsible for what kinds of service delivery (Cheezum et al.
2013). Finally, participatory research can increase interactions between citizens, politicians,
bureaucrats, and civil society organizations by bringing the government officials into contact with
citizens at planning or dissemination meetings for participatory research projects (Israel et al.
2010; Minkler et al. 2008).
Research Questions
Despite the prevalence of participatory research in many corners of international
development practice and related social science disciplines, there is limited quantitative evidence
about the ability of participatory research to empower individuals by increasing efficacy and
transforming accountability relationships. This lack of quantitative evidence on the effects of
participatory research is in part due to the dual normative and methodological origins of
participatory research. Because participatory research is rooted in the qualitative research
tradition, scholars who have contributed to the development of these methods have not typically
been inclined to design randomized evaluations that run alongside participatory research or which
actually evaluate participatory research itself. Most attempts to evaluate participatory research as
an empowerment intervention come from the US-focused literature on Community-Based
Participatory Research in public health. Most of these evaluations are qualitative, as in the focus-
group based assessment of the Neighborhoods Working in Partnership project in Detroit
(Cheezum et al. 2013; Israel et al. 2010), the mixed-methods observational evaluation of a HIV
! 9
education project with low-income Latina women in Los Angeles (Flaskerud and Nyamathi 2000),
and a case-study based evaluation of the effects of participatory research on policy advocacy
(Minkler et al. 2008).
On methodological grounds, scholars have argued that field experiments and quantitative
field research run the risk of oversimplifying the kinds of hidden and hard-to-observe social
dynamics that participatory methods are designed to uncover (van der Riet 2008). Participatory
research scholars have also opposed quantitative evaluation on normative grounds, arguing that
randomized evaluations and other modes of quantitative research run the risk of perpetuating the
forms of hierarchy, exclusion, and disempowerment that participatory research explicitly aims to
reverse (Sheely 2016).
While these tendencies pushing participatory research away from the practice of
randomized evaluations are not intrinsically problematic, they have meant that the methodology
has remained separate from the mainstream of international development practice and
scholarship, as both have turned towards quantitative evaluation. In development practice, this
shift has been driven in large part by an increasing emphasis on evidence-based policy, which
entails assessing the effectiveness of interventions and programs using Randomized Controlled
Trials (RCTs), and in reallocating funds from what doesn’t work to what does (Duflo and Kremer
2005; Rodrik 2009). This trend towards increased use of quantitative evaluation in development
practice has taken place alongside the spread of these tools in both development economics and
political science. In these fields, randomized experiments have been used to study a variety of
topics including information and accountability, the effectiveness of community driven
development, and efforts to build state capacity and resilience in weak states (Humphreys and
Weinstein 2009; Moehler 2010).
This gap between the branch of international development that utilizes participatory
research and the branch that works with quantitative evaluations prompts three broad families of
empirical questions about the effects of participatory research. The first broad empirical question
is “Does participatory research have a positive effect on individuals’ attitudes and capabilities
! 10
regarding research?”. This broad question encompasses a family of five sub-questions that
capture the various ways that individuals can engage with a research project (Table 1).
The second broad empirical question is focused on the hypothesized connection between
participatory research and efficacy (Table 2). Does participatory research increase individuals’
sense of efficacy? This broad question includes a family of questions focused on each on each of
the dimensions of efficacy.
The third broad empirical question is focused on the hypothesized connection between
participatory research and accountability (Table 3). Does participatory research increase
individuals’ ability to hold service providers accountable for their performance? This broad
question includes a family of questions focused on two aspects of accountability. First, does
participatory research increase citizens’ voice with and accountability over service providers?
Second, does participatory research change citizens’ assessment of how service providers are
doing their jobs?
Assessing Participatory Research- The Participant Driven Evaluation Field Experiment
The Participant Driven Evaluation Program
To answer these empirical questions about the relationship between participatory
research and empowerment, I worked with the staff of a Kenyan NGO called the SAFI Project to
design, implement, and evaluate a participatory research program. SAFI is based in Laikipia
County in north-central Kenya, and had previously implemented programming focused on
community participation in waste management projects and had evaluated those programs using
randomized controlled trials and surveys.
SAFI ‘s staff and I worked collaboratively to design Participant Driven Evaluation (PDE), a
curriculum and training program that combined high levels of participation in pedagogy, a medium
level of participation in organizational structure, and training in methods used in both participatory
research and quantitative evaluation.
The participatory pedagogy emphasized a wide array of interactive, learner-centered
teaching techniques, including case discussions, role-plays, simulations, and field-based practice
activities. The project’s organizational structure built on the program’s participatory pedagogy by
! 11
having the planned participatory research program focus on working closely with a small group of
individuals in each program village. These community researchers were nominated by their fellow
community members to participate in the participatory research program, with the expectation
that this group would form a Community-Based Organization (CBO) to conduct research on
locally-generated topics, and would report back to the community regularly on the results of their
research. Within each of these CBOs, decisions about what research questions to ask and what
methods to use were to be made collectively, and actual research activities were to be carried out
collaboratively.
This focus on creating a highly participatory organizational structure at the community
level was counterbalanced with a slightly more conventional structure for the broader PDE
research project. While SAFI’s staff and I worked closely with community members and other
stakeholders while designing PDE, we still had a disproportionate influence on the final design of
the program, and of the methods and indicators that we would use to assess it.
The medium level of participation in methods entailed combining participatory methods
with other qualitative methods, as well as quantitative methods. This mixed-methods approach
extended to both the content of the participatory research program, as well as the evaluation of
the program. As will be discussed in more detail below, we decided to evaluate the program pilot
using a mix of a randomized controlled-trial and reflexive, qualitative monitoring of the
implementation of the project itself. Within the PDE curriculum, we included training in a wide
range of methods and topics, from participatory tools such as mapping and collaborative scoring,
to qualitative methods such as interviewing and focus groups, to quantitative tools such as
surveys, graphs, randomized experiments, and non-experimental evaluation techniques.
The PDE intervention is made up of a series of three participatory workshops– Village
Research Workshops, Follow-up Village Training Workshops, and Research Conferences, as
well as 3 research opportunities– two Village Research Projects, and a Combined Research
Project. The contents and aims of each of these workshops and research projects are
summarized in Table 4. More detailed workshop materials are available in the online
supplemental appendix.
! 12
Implementing PDE- Structure, Plans, and Deviations
To implement the PDE Intervention, SAFI needed to recruit a field team of facilitators and
enumerators who had familiarity with both participatory development and social science research.
Because SAFI had been working at the intersection of community mobilization and research for 5
years, it had a large network of current and former employees that it could draw on when hiring
the PDE Implementation team. This team was supervised by one of SAFI’s project coordinators
and an international research manager.
While SAFI was able to recruit a sufficient number of staff members that were qualified to
implement the PDE program, the intensive nature of community interactions required to
implement each of the steps of PDE meant that it was impossible to implement the program
across the 32 villages in its planned project area all at once. As a result, the SAFI management
team decided to implement the intervention in two phases, with 16 villages included in the Phase
I implementation and 16 villages in the Phase II implementation.
In addition to necessitating a phased program roll-out, this program design also shaped
the organizational structure and team dynamics. Because the PDE team was a separate unit
within SAFI and because the project necessitated direct, sustained work in a dispersed cross-
section of communities, the individual facilitators and enumerators generally worked independent
of substantial oversight from SAFI. At the same time, because much of the fieldwork for the
project involved working in smaller teams, the PDE staff worked closely with each other, and
developed a sense of collective purpose and autonomy around the mission of the PDE
intervention that was separate from SAFI’s organizational mission.
As a result, the PDE team developed a sense of autonomy and collective identity, both as
a whole, and in the individual teams that were working with each set of villages. Near the latter
stages of the first phase of the PDE program, the staff decided to form their own Community-
Based Organization (CBO), CREED, to continue the PDE work after the end of the research
project. However, the development of organizational autonomy that led the PDE project team to
found CREED also created a new set of intra-organizational conflicts that ultimately impacted the
implementation of the PDE pilot and the viability of the new organization.
! 13
As a result of these conflicts, one group of PDE facilitators and enumerators found itself
excluded from the head of the new organization. This subset of the organization decided to
undermine and oppose the new leadership team by starting to shirk their duties in the planned
PDE Research Workshops, Follow-Up Trainings, and Research Conferences. They did this by
reducing the number of hours and days below the planned number, giving themselves extra days
off. This malfeasance also extended to the PDE evaluation activities, as the enumerators in this
group falsified questionnaires for the endline survey in the villages where they were deployed.
Shortly after the completion of the endline survey, one of the members of this faction
came forward to the management team to disclose the nature of the malfeasance. This
confession led to an investigation into the implementation of the program and the survey. During
this investigation, my research assistant and I interviewed all of the SAFI staff members and
CREED members that had been involved in the PDE project and were able to reconstruct both
the conflicts that preceded the organizational collapse, as well as the ways in which this
malfeasance shaped the duration and depth of the PDE program in each of the project villages.
As a result of this investigation, we decided to conduct the endline survey and Round II
workshops with a new research manager and the facilitators and enumerators who were cleared
of any wrongdoing in the investigation. The leadership of CREED decided to disband the
organization, given that this incident compromised its reputation and depleted many of the
resources and personnel that they had planned to use to begin operations.
This incident poses challenges for the empirical task of evaluating the data from the PDE
pilot evaluation. I discuss these analytic challenges and the empirical strategy that I use to
respond to these challenges in more detail below. A more detailed, reflexive account of this story
and an analysis of its connection to larger issues around the politics of participatory research
projects and randomized controlled trials can be found in a second paper (Sheely 2018).
Evaluation Design
Because of the necessity of implementing the PDE intervention in two phases, it was
possible to use a randomized phase-in field experiment to test the effect the PDE intervention on
the attitudes and behavior of participants and communities. The sample for this evaluation is a
! 14
randomly selected set of 32 villages in Kenya’s Laikipia East and Central Districts. 16 of these
villages were randomly assigned to the first wave of PDE workshops and 16 were assigned to the
second wave of workshops. The random assignment of villages to roll-out phases makes it
possible to conduct surveys before and after the first roll-out, which in turn makes it possible to
treat the second roll-out group as a control group for the purpose of evaluating the impact of the
PDE intervention (Table 5).
Data
Two types of data were collected to evaluate the system of Participant-Driven Evaluation:
1) quantitative data collected during baseline and endline surveys and 2) qualitative data
collected through interviews and observations during all phases of the project. The quantitative
data consists of the baseline and endline surveys. Baseline and endline surveys are administered
to 33 randomly selected community members in each village as well as to the workshop
participants and the village elder. Community members were sampled by working with village
elders to create a list of all adult residents in all households in the village, and then sampling
individuals from those lists.
The surveys measure indicators associated with each of the research questions outlined
above. The online appendix provides a list of the indicators used to answer each research
question. Table 6 provides summary statistics for a number of other characteristics of the
surveyed individuals, along with a test of the balance of these covariates across the two
treatment groups.
Detailed qualitative data were recorded by trained observers during the workshops,
follow-up meetings, and research conferences. In this paper, qualitative observations are used
sparingly to illustrate and supplement the quantitative analysis of the experimental data. More
detailed qualitative analyses are the focus of a second article (Sheely 2018).
Empirical Strategy
The choice of the core econometric model for analyzing the results of the PDE field
experiment is driven by several elements of the project. As noted above, one aspect of the
experiment that poses a particular analytic challenge is unevenness in the implementation of the
! 15
PDE workshops, trainings, and conferences, due to the internal conflict and malfeasance within
the enumerator and facilitator teams.
Given that the hypothesized effects of the PDE intervention are premised on members of
treatment communities participating in the training workshops, interacting with the PDE staff, and
conducting research projects in their communities, the effects of assignment to the treatment
group on the outcomes of interest will depend on the actual amount of time that they spent in
each PDE activity. During the investigation that followed the discovery of the research
malfeasance, my research fellow and I conducted interviews with each of the members of the
PDE team. I then used these interviews to reconstruct measures of the amount of hours spent on
each PDE program activity in each treatment village, as well as facilitator assessments of the
extent to which participants in each village understood the workshop material. Measures of the
number of hours spent on the village research workshops and facilitator assessments of
participant understanding are available in 13 of 16 treatment villages. Measures of the number of
hours spent on each of the follow-up trainings are available in 9 of 16 treatment villages.
Measures of the number of hours spent in the combined research conferences are available in all
16 treatment villages.
Given this unevenness in the implementation of the project, it is likely that simply
estimating the intent-to-treat (ITT) effect will lead to a downward-biased estimate of the effect of
PDE. As a result, the primary estimator of the treatment effect is the effect of Treatment on the
Treated (TOT), which is estimated via two stage least squares (Angrist and Pischke 2008; Bloom
1984). In this model, assignment to treatment is used as an instrument for a measure of
treatment intensity, in this case measured as the number of hours the facilitators spent on the
research conference in that village.1 All models include individual-level controls for wealth,
gender, age, education, involvement in community groups, exposure to nearby urban areas and
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1 This measure of treatment intensity is used because of the completeness of coverage across treatment villages. Robustness checks using the alternate measures of treatment intensity are available on request.
! 16
areas outside of Kenya, and the approximate distance between the respondent’s residence and
the nearest urban center/town.2
Another feature of the research design that has an implication for the empirical strategy is
the use of multiple measures to operationalize the broader research questions and outcomes of
interest (Casey, Glennerster, and Miguel 2012). As a result, two methods are used to minimize
the likelihood of false positives generated by multiple testing, both of which build on the structure
of broad groupings of research questions articulated above. The first method for avoiding
inference problems associated with multiple testing is the calculation of Average Effect Sizes
(AES) by normalizing the TOT effects for each of the individual outcomes in a given research
question, and combining them into an aggregate measure of the effect of the PDE program on
the group of outcomes that fall under that research question. Seemingly Unrelated Regression
(SUR) is used to estimate the covariance matrix for each question-level average TOT estimate
(Casey, Glennerster, and Miguel 2012; Clingingsmith, Khwaja, and Kremer 2009). The SUR
model is also used to estimate the TOT effect for each of the individual outcomes that make up
each broader research question (Miguel 2004).
Second, adjusted p-values are calculated using the Benjamini and Hochberg False
Discovery Rate (FDR) (Benjamini and Hochberg 1995; Newson 2012). These adjusted p-values
are calculated for each of the broad research questions, as well as each individual outcome
measure, adjusting for the number of measures used in each set of tests. For the AES model, this
adjustment is for the 10 broad research questions. For the individual outcomes, this adjustment
accounts for the number of individual outcomes in each research question. In the discussion
below, I first consider the Family- and Question-level results from the AES models, then look in-
depth at results for selected individual outcomes, examining each broad family of research
questions one-by-one. In all of the tables, Column 1 presents the results with hypothesis tests
based on the naïve p-values. Column 2 presents the same results with the FDR adjusted p-value.
Presenting these two hypothesis tests side-by-side will aid in interpretation of these exploratory
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!2 All models are estimated using endline data; robustness checks using difference-in-differences models are available on request. !
! 17
results (McDonald 2009). When both p-values are below conventional thresholds for statistical
significance, this can be interpreted as evidence for a targeted effect of PDE implementation on
that disaggregated outcome. When only the naïve p-value is statistically significant, it is not
possible to make an inference about the targeted impact of PDE on that outcome. However, this
type of result is helpful in identifying which specific outcomes may merit more in-depth exploration
in future studies.
Results
Average Effects of Implementing the PDE Program
Table 7 shows results of the Average Effect Size TOT models, summarizing the overall
effects of the PDE intervention on each of the three broad families of research questions and on
each of the more specific questions that make up each the larger family.
These results show that for the first broad research question, “Does participatory
research have a positive effect on individuals’ attitudes and capabilities regarding research?”, the
answer is strongly positive. Examining each of the research questions within this family indicates
that the intervention had a positive impact on each of the underlying domains of attitudes and
behavior towards research. In particular, the PDE intervention increases individual participation in
research, improves their understanding of research, increases their positive associations towards
research, increases individuals’ desire for community involvement in research, and increases
individuals’ expectations that they will benefit from research. The average effect for participation
in research is significant at the .01 level, while the average effects for the other measures are
significant at the .05 level. With the FDR adjustment, each of these average effects is significant
at the .10 level.
The average effect size for the second broad family of research questions indicates that
the PDE intervention may have had uneven effects on efficacy. In particular, the training appears
to have had a significant positive average impact on individuals’ sense of individual efficacy—the
extent to which they feel that they are capable of accomplishing goals. A one hour increase in
treatment intensity is associated with an .032 standard deviation increase in the average measure
of individual efficacy. This coefficient is significant at the .05 level using the naïve p-value, and at
! 18
the .10 level using the FDR adjusted p-value. In contrast, the PDE intervention does not appear
to have had a significant average effect on the other dimensions of efficacy.
Finally, looking at the two research questions that make up the third family, it appears
that this average effect of the PDE intervention on accountability relationships is also uneven. A
one hour increase in treatment intensity leads to a .031 standard deviation increase in average
reported stakeholder effectiveness. As above, the null hypothesis can be rejected at the .05 level
using the naïve p-value and at the .1 level using the FDR adjusted p-value. In contrast, there is
not a significant effect of the PDE intervention on individuals’ perceived voice over stakeholders.
Taken together, these AES results indicate that the PDE intervention had a large,
significant, and multidimensional impact on the most narrow set of outcomes of interest: attitudes
and behavior towards research. This indicates that at its core, PDE succeeded as a participatory
research intervention, given that it increased individuals’ capacity and willingness to engage in
research projects. The impact of the PDE intervention on both dimensions of empowerment—
efficacy and accountability—appears to be more mixed, with the AES results showing significant
impacts on some research questions, but none on others. In order to better understand and
interpret these results, I will next examine the effects of the PDE intervention on selected
individual outcomes within each family of research questions. I also supplement these
disaggregated statistical analyses with qualitative evidence from the observation of the PDE
activities. Full results for all of the outcome measures are available in the online supplemental
appendix.
The PDE Intervention and Attitudes/Behavior Towards Research
Table 8 shows the disaggregated results for the six outcomes that make up Question 1 in
the first broad family of questions: “Does the implementation of participatory research change the
way that people participate in research?” For three of the six measures included in this question,
the null hypothesis cannot be rejected. Increases in the total number of research conference
hours are not associated with statistically significant increases in the likelihood that a a field
research project has happened in their village or that individual has been asked questions as part
of a field research project. Overall, this set of results indicates that the PDE intervention does not
! 19
have a significant effect on individuals’ perceived likelihood of participating passively in externally-
initiated research projects.
The second set of outcomes associated with this question assess individuals’ self-
reported involvement in more active forms of participation in research projects. Rows 3 and 4
indicate that the implementation of PDE has a positive impact on individuals’ frequency of being
employed by a research project and participating in a research training or workshop. Rows 5 and
6 indicate that the intensity of PDE research conference implementation has a significant effect
on individuals’ likelihood of reporting that other people in their village have started their own
research projects, but not on the likelihood that the respondent had joined a research project
started by other members of the village.
Table 9 shows the disaggregated results for the outcomes that address Question 2:
“Does the implementation of participatory research change peoples’ understanding of research?”
These measures include three questions that ask individuals to assess their own understanding
of the purposes and methods of research, and assessments of individuals’ ability to answer four
different questions focused on the application of research concepts (Table 10). The results in
Column 1 of Table 9 indicate that the significant average effect of PDE implementation on
research understanding may be driven by four individual outcomes: self-reported understanding
of the purposes of research, self-reported understanding of research results, and individuals’
willingness to answer last two assessment questions described in Table 10, which focus on using
research in problem solving and understanding the concept of sampling. However, the results in
Column 2 of Table 9 indicate that it is not possible to reject the null hypothesis of no effect for
each of these individual measures of research understanding after using the FDR adjustment.
Table 11 shows the disaggregated results for a subset of the outcomes that make up
Question 3: “How is the way that people feel about research shaped by the implementation of a
participatory research project?” These results indicate that the significant average effect of the
implementation of PDE on attitudes towards research may be driven primarily by effects on 9 of
the 26 outcomes included in this question (Column 1), but that none of these coefficients remain
significant after adjusting for multiple testing (Column 2). As with Hypothesis 2, these results
! 20
provides some indication of relationships between participatory research and specific attitudes
about research that could be explored in future research, but not enough evidence to make
inferences about targeted effects of the intervention.
Table 12 shows the disaggregated results for four of the eleven outcomes that help to
answer Question 4: “ How does implementation of participatory research shape peoples’ feelings
about the desired level of community involvement in research?” These four outcomes measure
respondents’ assessments of the desired level of community involvement in the four key steps of
policy analysis research: designing programs, implementing programs, evaluating the
effectiveness of programs, and drawing conclusions and making recommendations.
Rows 1 and 2 show that the implementation of the PDE intervention has a positive effect
on desired community involvement in two domains: designing and implementing community
development programs. Rows 3 and 4 show that the implementation of PDE also had a
significant effect on individuals’ assessment of the desired level of community involvement in
evaluating the effectiveness of a community development program or in drawing conclusions and
making recommendations about the program. The implementation of PDE also appears to have
had an effect on individuals’ likelihood of saying that professional researchers and academics
should be responsible for conducting field research. However, as with Questions 2 and 3, the
significance of these individual coefficients is not robust to the multiple testing adjustment.
Table 13 shows the disaggregated results for the six indicators that are used to answer
Question 5, “How does the implementation of the PDE curriculum shape people’s expectations
about research?” These results are driven primarily by the effect of the implementation of the
PDE intervention on individual-level behavior. Row 6 indicates that the implementation of the
PDE intervention has a significant positive impact on individuals’ assessment of the likelihood that
they will organize their own research project within their village. This coefficient is significant at
the .01 level using the naïve p-value and at the .05 level using the FDR-adjusted p-values. Rows
1 and 3 also indicate that the implementation of PDE had an effect on individuals’ expectations
about answering research questions and participating in trainings, but that these results are only
significant when using the naïve p-values.
! 21
Taken together, this set of disaggregated results complement the AES analyses, which
indicated that the PDE intervention had significant and broad effects attitudes and behaviors
towards research. The disaggregated results indicate while there a number of potential
mechanisms that merit further exploration, the strongest targeted effects are associated with the
forms of research engagement that were part of the PDE program: participating in research
workshops and starting locally-generated community research projects. In addition, PDE also
appears to have an impact on the ability of individuals to find work as part of a research project.
These patterns from the disaggregated experimental data are supported and deepened
by qualitative observations from the implementation of the PDE project. Workshop notes indicate
that participants exhibited not only a high level of interest across the board, but also a high level
of understanding of the materials and concepts taught. Although literacy was not a prerequisite to
join the workshop, facilitators observed a tendency for the illiterate participants to look to more
literate participants for leadership and guidance. Similarly, literate participants also tended to be
more confident in answering questions and explaining difficult concepts to fellow participants.
In general, facilitators observed a high level of sophistication with regards to identifying a
clear research question in line with the community’s priorities. While some research topics dealt
with general issues like understanding why there is poverty in the village or identifying a solution
for the bad hygiene in the village, some research topics dealt with very specific issues like
identifying a solution for the domestic fighting in the village that impedes children’s performance
in school, or understanding how to solve the problem of people not wanting to pay for their water
bills.
In the 16 Phase 1 villages, 20 initial research and 15 more follow-up research projects
were conducted, for a total of 35 village-based research projects. Of the village research projects,
most were aimed at identifying solutions to problems within the community. Examples of these
types of project include include identifying solution to infestation by jigger bugs (a type of insect
that causes skin infections) problem in the village and understanding how to work as a community
to reduce the cost of access to water. A small portion of research projects focused on evaluating
existing solutions to problems, such as assessing whether or not planting trees can solve the
! 22
water pollution problem in the village and evaluating a project that employs youth to fix the road.
The PDE Intervention and Efficacy
Table 14 shows the disaggregated results for the 8 outcomes that measure Question 6:
“Does implementation of participatory research increase individual efficacy?” These measures
assess individuals’ assessment of their ability to accomplish goals, succeed at endeavors, and
overcome challenges. These results indicate that it is possible to reject the null hypothesis of no
effect of the implementation of the PDE program on 5 of the 8 these individual measures of
efficacy, and that all of these results are robust to using the more conservative FDR-adjusted p-
values. These results indicate that the implementation of PDE had a broad and consistent effect
on individuals’ sense of what they can accomplish, with the exception of cases in responding to
extreme hardships (Rows 2 and 8).
Question 7 asks “Does Implementation of participatory research increase political
efficacy?” The indicators associated with this research question assess individuals’ assessments
of their own qualifications for participating in political decision-making and leadership in the
village, their understanding of political issues, and their belief in the attentiveness and
responsiveness of leaders them. The results indicate that it is not possible to reject the null
hypothesis for any of these individual measures, other than “I consider myself well qualified to
participate in decision-making in the village,” which is significant at the .10 level using the naïve p-
value, but is not significant using the adjusted p-value. This finding is consistent with the
corresponding finding in the AES analysis, which indicated that there was not a statistically
significant effect of the PDE evaluation on Political Efficacy.
Table 15 shows the disaggregated results for a subset of the 18 outcomes associated
with Question 8- “Does Implementation of participatory research increase collective efficacy?”
These measures ask individuals’ to assess their community’s ability to enact laws and work
together, both in general, and on particular kinds of collective action, such as preserving the air
and water, sanitation, and other kinds of local infrastructure. The null hypothesis can be rejected
using the naïve p-value for six of these measures, which focus on the community’s ability to enact
fair laws, assist economically disadvantaged members of the community, resolve crises, improve
! 23
quality of life in the community, and improve physical conditions in the community. The effect of
implementing the PDE intervention is not significant for any of these outcomes when using the
FDR adjusted p-value.
Taken together, these results indicate that the PDE intervention had a positive impact on
the overall index of individual efficacy, as well as on most of the individual outcomes associated
with that index. They also indicate that while there are some possible dimensions of the
connection between participatory research, political efficacy, and collective efficacy that may be
worth exploring in the future, there is not evidence of either general or targeted effects on these
aspects of empowerment. This gap between effects on individual efficacy and political/collective
efficacy may be shaped in part by the design of the program, which focused on training and
supporting a subset of the residents of a given village. While the PDE participants were
nominated by the rest of their community and were expected to report back about mobilizing
action based on research findings, the qualitative reports indicate that that the training may have
inadvertently emphasized individual empowerment at the expense of collective empowerment. An
excerpt from Research Conference observation notes illustrates this dynamic, recording the
following interaction:
When going through a case study, the facilitator describes a hypothetical situation in which a researcher wrote to a newspaper editor in order to point out a mistake in a published research report. One participant commented ‘it is like for us, if we see errors in a research, we can be able to correct (them).’ Another agreed, saying ‘Yes, there is a difference between us and those who have not studied (research).’ The whole class agrees.
The PDE Intervention and Accountability Relationships
Tables 16,17, and 18 show the disaggregated results for a selection of the outcomes that
are used to answer Question 9: “Does implementation of participatory research change the
amount of voice that individuals have over the various civil society and government organizations
in their community?” Each of these measures assesses an aspect of respondents’ assessment of
their voice and accountability over seven types of service provider that operate in their village:
NGOs, CBOs, religious groups, researchers, the county council, and government chiefs and
assistant chiefs. The specific dimensions that are measured for each service provider include
! 24
whether the actor involves the community in its official activities, whether the stakeholder takes
the respondent’s thoughts and opinions seriously, whether individuals feel that they can make a
difference in how the stakeholder works, and whether the community speaks up when the
stakeholder is not doing well.
As noted above, it is not possible to reject the null hypothesis of no average effect of the
implementation of the PDE intervention on citizen voice over service providers. The results show
that broad interpretation also holds for each of the stakeholders and dimensions of voice and
accountability, as treatment intensity does not have a significant effect on any of the outcomes
when using the FDR adjusted p-value (Column 2). However, the results that the naïve p-values
(Column 1) indicate that there are some possible effects that are worth further exploration. Tables
16 and 17 show that for two measures of voice—community involvement in the stakeholder’s
activities and stakeholder engagement with the respondent’s thoughts and opinions—these
effects are focused on four stakeholders—NGOs, CBOs, chiefs, and assistant chiefs. For NGOs,
CBOs, and chiefs, the implementation of the PDE intervention led to an increase in perceptions of
both kinds of voice. For assistant chiefs, the effect appears to be focused on community
involvement.
There are also possible effects of the PDE intervention on the ability to hold stakeholders
accountable by sanctioning poor performance. Table 18 shows that for the two local
bureaucrats—chiefs and assistant chiefs—implementing the PDE program is associated with an
increased likelihood of individuals speaking up to improve performance. Similarly, for the
assistant chief, implementation of PDE increased the individuals’ agreement with the importance
speaking up against poor performance. In contrast, there is no evidence of a relationship between
the PDE intervention and perceived ability to influence these stakeholder’s actions. There is also
no evidence to suggest that the implementation of the PDE intervention led communities to have
greater voice over or ability to sanction the other actors—religious groups, researchers, and the
county council. As with the other outcomes associated with this research question, these
coefficients are significant when using the naïve p-values, but not with the FDR adjusted p-
values.
! 25
This pattern of the PDE intervention shaping voice with CBOs, NGOs, chiefs, and
assistant chiefs is echoed by several patterns in the qualitative evidence. The strength of the
effect of the PDE intervention on community engagement with CBOs is explained in part by the
fact that the PDE workshop program itself involved creating village-based research organizations
that were registered as CBOs. This feature of the program helped to educate community
members on the roles of CBOs and by making them members of their own CBO, empowered
them to engage with other CBOs and NGOs working in their community. In addition, many of the
village projects created as part of the PDE workshops focused on projects initiated by NGOs and
CBOs. The community workshops also engaged closely with the chief and assistant chief, who
are local administrators for the Kenyan government. The permission of chiefs and assistant chiefs
was required to implement PDE in each village, and the village research organizations continued
to coordinate with both of these civil servants as they designed and implemented their community
research projects and advocated for policy actions based on the findings from their projects. In
one village, the research team showed their results evaluating a community water dam to the
assistant chief and obtained a grant of 5 million shillings (US$58,000) to expand the dam. A
second village researching the effectiveness of pesticides shared their results with the chief and
District Officer (the chief’s supervisor), who helped them to invite an NGO to teach the community
how to apply pesticides in a way that causes least harm to crops.
Tables 19 and 20 show the disaggregated results for 9 of the 28 outcomes that are used
to answer Question 10: “Does implementation of PDE change individuals’ perceived effectiveness
of the various civil society and government organizations in their community?” These tables
report the effect of the PDE intervention on respondent perceptions about the same set of service
providers discussed in Tables 16 and 17 above, but instead with a focus on four dimensions of
effectiveness: satisfaction with how the stakeholder is working in the community, perception that
the stakeholder will be active in helping to solve the community’s problems, perceived
effectiveness in resolving community issues, and an assessment of whether the community has
needed the help of the stakeholder in the past month.
! 26
The analysis of the average effects indicated that overall, it is possible to reject the null
hypothesis of no effect of the PDE intervention on the perceived effectiveness of service
providers. This average effect is driven by the effect of the intervention on respondent
perceptions of specific dimensions of effectiveness for specific stakeholders, as there is no one
stakeholder whose perceived effectiveness was impacted on each of the four dimensions of
effectiveness measured here.
Table 19 indicates that the implementation of the PDE intervention had a statistically
significant effect on respondents’ perceptions of need for help from all of the service providers:
NGOs, CBOs, religious groups, researchers, the county council, chiefs, and assistant chiefs. For
NGOs, Religious Groups, chiefs, and assistant chiefs, the effect of PDE implementation is also
statistically significant using the FDR adjusted p-value. Qualitative evidence from implementation
of the PDE curriculum indicates that this mode of stakeholder engagement may have been driven
in part by the curriculum’s emphasis on identifying the multiple roles that each of these service
providers play in community. By providing information on what each stakeholder actually can do
and how this scope of action connects to each locally-driven research project, the program may
have helped communities to articulate ways in which they could use the help of each of these
actors.
In addition, Table 20 indicates that the implementation of the PDE intervention may lead
to an increased assessment of the effectiveness of several other stakeholders on three other
dimensions. Rows 1-3 indicate that the implementation of PDE may lead to an increased sense of
satisfaction with how NGOs, CBOs, and Chiefs work in the community. Row 6 indicates that the
implementation of the PDE intervention may lead to an increased sense that the Chief will be
active in helping to solve a community problem. Rows 7-10 indicate that the implementation of
the intervention increased the likelihood of a respondent reporting that NGOs, CBOs, the County
Council, and Chiefs are effective at resolving community issues. None of these effects are
statistically significant using the FDR adjusted p-value.
Taken together, these results indicate that the most significant impact of the PDE
intervention on accountability relationships is on individuals’ perceptions that they need the help
! 27
from a variety of governmental and non-governmental actors in solving local problems. In
particular, the targeted effect of PDE was significant and robust to multiple testing correction for
two types of service provider: local administrators from the central government (chiefs and
assistant chiefs) and non-state service providers (NGOs and Religious Groups). Overall, this set
of effects indicates that if PDE had an effect on accountability relationships, it was primarily on
the “short route” of accountability linking citizens and service providers through competition,
advocacy, and collaboration (Joshi and Moore 2004; World Bank 2004).
In contrast, these results show little evidence of an impact of PDE on the “long route” of
accountability linking citizens to politicians through elections. While the PDE intervention did
appear to lead to an increase in reported need for and effectiveness of the County Council (the
locally elected municipal government), the intervention did not lead to changes in the attitudes
and behaviors required to hold local politicians accountable.
Discussion and Conclusion
This paper presented the results of the first known experimental evaluation of the effect
of a participatory research intervention on empowerment. This gap in knowledge has persisted for
several reasons. Although practitioners of participatory methods have hypothesized that these
methods are capable of acting as an empowerment intervention in their own right, when they are
actually used, they are typically treated primarily as methods that facilitate attempts to design and
evaluate community projects. In addition, researchers and practitioners have typically been
reluctant to conduct quantitative evaluations of participatory research approaches due to the
divergent normative, epistemological, and methodological orientations of the two approaches to
research. As a result, even when studies do treat participatory research as an intervention—as in
the literature on Community-Based Participatory Research in public health—these studies
conduct qualitative and participatory evaluations, rather than mixed-methods evaluations that
incorporate randomized controlled trials (Minkler et al. 2008).
As a result, this study makes a contribution to the literature and community of practice
focused on participatory research by combining an intervention that incorporates participatory
pedagogy, organizational structure, and methods with quantitative evaluation using a randomized
! 28
controlled trial and a sample survey. The results of the PDE field experiment show that this
particular type of participatory research intervention was broadly effective at changing attitudes
and behaviors towards research, and had more targeted effects on efficacy and accountability
relationships. Examining the relationships between the implementation of the intervention and the
individual outcomes indicates that PDE was particularly effective at changing individuals’ own
sense of agency regarding research, interactions with stakeholders, and their perceived ability to
accomplish their own goals. In addition, there is evidence that the implementation of PDE did
empower individuals to use research to engage with community organizations and local civil
servants in new ways. These experimental results are consistent with findings of previous
qualitative evaluations of participatory research in low-income communities in the United States,
which also reported positive effects of interventions on engagement with research, individual
empowerment and policy advocacy behaviors (Cheezum et al. 2013; Flaskerud and Nyamathi
2000).
However, there is not evidence that the individual empowerment fostered by the PDE
program translated to community-level empowerment in the form of either viewing research as a
public good or in changing individuals’ assessments of their ability to collaborate with their
community members or to sanction service providers for poor performance. The finding is
consistent with Casey, Glennerster, and Miguel’s experimental evidence from Sierra Leone that
shows that a participatory development project had a significant effect on the provision of
infrastructure, but no significant effects on deeper social norms shaping collective action (Casey,
Glennerster, and Miguel 2012). The finding that PDE contributed to individual empowerment
without necessarily contributing to collective empowerment is also in dialogue with Faranak
Miraftab’s ethnographic evidence about how an empowerment-oriented social enterprise project
in South Africa increased the social and economic power of a small set of program participants
relative to the rest of the members of their communities (Miraftab 2004).
Rather than being the last word on empirical questions about the relationship between
participatory research and various forms of empowerment, the results of this exploratory study
should serve to motivate further collaboration and dialogue among the diverse communities of
! 29
practice focused on participatory research and evidence-based policy in international
development and beyond. While the results presented here do provide evidence about how the
PDE evaluation shaped empowerment, there are several limitations of this study that should be
addressed in future research. First, the self-reported survey measures of empowerment used in
this study should be supplemented with behavioral measures such as behavioral games,
structured social observations, or electoral data that measure efficacy and accountability in more
natural settings. Second, the PDE experiment primarily focused on short-run changes, due to
implementation timelines imposed by donors and the partner’s own programming schedule. As a
result, it is not known whether the observed effects dissipated or deepened over time.
Third, the internal conflict within the PDE and CREED teams both highlighted how
research projects themselves can become politicized (Dionne 2014; Sheely 2016, 2018) and
posed unexpected challenges for data analysis. Future evaluations of participatory research
projects should use this experience as a guide as when engaging with partners and communities,
and should incorporate these types of possible implementation challenges into their research
designs. Finally, the sample size for the PDE field experiment was limited by implementing
partner’s the budget and implementation capabilities. The results here are promising enough to
warrant larger scale experimental assessments of participatory research projects.
Another reason that this study should be the starting point for future evaluations of
participatory research is that more information is needed about how the particular nature and
level of participation in a given project shapes empowerment. The particular mode of participatory
research used in PDE – a highly participatory pedagogy combined with medium levels of
participation in organizational structure and methods—made it possible to combine participatory
research and quantitative evaluation in a novel way. However, it is also possible that the
elements of hierarchy that remained in the organization of the research team and the research
methods may have limited the ability of the intervention to foster collective efficacy and political
empowerment. Future evaluations of participatory research should comparatively assess the
relative effects of alternative interventions that vary with respect to the degree and nature of
community participation in the pedagogy, organizational structure, and methods.
! 30
Future work should also assess whether there is a type of intervention that is both
effective at fostering broad collective and political empowerment and consistent with quantitative
evaluation, or if practitioners and academics in fact face a trade-off between the goals of
participatory research and evidence-based policy. Given the diversity of interests and values
motivating the use of social science research in development, this next cycle of design,
evaluation, and deliberation will inevitably involve disagreements and conflict between and within
academics, policymakers, civil society organizations, and communities. However, engaging in this
difficult and daunting process of ongoing deliberation, collaboration, and compromise between
this diverse set of stakeholders is an important next step in truly realizing the potential for
research to be harnessed in service of empowerment.
!!
! 31
Works Cited Angrist, Joshua D., and Jörn-Steffen Pischke. 2008. Mostly Harmless Econometrics: An
Empiricist’s Companion. Princeton, NJ: Princeton University Press.
Ashby, Jacqueline A., and Louise Sperling. 1995. “Institutionalizing Participatory, Client-Driven Research and Technology Development in Agriculture.” Development and Change 26(4): 753–770.
Asiabaka, C. C. 2002. “Promoting Sustainable Extension Approaches: Farmer Field School (FFS) and Its Role in Sustainable Agricultural Development in Africa.” International Journal of Agriculture and Rural Development 3(1): 46–53.
Bandura, Albert. 2000. “Exercise of Human Agency through Collective Efficacy.” Current directions in psychological science 9(3): 75–78.
Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society. Series B (Methodological): 289–300.
Berg, Marlene, Emil Coman, and Jean J. Schensul. 2009. “Youth Action Research for Prevention: A Multi-Level Intervention Designed to Increase Efficacy and Empowerment among Urban Youth.” American journal of community psychology 43(3–4): 345–359.
Birner, Regina, and Madushree Sekher. 2018. “The Devil Is in the Detail: Understanding the Governance Challenges of Implementing Nutrition-Specific Programs on a Large Scale.” In Hidden Hunger: Strategies to Improve Nutrition Quality, Karger Publishers, 17–44.
Bloom, Howard S. 1984. “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation review 8(2): 225–246.
Bonney, Rick et al. 2009. “Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy.” BioScience 59(11): 977–984.
Braun, Ann R., Graham Thiele, and María Fernández. 2000. Farmer Field Schools and Local Agricultural Research Committees: Complementary Platforms for Integrated Decision-Making in Sustainable Agriculture. ODI London. http://www.researchgate.net/publication/228405399_Farmer_field_schools_and_local_agricultural_research_committees_complementary_platforms_for_integrated_decision-making_in_sustainable_agriculture/file/9fcfd50a6ca112f497.pdf (July 27, 2013).
Brisolara, Sharon. 1998. “The History of Participatory Evaluation and Current Debates in the Field.” New directions for evaluation 1998(80): 25–41.
Brunner, Ilse, and Alba Guzman. 1989. “Participatory Evaluation: A Tool to Assess Projects and Empower People.” New Directions for Program Evaluation 1989(42): 9–18.
Cargo, Margaret, and Shawna L. Mercer. 2008. “The Value and Challenges of Participatory Research: Strengthening Its Practice.” Annual Review of Public Health 29(1): 325–50.
Casey, Katherine, Rachel Glennerster, and Edward Miguel. 2012. “Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan.” Quarterly Journal of Economics 127(4): 1755–1812.
! 32
Catley, Andy, John Burns, Dawit Abebe, and Omeno Suji. 2013. Participatory Impact Assessment: A Design Guide. Somerville, MA: Feinstein International Center, Tufts University. http://fic.tufts.edu/assets/PIA-guide_revised-2014-3.pdf.
Chambers, Robert. 1994a. “Participatory Rural Appraisal (PRA): Analysis of Experience.” World development 22(9): 1253–1268.
———. 1994b. “The Origins and Practice of Participatory Rural Appraisal.” World development 22(7): 953–969.
———. 2008. Revolutions in Development Inquiry. London!; Sterling, VA: Earthscan.
———. 2010. “A Revolution Whose Time Has Come? The Win-Win of Quantitative Participatory Approaches and Methods.” IDS Bulletin 41(6): 45–55.
Cheezum, Rebecca R. et al. 2013. “Building Community Capacity to Advocate for Policy Change: An Outcome Evaluation of the Neighborhoods Working in Partnership Project in Detroit.” Journal of Community Practice 21(3): 228–47.
Clingingsmith, David, Asim Ijaz Khwaja, and Michael Kremer. 2009. “Estimating the Impact of the Hajj: Religion and Tolerance in Islam’s Global Gathering.” The Quarterly Journal of Economics 124(3): 1133–1170.
Conrad, Cathy C., and Krista G. Hilchey. 2011. “A Review of Citizen Science and Community-Based Environmental Monitoring: Issues and Opportunities.” Environmental monitoring and assessment 176(1–4): 273–291.
Cullen, Anne E., and Chris LS Coryn. 2011. “Forms and Functions of Participatory Evaluation in International Development: A Review of the Empirical and Theoretical Literature.” Journal of MultiDisciplinary Evaluation 7(16): 32–47.
Dalton, Timothy J., Nina K. Lilja, Nancy Johnson, and Reinhardt Howeler. 2011. “Farmer Participatory Research and Soil Conservation in Southeast Asian Cassava Systems.” World Development 39(12): 2176–2186.
De Kadt, Daniel, and Evan S. Lieberman. 2017. “Nuanced Accountability: Voter Responses to Service Delivery in Southern Africa.” British Journal of Political Science: 1–31.
Defoer, Toon, A. Budelman, Camilla Toulmin, and S. E. Carter. 2000. “Building Common Knowledge: Participatory Learning and Action Research.” Royal Tropical Institute, Amsterdam, The Netherlands: 207.
Dickens, Linda, and Karen Watkins. 1999. “Action Research: Rethinking Lewin.” Management Learning 30(2): 127–140.
Dionne, Kim Yi. 2014. “The Politics of Local Research Production: Surveying in a Context of Ethnic Competition.” Politics, Groups, and Identities 2(3): 459–80.
Duflo, Esther, and Michael Kremer. 2005. “Use of Randomization in the Evaluation of Development Effectiveness.” In Evaluating Development Effectiveness, eds. George Keith Pitman, Osvaldo Feinstein, and Gregory Ingram. New Brunswick, NJ: Transaction Publishers, 205–31.
Flaskerud, Jacquelyn H., and Adeline M. Nyamathi. 2000. “Collaborative Inquiry with Low-Income Latina Women.” Journal of Health Care for the Poor and Underserved 11(3): 326–342.
! 33
Freire, Paulo, 1921-1997. 1970. Pedagogy of the Oppressed. New York: Seabury Press.
Gallacher, Lesley-Anne, and Michael Gallagher. 2008. “Methodological Immaturity in Childhood Research?: Thinking through `participatory Methods’.” Childhood 15(4): 499–516.
Grillos, Tara. 2015. “Participation, Power and Preferences in International Development.” PhD Thesis.
Humphreys, Macartan, and Jeremy M. Weinstein. 2009. “Field Experiments and the Political Economy of Development.” Annual Review of Political Science 12: 367–78.
Irwin, Alan. 1995. Citizen Science: A Study of People, Expertise, and Sustainable Development. London; New York: Routledge.
Israel, Barbara A. et al. 2010. “Community-Based Participatory Research: A Capacity-Building Approach for Policy Advocacy Aimed at Eliminating Health Disparities.” American journal of public health 100(11): 2094–2102.
Israel, Barbara A., Amy J. Schulz, Edith A. Parker, and Adam B. Becker. 2001. “Community-Based Participatory Research: Policy Recommendations for Promoting a Partnership Approach in Health Research.” Education for health 14(2): 182–197.
Joshi, Anuradha, and Mick Moore. 2004. “Institutionalised Co-Production: Unorthodox Public Service Delivery in Challenging Environments.” Journal of Development Studies 40(4): 31–49.
Kim, Myoung-Sook, and Young-Bae Park. 2008. “The Effects of Self-Efficacy and Empowerment on Perceived Organizational Support and Organizational Citizenship Behaviors of Nurses.” Journal of Korean Academy of Nursing Administration 14(3): 268–277.
Lewin, Kurt. 1946. “Action Research and Minority Problems.” Journal of social issues 2(4): 34–46.
Lindberg, Staffan I. 2010. “What Accountability Pressures Do MPs in Africa Face and How Do They Respond? Evidence from Ghana.” The Journal of Modern African Studies 48(1): 117–142.
Mansuri, Ghazala, and Vijayendra Rao. 2004. “Community-Based and-Driven Development: A Critical Review.” The World Bank Research Observer 19(1): 1–39.
———. 2012. Localizing Development: Does Participation Work? Washington, DC: World Bank Publications.
Martin, Adrienne, and John Sherington. 1997. “Participatory Research Methods—Implementation, Effectiveness and Institutional Context.” Agricultural Systems 55(2): 195–216.
McDonald, John H. 2009. 2 Handbook of Biological Statistics. sparky house publishing Baltimore, MD.
Miguel, E. 2004. “Tribe or Nation? Nation Building and Public Goods in Kenya versus Tanzania.” World Politics 56(03): 328–62.
Minkler, Meredith et al. 2008. “Promoting Healthy Public Policy through Community-Based Participatory Research: Ten Case Studies.” Oakland, CA: PolicyLink.
! 34
Miraftab, Faranak. 2004. “Making Neo-Liberal Governance: The Disempowering Work of Empowerment.” International Planning Studies 9(4): 239–59.
Moehler, Devra C. 2010. “Democracy, Governance, and Randomized Development Assistance.” The Annals of the American Academy of Political and Social Science 628(1): 30–46.
Mukherjee, Neela. 2002. Participatory Learning and Action: With 100 Field Methods. Concept Publishing Company. http://books.google.com/books?hl=en&lr=&id=CPDylQ5_RKAC&oi=fnd&pg=PA7&dq=%22participatory+learning+and+action%22&ots=r4FV-qhj-j&sig=RxgQeMqIOljVhZAMak8blzbVWjw (August 6, 2013).
Newson, Roger. 2012. “QQVALUE: Stata Module to Generate Quasi-q-Values by Inverting Multiple-Test Procedures.” Statistical Software Components. http://ideas.repec.org/c/boc/bocode/s457100.html (August 23, 2013).
Ohmer, Mary L. 2007. “Citizen Participation in Neighborhood Organizations and Its Relationship to Volunteers’ Self-and Collective Efficacy and Sense of Community.” Social Work Research 31(2): 109–120.
Pande, Rohini, and Benjamin Olken. 2013. Governance Review Paper. Cambridge, MA: Abdul Latif Jameel Poverty Action Lab Governance Initiative, MIT. https://www.povertyactionlab.org/sites/default/files/documents/J-PAL_GI-ReviewPaper_vMay-2011.pdf.
Reid, R. J. et al. 2009. “Evolution of Models to Support Community and Policy Action with Science: Balancing Pastoral Livelihoods and Wildlife Conservation in Savannas of East Africa.” http://dash.harvard.edu/handle/1/9774652 (July 29, 2013).
van der Riet, Mary. 2008. “Participatory Research and the Philosophy of Social Science: Beyond the Moral Imperative.” Qualitative Inquiry 14(4): 546.
Rodrik, Dani. 2009. “The New Development Economics: We Shall Experiment, But How Shall We Learn.” In What Works in Development?: Thinking Big and Thinking Small, eds. Jessica Cohen and William Easterly. Washington, DC: Brookings Institution Press.
Sheely, Ryan. 2015. “Mobilization, Participatory Planning Institutions, and Elite Capture: Evidence from a Field Experiment in Rural Kenya.” World Development 67(March 2015): 251–266.
———. 2016. “Regimes and Randomization: Authoritarianism and Field Research in Contemporary Kenya.” Social Science Quarterly 97(4): 936–952.
———. 2018. “Participation, Experimentation, and the Politics of Community-Based Research.”
Wallerstein, Nina B., and Bonnie Duran. 2006. “Using Community-Based Participatory Research to Address Health Disparities.” Health promotion practice 7(3): 312–323.
Wetmore, Stephen B., and Francois Theron. 1998. “Community Development and Research: Participatory Learning and Action-a Development Strategy in Itself.” Development Southern Africa 15(1): 29–54.
World Bank. 2004. World Development Report 2004: Making Services Work for Poor People. Washington, D.C.: World Bank.
! 35
Yeich, Susan, and Ralph Levine. 1994. “Political Efficacy: Enhancing the Construct and Its Relationship to Mobilization of People.” Journal of Community Psychology 22(3): 259–271.
! 36
Family 1: Does Participatory Research have a positive effect on individuals' attitudes and capabilities regarding research? Research Question 1: Does the implementation of participatory research change the way that people participate in research? Research Question 2: Does the implementation of a participatory research project change people's understanding of research? Research Question 3: How is the way that people feel about research shaped by the implementation of a participatory research project? Research Question 4: How does the implementation of a participatory research project shape people's feeling about desired community involvement in Research? Research Question 5: How does the implementation of a participatory research project shape people's expectations about research?
Table 1. Research questions related to the relationship between participatory research and
community behaviors and attitudes towards research.
Family 2: Does participatory research increase individuals’ sense of efficacy? Research Question 6: Does implementation of a participatory research project increase Individual Efficacy? Research Question 7: Does implementation of a participatory research project increase Political Efficacy? Research Question 8: Does Implementation of a participatory research project increase Collective Efficacy?
Table 2. Research questions related to the relationship between participatory research and
efficacy.
Family 3: Does participatory research increase individuals’ ability to hold service providers accountable for their performance? Research Question 9: Does Implementation of a participatory research project change the amount of voice that individuals have over the various civil society and government organizations in their community? Research Question 10: Does implementation of a participatory research project change perceived effectiveness of the various civil society and government organizations in their community?
Table 3. Empirical questions related to the relationship between participatory research and
accountability.
! 37
Village Research Workshops
• 4 days long and held with participants nominated by the community itself
• Each Village Research Workshop engages participants in discussions about the motivations for and applications of research, while teaching them quantitative, qualitative, and participatory methodologies of data collection, project evaluation and data presentation.
• At the end of the workshop, if the participants desire to register as a Community Based Organization, PDE assists the group in registering in order to facilitate follow-up community research projects by the group.
Village Research Project I
• On the last day of the workshop, each group of participants is granted a small research fund (about $50 per project) to conduct a research project of their own, either to identify solutions to a problem, or to evaluate the effectiveness of a project in their community.
• All groups then present the results of their research to the community in order to general community-wide interest and accountability.
Follow-up Village Training Workshop
• Held about a month after the Village Research Workshops with the same participants.
• The aim is to solidify understandings of concepts and skills taught during the Village Research Workshop.
Village Research Project II
• After the Follow-up Village Training Workshop, another round of funding is made available for the village workshop participants to carry out follow-up research projects.
• Groups are encouraged to design the follow-up research based on the findings of their first research.
PDE Research Conference
• After the Follow-up Trainings, Village Representatives from each Treatment village are invited to join one of three Research Conferences (divided by geographical location), where they are taught more advanced methods of research, accounting for omitted variable biases and difference in differences methodologies.
Combined Research Project
• After the Research Conferences, Village Representatives are then given the opportunity to conduct a larger scale project evaluation in collaboration with Village Research Committees from neighboring villages and an NGO or government organization of their choice.
• $100 is granted to every participating village to carry out project implementation, and $100 is similarly granted for conducting project evaluation.
• Village Representatives decide as a group what intervention to evaluate, the evaluation strategy, as well as all financial and logistical arrangements. These research results are then presented during a Final Meeting held after completion of the Phase 2 workshops.
Table 4: PDE Program Components.
! 38
Timeline Activities 16 Phase 1 Villages 16 Phase 2 Villages
May 2011- April 2012 PDE Program Design and Piloting September-October 2012 Baseline Survey
January 2012-May 2013
Village Research Workshop (January-February 2013)
Follow-up Training (March 2013)
Research Conference (April-May 2013)
January-February 2014 Endline Survey
March-April 2014 Village Research Workshops
Table 5. Timeline for PDE Program Design, Implementation, and Evaluation
Covariate Category Full
Sample Control Treatment Likelihood
Ratio P-value N
District Laikipia East 55.8% 55.6% 56.0% 0.00 0.9816 1054 Laikipia Central 44.2% 44.4% 44.0%
Gender Female 55.1% 52.8% 57.4% 1.84 0.1856 1025 Male 44.9% 47.2% 55.1%
Age 18 to 35 39.8% 39.5% 40.2% 0.26 0.7162 1032 36 to 55 37.1% 36.2% 38.1% 56 and above 23.1% 24.4% 21.8%
Religion Christian 61.2% 61.4% 61.0% 0.99 0.3995 1053 Catholic 20.8% 19.3% 22.3%
Tribe Kikuyu 67.4% 71.3% 63.6% 0.76 0.5372 1053 Maasai 12.2% 6.4% 17.9%
Occupation Agriculture 55.0% 59.9% 50.0% 0.97 0.3982 1050 Laborer 15.1% 12.9% 17.2% Homemaker 11.9% 8.1% 15.7%
Table 6. Summary Statistics of Covariates and Balance Checks
! 39
AES Coefficients (1)
AES Coefficients (2)
Family 1 - Research Question 1: Does the implementation of PDE curriculum change the way that people participate in research? 0.015*** 0.015*
(0.01) (0.01) Question 2: Does the implementation of the PDE curriculum change people's understanding of research? 0.023** 0.023*
(0.01) (0.01) Question 3: How is the way that people feel about research shaped by the implementation of the PDE curriculum? 0.019* 0.019*
(0.01) (0.01) Question 4: How does PDE implementation shape people's feeling about desired community involvement in research? 0.021** 0.021*
(0.01) (0.01) Question 5: How does the implementation of the PDE curriculum shape people's expectations about research? 0.020** 0.020*
(0.01) (0.01) Family 2 - Efficacy
Question 6: Does implementation of PDE increase Individual Efficacy? 0.032** 0.032*
(0.02) (0.02) Question 7: Does implementation of PDE increase Political Efficacy? 0.016 0.016
(0.01) (0.01) Question 8: Does Implementation of PDE increase Collective Efficacy? 0.025 0.025
(0.02) (0.02) Family 3- Voice and Accountability (effectiveness)
Question 9: Does Implementation of PDE change the amount of voice that individuals have over the various civil society and government organizations in their community?
0.026 0.026
(0.02) (0.02) Question 10: Does implementation of PDE change perceived effectiveness of the various civil society and government organizations in their community? 0.031** 0.031*
(0.01) (0.01) Robust standard errors clustered at the village level in parentheses *** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2) refers to FDR-adjusted p-values for 10 research questions.
Columns show AES estimates. The AES averages normalized effects of treatment-on-treated obtained from a seemingly unrelated regression in which each dependent variable is an individual survey question that is part of the the broader research question. All results come from IV regressions where the instrument is assignment to treatment, and include individual-level controls for wealth, gender, age, education, involvement in community groups, exposure to nearby urban areas and areas outside of Kenya, and the approximate distance between the respondent’s residence and the nearest urban center/town.
Table 7. Average Effects of Implementation of the PDE Program for Each Broad Research
Question
! 40
(1) (2) Question Coefficient - Conference
total hours Coefficient - Conference
total hours
(1) To your knowledge, has there been field research, besides this one, done in your village in the past year? (Coding: 1=Never, 2=One time, 3=Two times, 4=Three times, 5=Four times, 6=Five times or more)
0.000453 0.000453
(0.00796) (0.00796) (2) In the past year, not counting right now, how many times
have you been asked questions as part of a field research project? (Coding: 1=Never, 2=One time, 3=Two times, 4=Three times, 5=Four times, 6=Five times or more)
-0.00103 -0.00103
(0.00640) (0.00640) (3) In the past year, how many times have you been
employed as part of a field research project? (Coding: 1=Never, 2=One time, 3=Two times, 4=Three times, 5=Four times, 6=Five times or more)
0.0114** 0.0114**
(0.00450) (0.00450) (4) In the past year, how many times have you participated in
a workshop or training as part of a field research project? (Coding: 1=Never, 2=One time, 3=Two times, 4=Three times, 5=Four times, 6=Five times or more)
0.0130*** 0.0130**
(0.00413) (0.00413) (5) To your knowledge, has there been field research
projects started by people in your own village? (Coding: 1=Yes, 0=No)
0.0204*** 0.0204***
(0.00466) (0.00466) (6) If yes, did you participate in them? (Coding: 1=Yes, 0=No) -0.000108 -0.000108
(0.00553) (0.00553)
Observations 1,053 1,053 Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2) refers to FDR-
adjusted p-values for 6 outcomes,
Columns give estimates of the effect of treatment-on-treated from IV regressions where the instrument is assignment to treatment, and which include individual-level controls for wealth, gender, age, education, involvement in community groups, exposure to nearby urban areas and areas outside of Kenya, and the approximate distance between the respondent’s residence and the nearest urban center/town. All results are obtained from a seemingly unrelated regression system in which each dependent variable is an individual survey question.
Table 8. Research Question 1 The PDE Intervention and Reported Research Behavior
! 41
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours
(1) Do you understand why people conduct research? (Coding: 1=Always Confused, 2=Confused most of the time, 3=50-50, 4=Understand most of the time, 5=Understand very well)
0.0236* 0.0236
(0.0122) (0.0122) (2) Do you often understand research? (Coding:
1=Never, 2=Seldom, 3=Sometimes, 4=Often, 5=Almost Always)
0.0208* 0.0208
(0.0106) (0.0106) (3) Do you understand how research is conducted?
(Coding: 1=Always Confused, 2=Confused most of the time, 3=50-50, 4=Understand most of the time, 5=Understand very well)
0.0162 0.0162
(0.0137) (0.0137) (4) Assessment Question 1 (See Table 10 for
details) 0.00582 0.00582
(0.00354) (0.00354) (5) Assessment Question 2 (See Table 10 for
details) 0.00659 0.00659
(0.00397) (0.00397) (6) Assessment Question 3 (See Table 10 for
details) 0.00824** 0.00824
(0.00386) (0.00386) (7) Assessment Question 4 (See Table 10 for
details) 0.00906** 0.00906
(0.00362) (0.00362)
Observations 1,053 1,053 Robust standard errors clustered at the village level in
parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2)
refers to FDR-adjusted p-values for 7 outcomes.
See notes for Table 8 for details on the specification.
Table 9. Research Question 2 The PDE Intervention and Understanding of Research
! 42
Assessment Question Text
Multiple Choice Options (Correct Answers for “Easy” Grading in Bold)
(1)
Research has shown that pill A is 95% effective in soothing headaches, and pill B is 75% effective in soothing headaches. Last week, your neighbor had a bad headache and took pill B. She felt better immediately and strongly recommends pill B. Imagine that you have a headache, would you take pill A or pill B, assuming they are both cheap and easily available?
0 = Don’t know 1 = Neither 2 = Pill B 3 = Pill A
(2)
Imagine that your community has two water sources – piped water and well water. The piped water is clear and the well water is murky, so you have always used piped water for drinking and well water for cleaning. A research organization comes to your community to test the two water sources. To your surprise, their research shows that the piped water is more likely to be contaminated than the well water. The research organization can do another test for a fee. What would you do?
0 = Don’t know 1 = Decide tomorrow 2 = Contribute some money to get the research organization to perform more rounds of testing 3 = Switch to filtering the well water for drinking 4 = Continue using the piped water for drinking
(3)
Your community is facing a problem regarding sanitation. However, no one knows for certain how to solve the problem. For a consultation fee, your community can hire researchers to come and research on the cause and possible solutions for this problem. Do you think your community should hire the researchers?
0 = Don’t know 1 = No 2 = Yes
(4) Suppose you are doing a research to find out the presidential candidate that is preferred by most people. You want to come up with the best way to collect data that will ensure that is not biased. Which of these ways would you use?
0 = Don’t know 1 = Randomly select 50 individual participants and interview them 2 = Randomly select 5 families to participate. From these families, select 10 members from each family to interview
Table 10. Research Question 2
Full Text of Questions and Answers Assessing Understanding of Research
! 43
(1) (2) Question Coefficient - Conference
total hours Coefficient - Conference
total hours (1) How would you describe your interest in research?
(Coding: 1=Not Interested At All, 2=Slightly Interested, 3=Moderately Interested, 4=Very Interested, 5=Extremely Interested)
0.0377** 0.0377
(0.0147) (0.0147) (2) “Research is a useful tool in solving real world
problems.” Do you agree with this statement? (Coding: 1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree)
0.0209 0.0209
(0.0150) (0.0150) (3) Do you trust research? (Coding: 1=Never, 2=Seldom,
3=Sometimes, 4=Often, 5=Almost Always) 0.0253** 0.0253
(0.0106) (0.0106) (4) How would you describe the impact of research on
people’s daily lives in your village? (Coding: 1=Very Detrimental, 2=Detrimental, 3=No Impact, 4=Beneficial, 5=Very Beneficial)
0.0313** 0.0313
(0.0123) (0.0123) (5) In general, how would you describe the impact of field
research on a person's financial situation? (Coding: 1=Very Negative, 2=Negative, 3=No Impact, 4=Positive, 5=Very Positive)
0.0412** 0.0412
(0.0153) (0.0153) (6) In general, how would you describe the impact of field
research on a person's intellectual experience? (Coding: 1=Very Negative, 2=Negative, 3=No Impact, 4=Positive, 5=Very Positive)
0.0255* 0.0255
(0.0137) (0.0137) (7) If the people in your village conducted a field research,
who do you think will be most interested in its results? (Govt. Officials - Binary variable)
-0.00386 -0.00386
(0.00444) (0.00444) (8) If the people in your village conducted a field research,
who do you think will be most interested in its results? (NGOs - Binary variable)
0.00350 0.00350
(0.00318) (0.00318) (9) If the people in your village conducted a field research,
who do you think will be most interested in its results? (Community Members - Binary variable)
0.00278 0.00278
(0.00400) (0.00400)
Observations 1,053 1,053
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2) refers
to FDR-adjusted p-values for 26 outcomes.
See notes for Table 8 for details on the specification.
Table 11. Research Question 3 The PDE Intervention and Attitudes Towards Research: Selected Survey Questions
! 44
(1) (2)
Question Coefficient - Conference total hours
Coefficient - Conference total hours
(1) On a scale of 1 to 5 (1 involved very
little, 5 is very involved), how involved should the community be in designing a community development program?
0.0423** 0.0423
(0.0178) (0.0178) (2) On a scale of 1 to 5, how involved
should the community be in implementing a community development program?
0.0378** 0.0378
(0.0175) (0.0175) (3) On a scale of 1 to 5, how involved
should the community be in evaluating the effectiveness of a community development program?
0.0344* 0.0344
(0.0178) (0.0178) (4) On a scale of 1 to 5, how involved
should the community be in drawing conclusions and making recommendations about a community development program?
0.0348* 0.0348
(0.0179) (0.0179) Observations 1,053 1,053
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values.
Column (2) refers to FDR-adjusted p-values for 11 outcomes.
See notes for Table 8 for details on the specification.
Table 12. Research Question 4
The PDE Intervention and Community Involvement in Research: Selected Survey Questions
! 45
(1) (2) Question Coefficient - Conference
total hours Coefficient - Conference
total hours
(1) In the next year, do you anticipate being asked more questions as part of a field research project? (Coding: 1=Yes, 0=No)
0.00897* 0.00897
(0.00454) (0.00454) (2) In the next year, do you anticipate
being employed as part of a field research project? (Coding: 1=Yes, 0=No)
0.00631 0.00631
(0.00402) (0.00402) (3) In the next year, do you anticipate
participating in a workshop or training as part of a field research project? (Coding: 1=Yes, 0=No)
0.00572* 0.00572
(0.00333) (0.00333) (4) In the next year, do you anticipate that
there will be field research projects started by people in your own village? (Coding: 1=Yes, 0=No)
0.00350 0.00350
(0.00211) (0.00211) (5) If the people in your village started a
field research project, do you think you will participate in it? (Coding: 1=Yes, 0=No)
0.00454 0.00454
(0.00283) (0.00283) (6) Do you see yourself organizing your
own field research project within this village? (Coding: 1=Yes, 0=No)
0.0104*** 0.0104**
(0.00351) (0.00351) Observations 1,053 1,053
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values.
Column (2) refers to FDR-adjusted p-values for 6 outcomes.
See notes for Table 8 for details on the specification.
Table 13. Research Question 5
The PDE Intervention and Expectations About Research
! 46
(1) (2) Question Coefficient -
Conference total hours
Coefficient - Conference total
hours
(1) I will be able to achieve most of the goals that I have set for myself. (Coding: On a scale of 1 to 5, where 1=Strongly Disagree, 5=Strongly Agree)
0.0458** 0.0458*
(0.0168) (0.0168) (2) When facing difficult tasks, I am certain
that I will accomplish them. 0.0370 0.0370
(0.0237) (0.0237) (3) In general, I think that I can obtain
outcomes that are important to me. 0.0281* 0.0281*
(0.0147) (0.0147) (4) I believe I can succeed at most any
endeavor to which I set my mind. 0.0302** 0.0302*
(0.0139) (0.0139) (5) I will be able to successfully overcome
many challenges. 0.0320* 0.0320*
(0.0171) (0.0171) (6) I am confident that I can perform
effectively on many different tasks. 0.0302* 0.0302*
(0.0149) (0.0149) (7) Compared to other people, I can do
most tasks very well. 0.0352* 0.0352*
(0.0184) (0.0184) (8) Even when things are tough, I can
perform quite well. 0.0304 0.0304
(0.0222) (0.0222) Observations 1,053 1,053
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values.
Column (2) refers to FDR-adjusted p-values for 8 outcomes.
See notes for Table 8 for details on the specification.
Table 14. Research Question 6
The PDE Intervention and Individual Efficacy
! 47
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours
(1) Our community can enact fair laws, even when there are conflicts in the larger society. (Coding: On a scale of 1 to 5, where 1=Strongly Disagree, 5=Strongly Agree) 0.0370* 0.0370
(0.0199) (0.0199) (2) Despite problems with the economy, we can assist the
most economically disadvantaged members of our community.
0.0306* 0.0306
(0.0169) (0.0169) (3) We can resolve crises in the community without any
negative aftereffects. 0.0383** 0.0383
(0.0186) (0.0186) (4) I am convinced that we can improve the quality of life in
the community, even when resources are limited or become scarce.
0.0295* 0.0295
(0.0172) (0.0172) (5) We can ensure that the air and water in our community
are clean. 0.0158 0.0158
(0.0195) (0.0195) (6) We can work together to preserve natural resources in
our community. 0.0271 0.0271
(0.0178) (0.0178) (7) Our community can cooperate in the face of difficulties to
improve the quality of community facilities. 0.0235 0.0235
(0.0171) (0.0171) (8) Despite work and family obligations, we can commit
ourselves to common community goals. 0.0172 0.0172
(0.0153) (0.0153) (9) The people of our community can continue to work
together, even when it requires a great deal of effort. 0.0265 0.0265
(0.0164) (0.0164) (10) We can work together to improve physical conditions in
the community like waste management and sanitation. 0.0321* 0.0321
(0.0170) (0.0170) (11) The members of this community have excellent skills.
0.0305* 0.0305
(0.0161) (0.0161) (12) As a community, we can handle mistakes and setbacks
without getting discouraged. 0.0226 0.0226
(0.0165) (0.0165) Observations 1,053 1,053
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2)
refers to FDR-adjusted p-values for 18 outcomes.
See notes for Table 8 for details on the specification.
Table 15. Research Question 8 The PDE Intervention and Collective Efficacy: Selected Survey Questions
! 48
(1) (2)
Question Coefficient - Conference total hours
Coefficient - Conference total hours
This stakeholder involves the community in its official activities:
(1) NGOs 0.0472** 0.0472 (0.0207) (0.0207)
(2) CBOs 0.0397** 0.0397 (0.0185) (0.0185)
(3) Religious Groups 0.0244 0.0244 (0.0164) (0.0164)
(4) Researchers 0.0291 0.0291 (0.0198) (0.0198)
(5) County Council 0.0103 0.0103 (0.0178) (0.0178)
(6) Chief 0.0397** 0.0397 (0.0166) (0.0166)
(7) Assistant Chief 0.0321* 0.0321 (0.0173) (0.0173) Observations 1,053 1,053 Coding for the above variables: 1=Strongly Disagree,
2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2) refers
to FDR-adjusted p-values for 35 outcomes
See notes for Table 8 for details on the specification.
Table 16. Research Question 9
The PDE Intervention and Individuals’ Voice over Stakeholders: Selected Questions Community Involvement in Stakeholder Activities
! 49
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours This stakeholder takes my thoughts,
opinions and information seriously:
(1) NGOs 0.0314* 0.0314 (0.0168) (0.0168)
(2) CBOs 0.0289* 0.0289 (0.0159) (0.0159)
(3) Religious Groups 0.0124 0.0124 (0.0101) (0.0101)
(4) Researchers 0.0194 0.0194 (0.0150) (0.0150)
(5) County Council 0.000781 0.000781 (0.0175) (0.0175)
(6) Chief 0.0208* 0.0208 (0.0118) (0.0118)
(7) Assistant Chief 0.0178 0.0178 (0.0141) (0.0141) Observations 1,053 1,053 Coding for the above variables: 1=Strongly Disagree,
2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2)
refers to FDR-adjusted p-values for 35 outcomes
See notes for Table 8 for details on the specification.
Table 17. Research Question 9 The PDE Intervention and Individuals’ Voice over Stakeholders: Selected Questions
Stakeholder Consideration of Citizen thoughts, Opinions, and Information
! 50
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours It is important for our community to speak up when this stakeholder is not working well.
NGOs 0.0279 0.0279 (0.0216) (0.0216)
CBOs 0.0239 0.0239 (0.0197) (0.0197)
Chief 0.0266 0.0266 (0.0160) (0.0160)
Assistant Chief 0.0269* 0.0269 (0.0154) (0.0154)
When this stakeholder is not doing well in our village, our community will speak up to try and improve them.
NGOs 0.0357 0.0357 (0.0225) (0.0225)
CBOs 0.0208 0.0208 (0.0205) (0.0205)
Chief 0.0294* 0.0294 (0.0163) (0.0163)
Assistant Chief 0.0300* 0.0300 (0.0171) (0.0171)
Observations 1,053 1,053 Coding for the above variables: 1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2) refers to FDR-adjusted p-values for 35 outcomes
See notes for Table 8 for details on the specification.
Table 18. Research Question 9 The PDE Intervention and Individuals’ Voice over Stakeholders:
Responses to Additional Selected Questions
! 51
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours In the past month, our community has
needed the help of this stakeholder.
(1) NGOs 0.0606** 0.0606* (0.0229) (0.0229)
(2) CBOs 0.0502** 0.0502 (0.0213) (0.0213)
(3) Religious Groups 0.0530*** 0.0530* (0.0189) (0.0189)
(4) Researchers 0.0398* 0.0398 (0.0209) (0.0209)
(5) County Council 0.0441** 0.0441 (0.0199) (0.0199)
(6) Chief 0.0585*** 0.0585* (0.0194) (0.0194)
(7) Assistant Chief 0.0569*** 0.0569* (0.0191) (0.0191) Observations 1,053 1,053 Coding for the above variables: 1=Strongly Disagree,
2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column (2)
refers to FDR-adjusted p-values for 28 outcomes
See notes for Table 8 for details on the specification.
Table 19. Research Question 10 The PDE Intervention and Stakeholder Effectiveness: Selected Questions
Need for Help from Stakeholders
! 52
(1) (2) Question Coefficient -
Conference total hours Coefficient -
Conference total hours I am satisfied with the way this
stakeholder works in our community
(1) NGOs 0.0387** 0.0387 (0.0182) (0.0182)
(2) CBOs 0.0322* 0.0322 (0.0164) (0.0164)
(3) Chief 0.0355** 0.0355 (0.0160) (0.0160) If there is a community problem, this
stakeholder will be active in helping to solve the problem.
(4) NGOs 0.0288 0.0288 (0.0175) (0.0175)
(5) CBOs 0.0285 0.0285 (0.0180) (0.0180)
(6) Chief 0.0373** 0.0373 (0.0159) (0.0159) This stakeholder is effective in
resolving community issues.
(7) NGOs 0.0307* 0.0307 (0.0174) (0.0174)
(8) CBOs 0.0318* 0.0318 (0.0163) (0.0163)
(9) County Council 0.0310** 0.0310 (0.0148) (0.0148)
(10) Chief 0.0300* 0.0300 (0.0169) (0.0169) Observations 1,053 1,053 Coding for the above variables: 1=Strongly
Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree
Robust standard errors clustered at the village level in parentheses
*** p<0.01, ** p<0.05, * p<0.1 Column (1) refers to the naïve p-values. Column
(2) refers to FDR-adjusted p-values for 28 outcomes
See notes for Table 8 for details on the specification.
Table 20. Research Question 10
The PDE Intervention and Stakeholder Effectiveness: Selected Survey Questions.