American Journal of Evaluation 2009 Chouinard 457 94 (1)

Articles

A Review and Synthesis ofCurrent Research onCross-Cultural Evaluation

Jill Anne Chouinard1 and J. Bradley Cousins1

Abstract

As a fairly new and emergent construct, there remain many gaps in our knowledge about how tointegrate notions of culture and cultural context into evaluation theory and practice, as well asgaps in our knowledge about how to conduct and implement evaluations in immigrant andindigenous communities. In this article, the authors provide a comprehensive review of theempirical literature on evaluations conducted in cultural communities, with an emphasis on therelationship between evaluators and stakeholders in the cross-cultural program context. Theanalysis of the literature selected for review leads to the development of a theoretical frameworkdescribing the inter-related and multi-textual dimensions (relational, ecological, methodological,organizational and personal) that interweave throughout the evaluation, and that ultimatelyinform the relationship between evaluators and stakeholders in the cross-cultural programcontext. The article concludes with an agenda for future research.

Keywords

cross-cultural evaluation, research relationships, evaluator role, cultural context

Introduction

Program evaluation is concerned with understanding and improving social programs so that they are

ultimately more responsive to program participant needs. At a very fundamental level, program eva-

luation is a sociopolitical process (Greene, 2000; Guba & Lincoln, 1989) intrinsically related to deci-

sions about societal priorities and resource allocations, as decisions are made concerning program

worth and merit. Within this backdrop, there is increasing awareness that these social programs are

embedded within specific social, cultural, and historical contexts, all of which profoundly affect pro-

gram development, implementation, and outcomes. Evaluations are thus far from being value-free

and culture-free (SenGupta, Hopson, & Thompson-Robinson, 2004), as culture and values permeate

all facets of social programs and their evaluations. Whose voices get heard? Whose interests

1Faculty of Education, University of Ottawa, Ottawa, Ontario, Canada

Corresponding Author:

Jill Anne Chouinard, Faculty of Education, University of Ottawa, 145 Jean-Jacques Lussier, Ottawa, Ontario, Canada, K1N

6N5. Email: [email protected]

American Journal of Evaluation30(4) 457-494 The Author(s) 2009Reprints and permission: http://www.sagepub.com/journalsPermissions.navDOI: 10.1177/1098214009349865http://aje.sagepub.com

457 at University of South Australia on March 22, 2015aje.sagepub.comDownloaded from

dominate? Who asks the questions? Whose knowledge is of most worth? Within the context of an

evaluation, these questions become paramount, not simply as methodological considerations, but as

theoretical and normative constructs guiding evaluative practice.

While still a fairly new and emergent construct, evaluations that are responsive to contextual and

cultural specificity are increasing in frequency, as growing disparities and increasingly multi-ethnic

and multi-cultural contexts globally are creating a heightened awareness of and need for this type of

evaluation. We can trace the early history of this type of evaluation to a small group of African

American researchers and evaluators who, from the 1930s to 1950s, utilized evaluation methods that

were responsive and sensitive to African American experiences during racial segregation (Hood,

2009). More recently, in the 1985 edition of New Directions for Program Evaluation (edited by Pat-

ton), evaluators for the first time explicitly questioned the impact of culture and cultural context on

program evaluation (Hopson, 2003). Almost a decade later, Karen Kirkharts presidential address at

the 1994 American Evaluation Association conference challenged attendees to explore the multicul-

tural influences on their work as evaluators. And in 1998, Stafford Hood presented a paper at the

Robert Stake Retirement Symposium that further extended the core dimensions of responsive eva-

luation to include culture and cultural context (Ryan, Chandler, & Samuels, 2007). More recently,

the American Evaluation Association formed a Task Force to review the Program Evaluation

Standards of the Joint Committee from a culturally competent standpoint. After significant input,

recommendations were approved for future revisions to the Program Evaluation Standards

(American Evaluation Association, Diversity Committee, 2004). Along with these recent develop-

ments in evaluation, cross-cultural variations continue to benefit from the important work on cultural

competence and cultural diversity in public and mental health and in social work (Lum, 2003; Sue &

Sue, 1999).

Despite the fact that widespread attention to cross-cultural evaluation is relatively recent, there

nonetheless exists a sufficient body of empirical research to warrant a systematic review. Several

contributors have noted that empirical research is essential to the advancement of the field (Cousins,

2004; Mark, 2008; Smith, 1993), a point that is especially relevant in a relatively new, yet rapidly

growing stream of inquiry such as evaluation in cross-cultural contexts. Such a review would thus

help shape discourse in the area by clarifying relevant concepts and interrelations among them,

identifying issues and research questions of concern, and revealing methodological gaps requiring

further elucidation.

The purpose of this article is to provide a descriptive review of the empirical literature on cul-

ture in evaluation and to contribute to the development of a theoretical framework to facilitate

future research and understanding concerning the complexity and multidimensionality of evalua-

tion within cross-cultural settings. Our aim in providing this framework is to clarify and describe

the multiple dimensions involved (e.g., relational, ecological, methodological, organizational,

and personal) in the cross-cultural program and evaluation context so as to further our thinking

about and guide research on cross-cultural approaches to evaluation. It is important to note at the

outset that although our review was guided initially by specific theoretical constructions (upon

which we will elaborate shortly), our theoretical framework was the result of our systematic

review of the empirical literature. The following questions also provided an initial focus to guide

our review:

1. How is culture conceptualized within the evaluative program setting?

2. How is culture thought to impact the evaluation, the program, and the context?

3. What rationale is given for the inclusion of culture in the evaluative strategy?

4. What methods/approaches are used to operationalize culture in the community program

setting?

5. What challenges do evaluators face in conducting cross-cultural evaluation?

458 American Journal of Evaluation 30(4)


Theoretical and Conceptual Orientation

Before moving to a description of the theoretical underpinnings of this review, we first begin with a

clarification of some of the key constructs in cross-cultural evaluation: program evaluation, culture,

cross-cultural evaluation, and context.

Program Evaluation

The number of different definitions of evaluation and the lack of overall consensus illustrates the

diversity of approaches to evaluation practice and theory. Some definitions of evaluation focus

on function (e.g., making judgments), others look at purpose (e.g., providing information), others

reflect method use (e.g., participatory evaluation), and still others include theoretical orientation

(e.g., Fourth Generation Evaluation) (Mark, Greene, & Shaw, 2006). For the purposes of this article,

we define evaluation as systematic inquiry leading to judgments about program merit, worth and

significance, and support for program decision making and knowledge production (Cousins, 2003;

Weaver & Cousins, 2004). This definition clearly situates evaluation as a systematic process guided

by social science inquiry methods. It also establishes the essential judgmental nature of evaluation

which, as a domain of inquiry, sets it apart from other forms of research. Evaluation is thus used to

make a judgment, (comparing findings against established goals or some other standard) about

program worth, to supply information for organizational or program decision making, and ultimately

to create new knowledge that may or may not be useful beyond the local program context. Although

necessarily broad, it is noteworthy that there is nothing explicit in this definition about the program

or program context, including the program community and intended program beneficiaries. Thus,

lurking behind this rather stark definition of evaluation are epistemological, ontological, and

normative questions that ultimately guide methodological choices and method selection, as well

as frame the role of the evaluator within the evaluative setting.

Culture

Culture is a contested concept and the subject of much theorizing and writing in numerous academic

fields, including sociology, anthropology, education, management studies, and communications.

Despite the volume of writing on the subject, there remains no agreed upon, universally accepted

definition of culture, though certain ideas have endured for centuries. To provide a comprehensive

description of culture and to impart a sense of its historical evolution, we will provide four charac-

terizations of culture, ranging from the aesthetic, ethnographic, symbolic, and ecologic, all of which

should be borne in mind throughout this article.

The aesthetic definition, introduced around the 18th century, associates culture with the arts and

is characterized by instances of high culture, sophistication, and refinement. The notion of high

culture is distinctly class-based, with the implication that only the wealthy could be bearers of such

elite qualities of heightened sensibility (Bocock, 1992; Edles, 2002). The ethnographic or anthropo-

logic definition includes shared meanings, knowledge, beliefs, morals, and customs (Bocock, 1992;

Edles, 2002), all of which is transmitted from one generation to the next (Guzman, 2003; Hughes,

Seidman, & Williams, 1993). These two characterizations, the aesthetic and ethnographic, both

describe what culture is, understood as qualities in individuals, or collectively in terms of the

contents of a culture (Bocock, 1992). The symbolic definition is from social anthropology and

characterizes culture as a system of shared meanings (Geertz, 1973) that are historically linked

to specific social groups at specific moments, intertwined in complex ways with other societal

dimensions (Edles, 2002, p. 6). Culture is portrayed as a social practice linked to specific groups,

fundamentally grounded in language and in the production of meaning (Bocock, 1992). This defi-

nition focuses on what culture does, rather than what it is, and is associated with the Birmingham

Chouinard and Cousins 459


School of Cultural Studies, whose members sought to shift the conception of culture from a literary-

moral definition to one based in sociology and the study of meaning (During, 1993; Seidman, 2004).

The ecologic description embeds culture within a larger, fundamentally interconnected social sys-

tem that is composed of a hierarchy of social forces (Guzman, 2003, p. 174). The ecologic def-

inition thus depicts the location of culture, as well as the context in which it is to be understood. It

provides, in essence, a conceptual bridge between culture and context, two terms that have signifi-

cance in the cross-cultural literature.

Although these characterizations are illustrative of the varied meanings ascribed to culture, for

our present purposes there are a number of characteristics worth noting and explicating from the

above: all individuals develop within a culture (Hughes et al., 1993); cultures are passed down from

one generation to the next (Guzman, 2003); cultures are learned (Rosaldo, 1989); cultures are

socially constructed through historical and political processes (Rosaldo, 1989); cultures are not static

but dynamic (Willging, Helitzer, & Thompson, 2006); cultures are related to language and meaning

and the production of knowledge (Gordon, Miller, & Rollock, 1990); and culture is implicated in the

politics of power and privilege (Seidman, 2004). These descriptions help move culture beyond a

mere demographic descriptor of communities, to a socially, politically, and historically vibrant and

embedded construct that is fundamentally constitutive of the values and norms that govern our

society. At the same time, this characterization of culture underscores the fact that epistemological

questions are bound up and implicated in social, political, and cultural assumptions that have

symbolic and very real material expressions. Our goal is not to attempt to reconcile these four per-

spectives of culture but merely to highlight the multifaceted, evolving and dynamic nature of culture,

and the varied lenses in which culture is understood across disciplines, as we carefully read through

the studies we have selected for our review.

Cross-Cultural Evaluation

Evaluations that endeavor to be responsive to culture and cultural context are referred to by

practitioners as culturally competent, culturally responsive, culturally consistent, transfor-

mative, culturally sensitive, culturally anchored, values-based, multicultural, or

cross-cultural. For the purposes of this article, the term cross-cultural evaluation will be used

as it highlights the social relations among stakeholders in evaluation (Abma, 2002) and acknowl-

edges that program evaluators do not always (or often) share cultural similarities (i.e., ethnicity,

gender, social class) with program participants (Yarbrough, Shulha, & Caruthers, 2004), though they

do work collaboratively toward common ends. More importantly, the term cross-cultural conveys

the sense of interaction between two or more cultures (Merryfield, 1985), highlighting the fact that

the evaluator him- or herself also has a culture that is itself worth exploring (SenGupta et al., 2004),

perhaps in relation to some other (Hall, 1992), and thus requiring that we critically examine our

own cultural values, assumptions, and biases (Nelson-Barber, LaFrance, Trumbull, & Aburto, 2005;

SenGupta et al., 2004), to more fully appreciate and apprehend the dynamic cultural context in

which evaluation takes place.

Context

The conceptualization of context, the parameters and dimensions considered relevant within an eva-

luation setting, varies across types of evaluation and fundamentally differentiate evaluation

approaches (Mathison, 2005). This point is particularly salient in the cross-cultural setting, as many

of the methodologies and approaches adopted for use in this type of evaluation are specifically

designed to attend to contextual factors (Johnson, 2005). The variation in theoretical and methodo-

logical approaches, as well as the diversity of programs and cultural settings, illustrates the complex-

ity of context within evaluation. In the Evaluation Encyclopedia (Mathison, 2005), context is



described as a multilayered and intertwined construct composed of demographic characteristics,

material and economic qualities, institutional or organizational dimensions of funders or evaluators,

interpersonal and interactive components, and political dimensions, including power, influence, and

privilege. This characterization of context provides a sense of the multidimensional and multifaceted

nature of a programs context, which in cross-cultural evaluation becomes all the more significant

(SenGupta et al., 2004), as the context itself becomes the site of confluence where program, culture,

and community connect. The dynamic interaction between diverse stakeholders in marginalized

cultural communities thus becomes important as the contextual dimension of the larger society and

the more local community, along with the challenges of diversity in terms of linguistic and cultural

characteristics, has an impact on social relations and program outcomes (Clayson, Castaneda, San-

chez, & Brindis, 2002).

Theoretical Orientation

It is our belief that the complexity of the cross-cultural evaluation context requires us to adopt a

multidisciplinary perspective to enhance and broaden our understanding of what we see, how we

see, and how we evaluate. As Symonette (2004) argues:

Attending to diversity and multicultural issues in evaluation invites and challenges the evaluation pro-

fession to expand its line of sight and the capacities of its practitioners in order to more authentically

perceive and receive the voices, vantage points, and experiences of the full spectrum of stakeholders.

(p. 98)

One of the principal theoretical constructs guiding this literature review is based on the conception

of evaluation as a relational endeavor (Abma &Widdershoven, 2008; Greene, 2005; Levin-Rozalis,

2003; Ryan & Destefano, 2001) that is fundamentally grounded in social relations (Symonette,

2004). Within this relationship, epistemologically, the evaluator and the diverse stakeholders are

interconnected, influencing each, as together they co-construct evaluation findings (Rebien,

1996). Evaluators are thus not considered passive purveyors of methodology but active co-construct

working with diverse stakeholders amidst a rich cultural program and community context (Greene,

2005). At the same time, stakeholders are not considered passive recipients of intervention, but

active participants who possess information and strategize (Long, 1992, p. 21). From a relational

perspective, both evaluators and stakeholders are considered active social agents who together influ-

ence practice and the construction of the social and ethnographic text (Long, 1992). Long (1992)

uses the concept of interface to convey a sense of a face-to-face encounter between people who

possess different interests, resources, and power and to depict the emergent forms of struggles and

interactions that take place between social actors as knowledge is created and co-create anew. The

production and creation of knowledge is thus conceived as dynamic, unfolding, and ongoing, giving

shape to the interface between evaluators and stakeholders, while at the same time being shaped by

these same face-to-face encounters (Villarreal, 1992). The significant point is that we co-create

meaning, and in so doing we transform the very meanings that we seek to understand. Relationships

thus play a dual role within the evaluative encounter, as they help shape the knowledge created and

they impart important norms and values that guide the evaluation (Abma & Widdershoven, 2008).

The other key construct guiding our review is based on the notion that evaluations are contex-

tually embedded within a program setting, as well as intertwined and immersed in specific cultural,

social, and institutional structures and practices (House & Howe, 2000), what we refer to as the eco-

logical perspective. This perspective situates the cultural context of the program and its evaluation

within a broader and more interactive historical, political, and social framework. Bronfenbrenners

(1979) renowned ecological model, composed of concentric circles depicting differing layers of con-

text, provides a sense of the interconnectivity and depth of the cultural milieu. Taking the ecological



theory a little further, Kellys (2006) ecological approach to community psychology, based on social

constructionist principles, emphasizes both the relational and the ecological components of

community development, with a special focus on the research relationship as a key consideration

in community intervention and development (Espino & Trickett, 2008).

These two theoretical constructs, the relational and the ecological, thus become significantly

intertwined and enmeshed in the cross-cultural evaluation setting, as evaluators and stakeholders

engage in complex interactions within and across diverse cultural contexts. As such, they provide

a sense of the relational nature of culture and of human interaction, where cultural processes are

involved in complex and multidimensional relations among different aspects of a communitys func-

tioning (Rogoff, 2003). The relational and ecological perspectives thus provide an overarching

framework in which to understand culture, context, and social relations involved in the evaluative

encounter, all particularly salient concepts within the cross-cultural setting. For us, they provide a

starting point on which to build a more elaborated conceptual framework through a review and

integration of extant empirical studies.

Method

Sample Selection

The purpose of this comprehensive literature review is to map the territory of cross-cultural program

evaluation, to learn from experiences in the field and from the diverse communities of practice, and

through our analysis, to provide a more elaborated theoretical framework to aid in further under-

standing the multiple dimensions involved in this context. While there remain many gaps in our

knowledge about how to best integrate culture and cultural context in evaluation and program set-

tings (Thompson-Robinson, Hopson, & SenGupta, 2004), the many theoretical and empirical studies

we located are encouraging, as they all further our attempt to make sense of the increasing complex-

ity and diversity of program contexts.

We limited our search criteria to empirical studies of community-based program evaluations and

evaluations that considered culture as a key variable to be included in methodological processes in

multi-ethnic or multi-cultural contexts (including within-group contexts as well). We considered cul-

ture a key construct in studies that included a culturally specific rationale or evaluation focus, or in

those that highlighted culturally-based findings or lessons learned. For our purposes, empirical

research was understood to include not only traditional social sciences methods (e.g., case studies,

mixed-method inquiry), but reflective narratives based on participant experiences with one or more

program contexts. Search terms or key words for this study included cross/cultural evaluation, cul-

turally responsive evaluation, cultural context, culturally competent evaluation, and anthropo-

logical evaluation. Because we intended the literature search to be broad and far-reaching, we

searched a number of key databases, including Educational Resources Information Clearinghouse

(ERIC), PsychINFO, Social Work Abstracts, Sociological Abstracts, and Medline. While we made

every effort to extend our sample beyond the North American context, the vast majority of our articles

(dating from 1991 to 2008) were nonetheless located in the following peer-reviewed journals: Amer-

ican Journal of Community Psychology, American Journal of Evaluation, American Journal of Pre-

ventive Medicine, Canadian Journal of Program Evaluation, Educational Evaluation and Policy

Analysis, Evaluation, Evaluation and the Health Professions, Evaluation and Program Planning,

Journal of Multidisciplinary Evaluation, the Journal of Primary Prevention, New Directions for Eva-

luation, and Studies in Educational Evaluation. As a measure of quality assurance, the majority of the

articles came from peer-reviewed journals, although we also included a few book chapters and foun-

dation reports that involved community-based program evaluations with a specific focus on culture

and cultural context that we believed would expand our analysis further. We also followed up



bibliographies of recently published work to locate otherwise undetected articles and studies. While

we do not make the claim that our sample is exhaustive, we are satisfied that it is sufficiently extensive

so as to capture the state of the art of empirical research in the area.

Sample Characteristics

Through our literature search, we located 52 empirical studies, taking the form of articles, book

chapters, and foundation reports, written between 1991 and 2008. These studies are summarized

in Table 1.

A majority of the studies were reflective narratives of single, multiple, or comparative case stud-

ies highlighting evaluator experiences, challenges, and lessons learned. A number of these case stud-

ies involved an analysis of a specific intervention or program across multiple communities, whereas

others focused more specifically on the qualitative components of a single case in specific cultural

contexts. Only a few of the studies involved single or collective reflections across a range of pro-

grams and contexts. Thirty-five of the articles were published in the last 5 years, between 2003 and

2008, eight were published between 2000 and 2002, and the remaining nine articles were published

in the 1990s. Given the dearth of cross-cultural evaluation articles published in countries outside the

North American context, 41 of the articles we located were based on evaluations in the United

States; 4 in Canada; 3 in New Zealand; 1 in each of Brazil, Australia, India, and Papua New Guinea.

The program practice contexts included in our sample of articles fall roughly into four primary cate-

gories, health (n 16), education (n 24), social services (n 5), community (n 6), and onenational symposium.

All the programs described in the articles were designed to ameliorate inequalities or to provide

specific targeted assistance to improve educational, health, or social issues in the following program

areas: violence prevention, HIV prevention, drug and substance abuse, improving possibilities for

at-risk students, enhancing developmental outcomes, and increasing under-representation. Target

populations for programs include Native American/First Nations and Inuit (n 25), African Amer-ican (n 12), Hispanic/Latino/Latina (n 12), Cambodian, Brazilian, Maori, East Indian, AsianAmerican, and Hmong, with some overlap in targeted populations. Most of the studies provided

implications for research or evaluation practice and/or lessons learned, reported on challenges

and strategies, provided guiding principles, and raised questions about evaluators experiences

working in cross-cultural program and community settings.

Review Strategy and Analysis

After identifying the 52 articles, we read each closely to ascertain program and population context,

focus, approach, theoretical orientation, rationale, and findings and challenges. Summarizing the

studies in a matrix format as we have done in Table 1 enables a descriptive cross-case analysis to

assist in identifying patterns, themes, and atypical findings. Our analysis was further guided by our

conception of evaluation as a relational endeavor, as well as an ecologically situated practice. The

theoretical literature on culture and cultural context in evaluation further complemented our analy-

sis. In the following section, we provide an overview and integration of the empirical literature we

selected for our review.

Review and Synthesis

Descriptive Analysis

Although all the studies we included in our review focus on the cultural context of the evaluation, we

were nonetheless able to identify over 38 different designations reflecting the specific focus of each



Table

1.

SummaryDescriptionofResearchStudiesonCross-C

ulturalEvaluations

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale

Relevantfindings/challenges

1.Aktan

(1999)

United

States:Substance

abuse

prevention

program

forAfrican

American

families

Evaluationusedto

make

theprogram

more

reflectiveofAfrican

American

norm

s

Culturalconsistency

approach;mixed

method;process

and

outcome(pre

andpost)

donebyindependent

consultant

Nonespecified

Culturalconsistency

Culturalmodificationshad

positive

effect

on

program

,withparticipants

more

engaged

andmore

likelyto

complete

program

.Culturalconsistency

approachthuseffective

inenhancingprogram

outcomes

2.Alkonet

al.

(2001)

United

States:Violence

preventioneducation

program

forchildcare

staffandHispanic

parents

Todescribechallenges

encountered

Randomized

experimentalstudy;

mixed

method:surveys,

interviews,observation

Nonespecified

Needto

use

culturally

sensitive

methodsto

evaluateprogram

sthat

includeethnically

diverse

populations

Challenge

findingdatacollectioninstruments

that

werevalidated

withethnicallydiverse

populationas

nolinguistic,conceptual,or

measurementequivalence

was

established

makinginterpretationdifficult;merely

translatinginstrumentunhelpful;need

identifiedto

elicitmore

inform

ationabout

culturalvalues

inresearch

tools;difference

incommunicationstyles

betweeninterviewers

andintervieweesadded

todifficulty

3.Anderson-

Draper

(2006)

Canada:Afamily

violence

prevention

program

inan

immigrant

community

Guidingquestions:what

activities

reflect

cultural

competence?W

hat

lessonscanbedrawn

that

willcontribute

toknowledge

base?

Socialcognitivetheo

ry(precede-proceed

planningandevaluation

model),datacollected

through

monthlyfocus

groups(lookingat

both

process

andoutcomes)

Participatory

Theroleofculture

asevaluationcanshape

how

weview

and

understandan

issue

Identifiedtheneedto

spendtimebuilding

trustingrelationshipsandprovidingtraining

andsupport

sothat

participants

canbe

meaningfully

involved;process

of

collaborationas

importantas

outcomes;

learningoccurs

both

ways;evaluatorsneed

todevelopfacilitationskills,cultural

competencies,andinterpersonalskills;

participatory

approachenablesevaluators

withdifferentbackgroundsthan

participants

toconduct

cross

culturalevaluations

4.Baizerm

anandCompton

(1992)

United

States:State

policiesforat-riskhigh

schoolstudents

Toreport

onapolicy

evaluationusedto

create

ongoingdialogue

withcommunity

stakeholders

Methodologically

diverse

soas

tolearn

from

participants

andbe

more

sensitive

todeep

culturalandsocialclass

difference

Evaluationas

ethicaland

political

Moralo

bligationto

hear

thevoices

ofthe

targeted

population;

sensitive

toculturaland

socialdifferences

Evaluationas

highlypolitical(purposeful,

sponsored,andjudgm

ental),allofwhich

impliespower

differences;also

amoral

enterprise;useddialogicalperspective

totransform

technicalactivities

into

possibilities

flowingfrom

relationshipsand

truedialogue


5.Barnes

(2000)

New

Zealand:Alcohol-

relatedroad

traffic

preventionprogram

for

Maori

Toexam

inethe

partnership

between

twoindigenous

communitiesand

researchers

Collaborative;

naturalistic;utilization-

focused

Participatory

based

on

communityaction

Needto

developbased

onMaorisocial

structures,delivery

system

sandcultural

context;ownership

and

empowerment

Although

time-consuming,thebuildingof

alliancesseen

asimportantandlasting;

possibleto

implementprogram

where

research

knowledge

isnotoverriding

concern

butanegotiated

component

6.Berendsand

Roberts

(2003)

Australia:Twenty

indigenousalcoholand

drugprogram

s

ToapplytheAES

guidelines

andtheAEA

standardsto

lookat

anindigenousevaluation

Extensive

consultation;

mixed

method;used

steeringcommittee

Nonespecified

Historicalandcurrent

statusofindigenous

populationrequires

carefulconsiderationin

designingevaluation

Establishingtrust

ledto

increasedKoori

involvem

ent;established

inform

alarrangementwithnotedKoorileaderswho

actedas

guardiansoftheprocess

7.Bevan-

Brown(2001)

New

Zealand:Two

projects:Special

educationresources

and

bestpractices

forMaori

childrenwithspecial

needsandnew

special

educationpolicy

Challengesencountered

andstrategies

used

Culturally

sensitive

approach;mixed

method

Nonespecified

Validityandreliability

of

data;program

effectiveness

Foraccuracy

ofresearch

datamustconsider

6Rs:therightpersonmust

asktheright

questionsoftherightpeo

pleintherightway

attherightplace

andtime;should

use

someo

nefrom

sameethnicgrouporelse

someo

newithculturalcompetence;m

ustbe

awareofdifferentculturalnorm

s8.Buttyet

al.

(2004)

United

States:C

RESPAR

program

:Urban

school-

to-careerintervention

program

forat-risk

middleschoolstudents

mostlyofAfrican

American

background

Explore

thesuccesses

andchallengesofusing

thisapproachwithan

emphasison

implementationrather

than

onfindings

Culturally

responsive

approach;mixed

method

Evaluationas

political

andvalue-laden

Toensure

evaluation

validity

(methodological,

cultural,interpersonal,

andconsequential),

ultimatelyleadingto

increasedadvocacy,

socialbetterm

ent,and

justice

Threemainprocess

challenges:engaging

stakeholders,culture

andculturalrelevance,

andtriangulation.Challengesalso

included

thelabor-intensive

nature,lack

ofpeo

pleto

dothework,andtherequirem

entofhaving

peo

pleperform

multipletasksandroles.The

respectforculturalcontextnonetheless

allowed

forgreatercollaboration,higher

engagement,andusefulandvalid

evaluation

results

9.Caldwell

etal.(2005)

United

States:American

IndianResearchand

Program

Evaluation

Methodology

National

Symposium

Collectiveexperience

toprovidelessons

learned

andguiding

principles

Community-based;

collaborative,PAR;

culturally

anchored

methodology;

retraditionalization

(return

culturalnorm

s)

Empowerment;

participatory;cross-

cultural

Culturalrespectand

understandingofspecial

circumstances;validity;

empowerment

Understandingofpostcolonialstress;

relationalresearch;authenticpartnerships;

communityinvolvem

entin

data

interpretation;research

codes

ofethics;

tribal,cultural,andlinguisticdiversity;

strengthsandculturalprotectivefactors;

locally

meaningfulconstructs;trainingand

employm

entofcommunitymem

bersas

evaluationproject

staff;capacitybuilding

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


10.Cervantes

andPena

(1998)

United

States:High-risk

Hispanic/Latinoyouth

andfamilies

(alcoholand

drugtreatm

entand

prevention)

Toprovideguidelines

forthedevelopment

andimplementationof

culturally

competent

evaluationpractices

Culturally

competent

evaluationstrategy;

process

andoutcome

evaluation;quantitative

Culturalcompetence

Theneedto

understand

theroleculture

plays

inthedevelopmentof

drugandalcoholuse,

misuse,andabuse.

Failure

todoso

may

resultin

inappropriate

conclusionsabout

program

effectiveness

Needto

hirequalified,bicultural/bilingual

Hispanic/Latinoevaluators

orculturally

competentevaluators;trainstaffoncultural

issues;communitysensitivity

(socialand

dem

ographiccharacteristics);recognize

heterogeneity

within

population;assess

nontraditionalcharacteristics;consider

language;create

communityadvisory

group

forcommunicationpurposes;develop

scientific

pre-andpost-testrigor;involve

evaluators

ininform

ationdissemination

11.C

hristieand

Barela(2005)

United

States:Student-

centeredoutreach

program

toincrease

UCLA

admissionof

underrepresented

groups

Toprovidean

exam

ple

oftheDelphitechnique

asameansto

amore

inclusive

approachto

evaluation

Delphitechniqueused

todevelopconsensus

Socialjusticeand

empowerment

More

accurate

understandingofthe

socialbenefitsofthe

program

;stakeholders

bestplacedto

assess

program

;develop

strength-based

program

model

Delphitechniqueprovided

ameansto

successfully

involveminority

stakeholders

anditeliminated

power

imbalances;worked

indevelopingconsensusandin

considering

allstakeholder

view

sandgivingweightto

all;

Delphitechniquethussuccessfulin

promotingsocialjusticeevaluationsand

increasingparticipation

12.Clayson

etal.(2002)

United

States:Three

low-income,Latino

communityinitiatives

aimed

atbuilding

community,

strengthening

leadership,and

enhancingcivic

engagementat

the

grassroots

level

Todiscuss

interactions

betweenmajor

stakeholder

groups

(funders,community-

based

organizationstaff,

communitymem

bers,

andevaluators)

Participatory;context-

sensitive

lens

Criticaltheo

ry;

constructivist;theo

ryof

change;evaluationas

political

Provides

amore

complete

analysis;

acknowledgestherole

ofevaluatoroccurs

within

aparticular

contextandwithin

alarger

politicaland

economicenvironment

Evaluationseen

ashighlypoliticalwithpower

distributionam

ongstakeholders

instrumentalin

shapingfindings;dynam

icinteractionam

ongstakeholdersshaped

by

contextualdimensionsandthechallengesof

diversity

(culturally

andlinguistically);

challenges:funderslocked

inconcepts

they

regarded

asrelevantacross

allcontexts;

consensusam

ongstakeholder

difficult,given

unequalpower

relationships

13.Conner

(2004)

United

States:HIV

preventionprogram

intw

oLatinocommunities

Todescribefivefactors

thatfostered

aculturally

sensitive

evaluation

Multiculturalvalidity;

culturally

sensitive

approach

Culturalsensitivity

Tomore

meaningfully

assess

andengage

program

s;multicultural

validity;program

understanding

Five

facilitatingfactors:involvingparticipants

forincreasedunderstanding;speakingliteral

andfigurative

languageofparticipants;

workingcollaborativelyduring

implementation;sharingbenefitswith

participants


14.C

ooper

and

Christie(2005)

United

States:

University-sponsored

parenteducation

program

forlow-income

Latinomothers

Use

ofsocialjustice

approachto

enhance

understandingand

promote

educational

equity

Began

withresponsive

evaluationapproachand

latersw

itched

toasocial

justiceapproachto

emphasize

underrepresented

voices;qualitativecase

study

Socialjustice;

empowerment

Inclusive

ofleast

powerful;socialjustice;

preventstakeholder

bias

Recognized

importance

ofaddressing

culture,class,andgender

issues;needfor

evaluatorflexibility;could

notbeobjective

norvalue-free;theo

ryas

usefulguideto

practicebutmethodologicalchoices

madein

asocialandpoliticalcontext

15.Copeland-

Carson(2005)

United

States:

Community

revitalizationproject

for

African

American

population

Toexplore

how

anthropology

can

contribute

toevaluation

designofcommunity

initiative

Theo

ry-based

anthropological

evaluation;collaborative

Collaborative;based

on

anthropologicalmodels

ofsocialchange;

Giddenssandindividual

agency

focus;Bourdieu

andFo

ucaultonpower

andknowledge

Tobetteraddress

the

complexityof

communityinitiatives

Bringingdiverse

stakeholderstogether,along

withtherealitiesoftheeconomicand

politicalenvironmentresulted

inadynam

icorganizationalculture

fortheproject;ledto

thepositioningofthecommunitysethnicity

asaresourcerather

than

abarrier;

qualitativeandquantitative

dataseen

asculturalconstructions;reliedupon

communityconsultantsforknowledge

and

feedback;lots

oftimeandcostsassociated

withusingthistypeofevaluation

16.Coppens

etal.(2006)

United

States:

Cam

bodianyouth

dance

program

Toexplore

importance

ofclearcommunication,

culturalaw

areness,

tailoringevaluation,and

meaningfulparticipation

Collaborative;culturally

sensitive;usedmultiple

methods

Collaborative

Toconduct

culturally

sensitive,community-

based

research

and

evaluation

Dynam

icsofdiversity

influencedseveral

dimensionsofevaluation;effectsofcultural

influencesondynam

icofgroup;challenges

involved

ingeneratingfindings

using

standardized

proceduresandmeasures

required

byfundingagency;noted

differencesin

communicationand

perceptions;contextcriticalto

communication

17.Fetterman

(2005)

United

States:Aproject

designed

tobridge

the

digitaldividebetween18

American

Indiantribes

and2African

American

communities

Todescribetheinsider

perspective

through

the

use

ofstories

Combinationof

empowerment

evaluationand

ethnographicevaluation

Collaborative;

empowerment

Tofoster

self-

determinationand

improvement

Communitiesusingthesetypes

ofevaluation

toplan,implement,assess,improve,and

refinetheirefforts;manyexam

plesof

reflexivitydem

onstratedduringtheproject

18.Fisher

and

Ball(2002)

United

States:Indian

family

wellnessproject

TodescribetheTribal

Participatory

Process

(TPR)

TPRmodelbased

on

tribalculturalandsocial

values;evaluationwas

culturally

specificand

developed

byaworking

group;usedamultiple-

baselinedesign;data

collected

atmultiple

intervals

Participatory;cross-

cultural;em

powerment

Tomeettheneedsof

thecommunityand

build

onspecific

strengths;include

historicalcontextand

use

evaluationas

aninstrumentof

empowermentand

socialchange

Indevelopingtribal-specificmodelsofwell-

being

consider

historicalcontextin

evaluation;multiplebaselinedesign;language

changedto

reflect

localnorm

s;domains

measuredincludeparticipationin

cultural

events,connectednesswithextended

family,

tribe,andcommunity;use

ofstorytelling;

assessmentem

phasizes

prosocialdomains

such

asrespectfulbehaviorandsocial

competence

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


19.Garaw

ay(1996)

India:Aliteracy

acquisitionproject

Todescribethe

evaluationapproach

usingtw

olevelsof

analysis

Cross-cultural

evaluationusinga

multiplecase,replication

design;mixed

method

Cross-cultural;

evaluationas

political;

limited

participation

Toprovideforthe

broader

perspective

within

thecomplex

linguisticandcultural

milieu

ofIndiaandto

promote

fairness

Theapproachprovided

overarchingcross-

case

answ

erswhile

maintainingsensitivity

toeach

specificculturalsetting;cross-cultural

evaluationsin

developingcountriespresent

extrem

echallengesin

term

sof

uncontrolledforvariables;cross-cultural

evaluationsareparticularlycomplex

politically

20.H

arklau

and

Norw

ood

(2005)

United

States:A

summer

college

readiness

program

forAfrican

Americans,Asian

Americans,andAnglo

youth

Tolookat

the

researcher

roleand

reflexivity

Ethnographicevaluation;

participatory

Postmodern;

participatory

Toilluminatetheroleof

evaluators

(lookingat

power

dimensionand

subjectivities)

Evaluators

take

onmanydifferentroles

duringthecourseofan

evaluation;roles

intersectedwithother

personalidentities

andsubject

positionsandaffected

how

they

understoodtheprogram

andnegotiated

roles;rolesas

inherentlyrelational

21.Honget

al.

(2005)

United

States:HIV

preventionprogram

targetingAfrican

American

injectiondrug

users

Tolookat

aprocess

evaluationduringthe

pilotstageusing

ethnographicmethods

Ethnographicprocess

evaluation;data

collectionthrough

observation,interviews

Critically

inform

edevaluation;lim

ited

participation

Culturalrelevance

tounderstandbehaviorin

context

Ethnographicmethodsprovideadynam

ic,

flexible,anditerativeprocess

forevaluating

thedevelopmentoftheinterventionand

ensuringculturalrelevance;strengthsare

that

they

built

theevaluationinto

the

program

pilotphase;developed

asystem

atic

strategy

throughout;evaluationteam

and

implementationteam

worked

closely

together.Limitationsincludethesheer

volumeofdatagenerated

createdtime

issues;difficultto

collect

inform

ationin

naturalsetting;andcouldnotcollect

dataon

comparisongroupdueto

timeandresource

constraints

22.Jayet

al.

(2005)

United

States:Summer

PregraduateResearch

Experience

Program

involvingAfrican

Americans,Native

Americans,Mexican

Americans,andPuerto

Rican

Populations

Tolookat

the

significance

ofcultural

contextandcultural

influencesonthe

experiencesofprogram

participants

Culturally

responsive

evaluation;qualitative

Culturalresponsiveness

Toprotect

the

evaluationfrom

being

seriouslyflawed

or

skew

ed;validity

Culturalresponsivenessplayedakeyrolein

allphases;provides

additionalmeasuresfor

assessingprogram

worthbeyondthesuccess

ofimplementationandachievementof

program

goals


23.Johnson

(2005)

United

States:Program

designed

toincrease

participationof

minorities

inscience,

technology,engineering,

andmathem

atics(STEM)

education

Toaddress

how

todevelopevaluation

strategies

that

are

culturally

responsive

Culturally

relevant

evaluation;participatory;

purposefulsampleof

eightexperienced

evaluators

Participatory;cultural

sensitivity

Needto

include

contextualfactors

inevaluation

Challengesin

conductingculturally

and

contextuallyrelevantevaluationsinclude

socialpressuresandpsychometric/design

concerns;politicalunderpinnings

ofpolicy,

research,andpracticearerealyetoften

left

unexam

ined

24.Kinget

al.

(2004)

United

States:Fo

ur

multiculturaleducation

initiativesaddressing

curriculum

and

individualneeds

Torecast

critical

incidents

asdilemmas

andto

highlight

challenges

Participatory

evaluation;

culturally

competent

evaluation;use

of

multiplemethods

Participatory;

multiculturaleducation

andcultural

competence

fram

efor

study

Participatory

approach

recognizes

complexity

andvaried

worldview

s

Dilemmas

suggestlim

itations(a)evaluation

fram

ingissue;(b)roleofevaluationin

supportingimplementation;(c)evaluation

use;points

toneedto

explicitlyidentify

stakeholder

values

andinterests;conflict

betweenproprietyissues

andfeasibility

and

utilityconcerns;tensionbetweensocial

actionandutilityandfeasibility

concerns

25.LaFrance

(2004)

United

States:

Experiencesconducting

evaluationsin

Indian

country

Toprovideindigenous

epistemology

asa

culturalfoundationand

todiscuss

methodological

practices

Culturally

responsive

evaluation;participatory;

form

ativeevaluation

Indigenous

epistemology;

participatory

Toestablishnew

evaluationprocesses

that

arebroad

enough

toaccommodateand

valuedifferentwaysof

knowing,build

ownership

andsense

of

community,and

contribute

tohigh

qualityprogram

s;validityandreliability

Importance

ofform

ativeevaluationandthe

needto

beevaluated

within

owncontext;

valueofbuildingconceptualmodelswith

stakeholders;importance

ofparticipatory

practices

andevaluationcapacity;challenges

indoingcomparativeresearch;evaluationas

knowledge

creation

26.Laperriere

(2006)

Brazil:HIV

prevention

program

forsex

workers

Toillustrate

challenges

involved

inconducting

evaluationsin

cross-

culturalandhighly

unpredictable

environments

Ethnographicand

community-based

approachwith

participationoflocal

actors;goal-free

evaluation;qualitative

research

fram

ework

Participatory

Creatingan

evaluation

relationship

asameans

oflearning

Contextimposedlim

itsto

predictability

makingevaluationmore

difficultwithin

aW

estern-based

scientific

fram

ework;

challengesincluded

translatingevaluators

intentionsin

away

that

madesense

toparticipants;participationnecessary;cultural

variationsofunpredictability;notedrelations

ofinstitutionalized

influencesandfield

inform

ation

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


27.LaPointand

Jackson(2004)

United

States:Family,

school,andcommunity

partnership

program

for

Black

students

inalow-

incomeurban

high

school

Todiscuss

challenges

workingwith

communitymem

bers

that

aremarginalized

by

institutionalbarriers

Practicalparticipatory

evaluation;

co-constructionand

evidence-based

practice;PAR;followed

thetalentdevelopment

evaluationstrategy;

participantobservation

ofacase

study

Participatory;

empowerment

Tovalidatesoundness

ortrustworthinessof

findings;

responsiveness;cultural

andcontextual

relevance

Significance

ofprogram

activities

from

perspective

ofdisenfranchised

participants;

triedto

build

onsimilarities

between

evaluators

andparticipants

interm

sofrace/

ethnicity,experienceswithculturalgroup,

personalfamily

background;evaluators

need

toobtain

experience

intrainingand

professionaldevelopmentprogram

sfor

workingwithmarginalized

groups;

contextuallyandculturally

responsive

evaluationfollowsasset-based

approach

28.Letiecqand

Bailey(2004)

United

States:American

Indianyouth-based

initiative

toimprove

the

qualityandquantity

of

comprehensive,

community-based

program

sforchildren,

youth,andfamilies

Toconduct

aculturally

sensitive

and

appropriatecross-

culturalevaluationand

explore

outsider

perspective

TPRmodels(Fisher

&Ball,2002)

Socialclass,culture,

ethnicity,andrace-

based

perspective;

cross-cultural

Needto

putinteraction

ofsocialclass,culture,

ethnicity,andrace

atthecore

tominimize

theeffectsofsocial

inequalityand

oppression;outsiders

must

consider

their

place

andperspective

Challenges:power

differentials;resistance

toevaluationandbuy-in;measurement

considerations

whodetermines

what

isvalid,reliable,andaccurate;w

aysofknowing

differ;confidentiality;logicalconstraints;

lessonslearned

included

theneedto

focus

onrelationships,evaluationapproaches

do

notalwaysfit

withW

estern

scientific

methodology

29.M

aciaket

al.

(1999)

United

States:

Partnership

toprevent

intimateviolence

against

Latinawomen

Toaddress

theneedfor

greaterunderstanding

oftheform

ativestages

oflocally

based

partnerships

Community-Based

Participatory

Research;

form

ativeapproach

Collaborative;

empowerment;

ecologicalperspective

Strengthen

theability

of

communitiesto

address

healthconcerns

Challengesin

thedevelopmentof

partnershipsincludemaintainingownership

inthecommunity;lackingtrust

andrespect;

strikingabalance

betweenresearch

and

action;lackingknowledge

aboutcultural

differences;lackingfundingfordevelopment

activities;lessonslearned

includemaintaining

ownership

andlocalcontrol;developing

strongandstableleadership

within

community;needlong-term

commitment;

communityinvolvem

entcanhelpwith

understandingofhistory

andcultural

context;ensure

culturalcompetence


30.McK

enzie

(1997)

Canada:Child

andfamily

services

program

sin

eightFirstNations

communities

Todevelopculturally

appropriatechild

and

family

servicestandards

inFirstNations

communitiesandto

defineandassess

the

evaluationprocess

Community-Based

Participatory

Research;

focusgroupsfollowed

by

feedbackand

consultations

Participatory;

empowerment;

evaluationas

catalyst

forchange;cross-

cultural

Use

participatory

approachthat

recognizes

importance

ofculture

andto

promote

mutual

learning;contribute

tocommunity

empowerment

Focusgroupsparticularlyeffectiveineliciting

meaningfuldialogueandparticipation;

culture

recognized

asessential;em

phasison

traditionalpractices;time-consuming

process;lim

ited

resources;connection

betweentheo

ryandpracticedifficultto

achieve

31.Mertens

andHopson

(2006)

United

States:Increased

participationof

underrepresented

groupsat

multiplelevels

inscience

and

engineeringfields

Toexam

ineimplications

ofusingtransform

ative

lensandcultural

competency

toincrease

understandingofhow

evaluationcontributes

toimprovingoutcomes

Culturally

responsive

evaluationandcultural

competency

approach

Transform

ative(rooted

indiversity,privilege

andpower)

Services

perceived

aslegitimate;acts

asa

prosocialchange

agent;

showsgenuinerespect

isactive

process

of

becomingaw

are

Transform

ativeparadigm

usefultheo

retical

construct

toexplore

philosophical

assumptionsandto

guidemethodological

choices

forapproachto

evaluationthat

are

inclusive,human

rightsbased,dem

ocratic,

constructivist,andresponsive;enables

lookingbeyondmandateofscientifically

based

research

32.Nagai

(2001)

PapuaNew

Guinea:

Curriculum

project

inan

elem

entary

schoolin

aMaiwalacommunity

Toexam

inethe

experiencesofan

expatriateattemptingto

sharetheownership

of

theresearch

andto

developcommunity

ownership

Participatory

Action

research

(PAR);

ethnography

Collaborative

Tohelpthecommunity

reclaim

itscultural

identity

Sherealized

that

thecommunitysinitial

dependence

uponher

meantitwas

too

radicalforthem

toseethem

selves

asequal

andshewouldhaveto

change

tobreakdown

theirview

ofher

assuperior;shemadesure

notto

introduce

Western

ideasbutletthem

discoverlocalprinciplesofassessmentand

evaluationthat

madesense

tothem

33.Nelson-

Barber

etal.

(2005)

United

States:

Community-based

program

sin

indigenous

communitiesand

educationalprogram

sin

multicultural/multiethnic

urban

settings

Toexplore

howcultural

competence

contributesto

the

reliability

andvalidityof

program

evaluation

Culturally

competent

evaluation

Cross-cultural;

participatory

Validity;ethical

Culture

andculturaldiversity

affect

all

contexts

andthusneedto

surface

culture-based

assumptions;evaluators

need

increasedaw

arenessofexternalandinternal

factors

affectingprogram

goalsand

understandingofbroad

politicalandhistorical

context;participatory

andem

powerment

evaluationlendsitselfmore

readily

toculturalresponsiveness;American

mainstream

practices

considered

baseline;

power

differencesmakeithardformem

bers

ofnondominantgroupto

participate;needto

consciouslyaddress

power

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


34.Novins

etal.(2004)

United

States:Mental

healthservices

program

forAmerican

Indian/

AlaskaNatives

Tocreate

culturally

appropriateoutcome

measurementplans

Participatory;grantees

selected

own

assessmentapproaches

(measurement,

inform

ants,timelines,

specificmeasures)

Participatory;cross-

cultural

Mainstream

approaches

inappropriate;needto

developculturally

and

program

matically

relevantapproaches

tomeasuringoutcomes

Needto

focusonstrengths;needto

select

ownoutcomes;themore

fundersspecify

the

use

ofspecificoutcomemeasures,theless

communitieswillpursueinnovative

approaches

tomeasurement;importance

of

communityleveloutcomes;relationship

betweenfundersandcommunityand

difficultybalancingdualevaluationneeds;

plandem

onstratesthepower

fundershavein

shapingentire

discussion(continueto

mandatetheirownoutcomemeasurement

plans)

35.Peter

etal.

(2003)

United

States:Preschool

immersionprogram

for

theCherokeeNationof

Oklahoma

Todevelopaculturally

responsive

evaluationof

apreschoollanguage

program

Culturally

responsive

evaluation;combination

offourthgeneration

evaluationand

empowerment

evaluation

Criticaltheo

ry;

naturalisticinquiry;

constructivist;

participatory

and

emancipatory;cross-

cultural

Tobemore

responsive

totheclaims,concerns,

andissues

of

stakeholders;more

appropriatebecause

more

respectfuland

allowsforfull

participation,parityand

control

Enableslegitimacyandhelpssurfacediverse

culturalvalues

andperspectives;develops

autonomyandownershipmakes

the

process

uniquelyCherokee

36.Prilleltensky

etal.(2000)

Canada:Sm

oking

preventionprogram

for

LatinAmerican

immigrants

Todiscuss

findings

and

implicationsofapproach

Approachbased

on

participatory

community

planning,sensitivity

toculturaldiversity,and

holisticphilosophyof

health;inform

edbyPAR

Values-based

approach;

participatory

Tohelpmarginalized

peo

pleexperience

personalandpolitical

empowerment;build

buy-in

todevelop

commitmentto

use

data;foster

community

development

Partnership

established

betweenuniversity

researchersandcommunityhelped

facilitate

theevaluation;value-based

partnership

enabledthevariouspartnersto

contribute

differentstrengthsto

theproject;noone

party

heldabsolute

power

todictate

values

andprinciples

37.Richmond

etal.(2008)

United

States:

Promotinghealthy

relationshipsproject

for

American

IndianYouth

Todescribehow

evaluationplanchanged

andfactors

influencing

thischange

TPRModel

Participatory;cross

cultural;em

powerment

Importantto

know

the

culturalcontextand

recognizeunique

strengthsofeach

partner

EvaluationofAmerican

Indianprogram

sneedto

beinwardfocused;hardfor

evaluators

andfundersto

letgo

of

preconceptionsaboutwhatconstitutesgood

research;experience

was

areciprocal

learningexperience

withallpartnersbringing

differentskillsandareasofexpertise;

evaluatorplays

dualrolesometimes

conflictingandam

biguous


38.Robertson

etal.(2004)

United

States:

Community-based

law

enforcem

entprogram

fortheOglalaLako

taNation

Todescribethe

evaluationprocess

asa

meansto

mirrorLako

taapproachto

evaluation

andresearch

PAR;em

powerment

evaluation;local

researchersdesigned,

andimplemented

evaluation

Cross-cultural;attempt

tomirrorLako

taapproachto

research

andevaluation

Tomaketheevaluation

asusefulas

possibleto

theLako

taandto

mirrortheirapproach

toresearch

and

evaluation(Lakota

methodologies)

Situatingevaluationin

communitybuilds

culturalresonance

andcreatespossibility

of

new

datageneration;process

enabled

communitymem

bersto

deepen

understandingandwork

collectivelyfor

change;usedvarietyofapproaches

toshare

findings;challengesincluded

not

overburdeninglim

ited

capacityof

community;notbuildingfundingdependency;

difficultymeasuringlinear-based

outcomes;

raised

possibility

ofsocialandinstitutional

change

39.Running

Wolfet

al.

(2002)

United

States:

Communitymental

healthforAmerican

Indian/AlaskaNative

childrenandtheir

families

ineighttribal

communities

Todescribechallenges

andsuccessesoftribal

communityin

research

andevaluationandthe

influence

ofhistorical,

cultural,andother

factors

onevaluation

findings

Fourprimary

components:system

levelassessment,

descriptionofchildren

served

byprogram

,assessmentofservice

experience

and

longitudinaloutcomes,

assessmentofservices

Notcross-cultural;

historicalandcultural

influences

More

in-linewithvalue

structure

and

worldview

sof

participants;make

services

more

culturally

appropriate

Needto

understandextended

family

system

;w

raparoundprocess;allcommunities

different;to

helpbuild

community

empowermentin

evaluation,used

community-based

advisory

committees

and

established

acollaborative

skill-building

relationship

withevaluationteam

;challenge

protectingconfidentialityin

small

communities

40.Ryanet

al.

(2007)

United

States:Culturally

responsive

school-based

initiativesinvolvingfour

schoolsconsidered

at

riskwithNative

American,Latinoor

African

American

populations

Identifyandexam

ine

challengeswith

culturally

responsive

school-based

evaluation

Instrumental;mixed

methodscase

study;

interviews,focusgroups,

andquantitative

documentandvideo

analysis

Culturally

responsive;

values-based;grounded

theo

ry

Honorculturalcontext

inwhichprogram

takes

place;values

diverse

needsandinterestsand

socialjustice;ensure

power

imbalancesdo

notimpactevaluation

Schoolsmovedfrom

amore

superficial

understandingofculture

toamore

nuanced

understanding,afact

that

resulted

intheir

lookingatdatadifferently;strongoncapacity

buildingforboth

evaluationandculture;

constraints

included

timeanddifficultiesin

operationalizingculture

41.Senese

(2005)

United

States:Dine

WellnessCentre

bilingual/bicultural

lifelonglearningprogram

atLittleSingerSchool

Toraisequestions

abouttherelationship

betweenevaluationand

research

andquestions

ofrace,culture,and

socialclass

Identifiedcommunity

stakeholdersto

understandhow

wellnessconceptfram

edconnectionswith

traditionalNavajo

spirituality;used

interviewsandfocus

groups

Cross-cultural

Tocreate

the

educationalexperience

tomakethecommunity

strongerandmore

culturally

competent

Culturalaw

arenessnecessary

butnot

sufficient;relationship

betweencultural

relevance

andsilence

inevaluation

concerningrace

ineducationandtheculture

ofsocialclassin

postindustrialcapitalism

(see,itas

contradiction);confusionaround

notionofculture

asappliedto

traditional

waysofknowingandliving,andeffectsaftera

history

ofstate-directeddispossession

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


42.Slaughter

(1991)

United

States:Haw

aiian

languageimmersion

program

Todem

onstrate

the

needto

includecultural

mem

bersonthe

evaluationteam

Qualitative;

ethnographic

Culturalsensitivity

Toensure

validity,

credibility,andfairness

Theinclusionofculturalinform

ants

can

strengthen

anevaluationthat

mediates

power

differencesbetweenvariouscultural

perspectivesandgroups;aresearcher

from

sameculturalgroupbrings

valuable

knowledge

oftheprogram

context,

establishes

credibility

andreceptivity;also

guardsagainst

ethnocentrism

ofexternal

evaluators

43.Sm

allet

al.

(2006)

United

States:

Communitypartnership

withHmongfamilies

withearlyadolescent

childrento

enhance

developmental

outcomes

Toexplore

the

dynam

icsandchallenges

that

non-H

mong

academ

icevaluators

experiencedin

cross-

culturalcontext

Collaborative;

evaluation

subcommittee;mixed

methods

Collaborative

Tobeculturally

sensitive,respectful,

collaborative,open-

minded,andflexible

Inwritingarticlerealized

how

their

privilegedstatusledto

power

differences;

thoughtthey

werebeingcollaborative

but

theirinsistence

onspecificscientific

proceduresallowed

forlittleinput;also

required

byprogram

funder

touse

aninstrumentthat

did

notfit

culturally

anddid

notmeetthestandardsofmulticultural

validity;notsuccessfulat

gaininglocal

ownership

because

evaluationdriven

externally

44.Stockdill,

Duhon-Sells,

Olsonand

Patton(1992)

United

States:

Multicultural,

community-based

educationprogram

calledSupporting

Diversity

inSchools

Tosharelessons

learned

aboutinvolving

communitiesofcolorin

evaluation

Developmental

approach

Culturally

and

contextuallysensitive

Foroneofthe

evaluators,therational

was

that

evaluationsof

multiculturalprogram

sshould

modeldiversity

them

selves

Lessonslearned

weredeveloped,shared,

discussed,andusedbyprogram

staffand

schoolpartnershipsforlearning;lessons

included

needformutualcommitmentfor

healthypartnerships;needto

shareavision

amongpartners;needstability

on

partnership

team


45.Thomas

(2004)

United

States:Talent

developmentevaluation

modelofschoolreform

forstudents

whoare

mostoften

placedatrisk

foracadem

icfailure:

low-income,minority

students

inurban

public

schools

Todiscuss

them

esand

conceptualfram

ework

oftalentdevelopment

evaluationmodel

Incorporatedmajor

tenetsofparticipatory,

responsive,deliberative,

culturally

competent,

multicultural,and

inclusive

Socialjusticeandcritical

perspectives

Responsivenessto

context

Centralandoverlappingthem

es:engaging

stakeholders

complexandlaborintensive

butcanminimizeproblemsrelatedto

unequaldistributionofpower

andstatus;

co-constructiondevelopspartnershipsso

canbemore

responsive

tocontextandalso

involves

aredistributionofpower

and

assumptionofequalityam

ongstakeholders;

culturalandcontextualrelevanceculturally

competentresearch

bestdonebyqualified

representativesoftheculture

beingserved

andifunavailablethen

needto

findpeo

ple

whoareunderstandingofculture

andopen

toself-reflection

46.T

homas

and

Bellefeuille

(2006)

Canada:Mentalhealth

program

foraboriginal

peo

pleswhowerein

residentialschools

Toreportonfindings

of

evaluationthat

used

aboriginalmethodology

Aboriginalmethodology;

grounded

theo

ry;cross-

cultural;qualitative

interviews;andfocus

groupsto

assess

aboriginalhealingcircle

andpsychotherapy

techniqueoffocusing

Cross-cultural;

aboriginal;thefour

criteriaofLincoln

and

Guba(1985)usedto

assess

qualityof

research

(ascross-

cultural)

Cross-culturalapproach

sothat

aboriginal

peo

plescandefinetheir

ownprogram

sand

interventionsbased

on

theirownexperiences;

culturalsensitivity

Communitiesto

decideresearch

priorities;

mentalhealthconsidered

within

wider

contextofhealthandwell-being;healingand

wellnessmust

draw

ontheculture

for

inspiration;w

ork

mustreflect

acommitment

tosocialjustice,acriticalpedagogy

of

decolonializationandastrength-based

philosophyofpersonal,community,and

culturalcapacitybuilding;recognizedisparity

betweenAboriginalandnon-Aboriginal

conceptionsofresearch

methodologies

47.Thurm

anet

al.(2004)

United

States:Mental

healthservicemodelfor

American

Indian/Alaska

nativechildrenandtheir

families

Toreportonevaluation

process

andlessons

learned

Participatory

evaluation;

createdtw

otechnical

assistance

team

sto

enablemutuallearning

Culturalrelevance

Lots

ofdiversity

ineach

community,so

needto

developsolutions

specificto

localneeds

that

areculturally

relevant

Tribes

wereableto

putevaluationmethods

into

more

culturally

relevantcontexts

(e.g.,

healingcirclesrather

than

focusgroups);the

more

culturally

relevant,themore

engagementandbuy-in

they

got;lots

of

inform

ationsharingacross

sites;planning

integrated

withevaluation;recognitionof

difference

valued

more

than

compromise;

evaluationmust

beunderstoodwithin

broader

fram

ework

48.Uhlet

al.

(2004)

United

States:HIV

preventionintervention

program

forAfrican

American

women

Todescribes

challenges

andbenefitsofinvolving

thecommunityin

the

evaluation

Collaborative

process

drawingfrom

empowerment

evaluationand

utilization-focused

evaluation;randomized

controltrial

Tenetsfromcommunity

psychology

and

ecologicalperspective;

collaborative

Increase

relevance

and

appropriatenessof

evaluation;respect

culture,history,and

localcontext;increase

culturalsensitivity,

communityacceptance,

andrelevance

ofproject

Challengesto

involvingthecommunity

included

timeandresources;benefits

included

thefact

that

itimprovedthequality

ofthestudyandthevalueoftheintervention

andtheevaluation;buy-in;helped

build

communitycapacity;improvedparticipant

recruitmentandretention;more

resources

would

haveledto

theimplementationof

more

communitysuggestions

(con

tinue

d)


Table

1.

(continued)

Study

Context

(program

and

population)

Focus

Approach

Theo

reticalorientation

Rationale


49.Voyleand

Simmons

(1999)

New

Zealand:

Community

development

partnership

withMaori

populationforhealth

promotion

Todiscuss

the

form

ativeandprocess

evaluation

Collaborative;form

ative

andprocess

evaluation;

committeeledprocess

Collaborative;

empowerment;self-

determination

Tobuild

apartnership;

empowerment;enable

therightto

self-

determination

Key

issues

weretrust,prioritizationof

health,andfindingappropriateresearch

paradigms;devolutionofpower

isessential

aspectofbuildingasuccessfulpartnership;

needto

findappropriateroles;needto

work

withculturaladvisors

familiar

withthelocal

community

50.W

hiteand

Hermes

(2005)

United

States:Evaluation

ofHopiteachersfor

Hopischoolsproject

Touse

jazz

asa

metaphorfor

understandingspaces

betweentraditional

Western

waysandHopi

waysofknowing

Collaborative

PAR;

criticalrace

theo

ry;

qualitativeapproach;

methodological

bricolage,forexam

ple,

focusgroups,reflexive

autoethnography,story

telling,testimonies

Cross-cultural;critical

andinterpretive

paradigm;explore

own

positionality

Todevelopmore

descriptive

inform

ation

abouthow

Native

American

evaluations

areplayingout

Self-reflectivethroughoutevaluationasking

questionsaboutculturalappropriateness;

methodologicalandepistemologicalhumility

required

51.W

illging

etal.(2006)

United

States:

Curriculum-based

diabetes

prevention

program

forurban

American

Indianwomen

Toassess

thecultural

appropriatenessof

interventionand

providelessonslearned

Participatory;focus

groupsforpilottesting

Participatory;critical

self-exam

inationofown

biasesandposition

Given

callforcultural

competenceneedto

incorporate

culture

ineveryday

practice

Must

accountforbroader

socialcontext;

operationalizationofconceptofculture

requires

considerableflexibility

toaccommodatedifferingvalues,beliefsand

practices;intraculturalvariationisthenorm

;constituent

involvingstrategies

donot

necessarilymeanthat

itwillbeculturally

appropriate(dueto

difference

withintended

audience);focusonculture

asdynam

icprocess

andas

situated

within

abroader

socialandphysicalcontext;needto

assess

ownbiases

52.Zulliand

Frierson(2004)

United

States:Outw

ard

boundprogram

for

African

Americans

Tofocusonaspects

of

evaluationthat

exam

ined

theperceived

influence

onprogram

effectivenessofthe

culturalandeconomic

similarity

ofstaffand

program

participants

Culturally

responsive

evaluation

Culturalresponsiveness

Tobettertake

account

ofculturalcontext

Thesimilarity

ofbackgroundsensures

culturalcompetency

andenablesstaffto

relate

tothestudents

inaway

that

others

may

nothavethecapacityto

do;program

culture

andclimatehaveenorm

ousimpact

onprogram

success


evaluation and the role of the evaluator (e.g., cultural consistency approach, culturally sensitive

approach, culturally responsive approach, culturally competent approach). At the same time, over

43 different rationales for the use of the cross-cultural approach were identified (e.g., moral and ethi-

cal obligations, validity, empowerment, utility), with many studies citing numerous, overlapping

rationales. Theoretical orientations reflected the need to adopt methodologically diverse approaches

(e.g., naturalistic, emancipatory, social justice, critical theory, constructivist, anthropological, and

ethnographic) to better understand the community context or to satisfy external requirements.

Despite the lack of consistent terminology or evaluative approach, studies are predominantly qua-

litative or mixed-method, and many are what might be termed reflective case narratives of evaluator

and/or stakeholder experiences with cross-cultural evaluation.

Research Synthesis

Through our analysis and synthesis of the empirical literature and guided by our initial questions, we

were able to identify seven broad themes or categories that capture strategies, consequences, and

organizing conditions and influences. Having identified emerging categories, we used the

constant-comparative method (Cresswell, 1998) to further refine our initial categories and validate

our preliminary findings. Our seven broad categories are (a) use of participatory and collaborative

approaches, (b) developing culturally specific measures, (c) emergent cultural conceptualizations,

(d) focus on evaluator-stakeholder relationships, (e) evaluator positionality and roles, (f) facilitating

cultural understanding, and (g) methodological dissonance. Although there is some overlap across

categories, we believe these themes to be sufficiently unique so as to merit specific attention. We

now turn to a description of them

Documents

American Journal of Evaluation 2009 Chouinard 457 94 (1)