Moral knowledge and responsibilities in evaluation implementation: When critical theory and responsive evaluation collide

Freeman, M., Preissle, J., & Havick, S. (2010). Moral knowledge and responsibilities in eval-uation implementation: When critical theory and responsive evaluation collide. In M. Freeman (Ed.), Critical social theory and evaluation practice. New Directions for Eval-uation, 127, 45–57.

Moral Knowledge and Responsibilities inEvaluation Implementation: When CriticalTheory and Responsive Evaluation Collide

Melissa Freeman, Judith Preissle, Steven Havick

Abstract

An external evaluation documented what occurred in an inaugural summercamp to teach high school students how to preserve religious freedom by learn-ing about and acting on the history and current state of church–state separationand other first amendment issues. Camp designers hoped to promote religiousdiversity values and civic engagement in youth. An analytic vignette groundedin an inductive analysis of observations, interviews, and document collectionrepresents the competing demands of responsive and critical approaches to eval-uation. Balancing obligations to promote the social well-being of society withresponsibilities to clients and other stakeholders presents challenges that can bemet only by identifying priorities with clients in ongoing dialogue. © WileyPeriodicals, Inc., and the American Evaluation Association.

What does it mean for evaluators to be morally responsible? TheAmerican Evaluation Association (AEA, 2004) Guiding Princi-ples for Evaluators, developed in 1994 and ratified in 2004, sug-

gests an ethics of responsible professional action. These include openlydisclosing the strengths and limitations of the evaluation’s design andprocess, respecting diverse individual needs and perspectives, and maxi-mizing benefits and minimizing harms. The final principle, responsibility

45

4

NEW DIRECTIONS FOR EVALUATION, no. 127, Fall 2010 © Wiley Periodicals, Inc., and the American Evaluation Association. Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/ev.338

46 CRITICAL SOCIAL THEORY AND EVALUATION PRACTICE

for general and public welfare, however, pushes evaluators to go beyond aresponsive evaluation design to one that engages with evaluation practicemore critically and reflexively. It requires that evaluators balance client andstakeholder welfare with that of the broader community (Schwandt, 2007).

We agree with Segerholm (2003), who argues that all evaluationsbecome part of the fabric of everyday institutional and social practice andplay a role in how societies function and change. Embracing a criticalstance, then, means more than criticizing one’s methodological approach; itmeans considering intended and unintended consequences of evaluationdecisions. In a democratic society evaluators are responsible for identifyingand challenging program and evaluation processes that disempower,oppress, exclude, misrepresent, or dismiss particular stakeholders or stake-holder views, practices, and discourses, while alternatively identifying andunderstanding program and evaluation processes that are inclusive, demo-cratic, and empowering. Furthermore, a critical orientation for evaluationis necessarily self-reflexive of its own practice and how that practice servesto empower or disempower our client, those our client serves, and society.This involves the kind of reflexivity Macbeth (2001) describes as “the turn-ing back of an inquiry or a theory or a text onto its own formative possibil-ities” (p. 36). Only through this kind of self-examination can we understandwhat we mean when we talk about our responsibility for general and pub-lic welfare. In this chapter, we examine some of the difficulties of puttingthis principle into practice and consider the role moral discourse might playin supporting its facilitation.

The program we evaluated, and that challenged our responsibility forthe general welfare, was a camp collaboratively run by a liberal religiousgroup and a national advocacy organization devoted to religious diversityin civic life. Combining camp activities (such as cabin council, humanbingo, and field games) with academic sessions (such as lectures on the firstamendment, workshops on conflict resolution and cultural diversity) andadvocacy groups to develop action plans, the collaborating founders fromthe two groups designed a week-long program. Their goal was to provide29 religiously and culturally diverse high school students with what thefounders believed to be the necessary knowledge, skills, and values for the youth to act as leaders in their local communities against threats to reli-gious liberty. The founders’ intention was to develop a program model thatthey could reproduce across the United States. Joining the founders abouta year after the initiative was conceived, we three evaluators participated in theplanning sessions for the program in December 2005, we provided aninterim report the following March, we were on site for the program inAugust 2006, and we submitted our evaluation report to the founders thefollowing fall.

We used a responsive approach because our clients wanted us to docu-ment the evolving understandings and issues stakeholders had about the pro-gram, as well as provide feedback on the quality of the participants’

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

47MORAL KNOWLEDGE AND RESPONSIBILITIES IN EVALUATION IMPLEMENTATION

experience (Abma, 2006; Greene & Abma, 2001). Furthermore, the found-ing team emphasized collaboration, dialogue, and responsiveness to eachother, qualities inherent to responsive evaluation (Abma, 2006), so we builtour participant observations, interviews, and focus groups into their nascentdesign. Our intent was to witness and elicit their value perspectives, agree-ments, and disagreements as they were occurring. Overall, our report was pos-itive because the experiences the youth reported, and we observed, werepositive. However, several issues arose during implementation and after theprogram ended that challenged us to reassess the effect our actions and inac-tions may have had on the collaborative democratic process the foundershad aspired to, but had failed to achieve. This chapter reflects on what wewitnessed happening during program implementation, the ethical issuesthese events raised for us, how we considered them in our report, and howwe might have addressed them in light of what has occurred since dissem-ination of the report. Central to our argument is that dialogue should gen-erate a critical awareness of the interrelatedness of knowledge and values sothat stakeholders can develop a better understanding of the reasons for theiragreements and disagreements while they are having them, not retrospec-tively, when reading the evaluation report (Beiner, 1983, p. 152).

The chapter is organized around a narrative vignette (Erickson, 1986;Humphreys, 2005; Polkinghorne, 1995) highlighting core programmaticissues that were dividing the founding group. The incident that the vignettedramatically reconstructs stands out because it represents the magnitude ofcompeting issues faced by the founding group. It shows the emotionalresponse of the group when faced with several students who did not seemat first to represent the kind of student they had expected (i.e., academicallyengaged), and how conceding to charismatic authority can hamper demo-cratic decision making. In our discussion of these issues, we questionwhether and how we could have played a more active role in pushing for amoral discourse about “what is right to do in a given situation” (Schwandt,2002, p. 7) among the stakeholders involved. Whether explicitly spoken ornot spoken at all, how we act and interact with others, as people or evalua-tors, carries moral value, and in turn this moral value affects the practiceand outcome of the evaluation and the program evaluated.

The Camp Devoted to Religious Diversity in Civic Life

Cast of Characters

The founding team of 20 adults was comprised of people in their 20s and30s, some of whom worked for the camp and had training in youth pro-gramming and others of whom worked for the national organization asadvocates for religious liberty, and people in their 60s and 70s, most ofwhom were retired civic or academic professionals recruited by Frankie, aretired public-relations professional and long-time board member of the



national advocacy organization. Frankie was the impetus behind the pro-gram and the identified leader of the entire group. She had contacted thepeople at the liberal camp, located two curriculum developers to translatethe group’s ideas into a program, and hired the evaluation team to show evi-dence of success for funding purposes. Figure 4.1 depicts the groups asso-ciated with this program.

Vignette: A Program for Diversity Balks When Diversity Arrives

The vignette that follows is written in Melissa Freeman’s voice to portraythe events as one of us experienced them; it draws on field notes, notaudiorecordings, collected during the August program, so none of the attrib-uted quotes can be considered verbatim of what was actually said by us orany of the participants involved. They are written as quotes to retain theevocative purpose of the vignette.

It is mid-afternoon, first full day of the week-long program. I enter the cabinwhere Jude and Steve have been waiting for me while I attended one of theprogram founders’ daily debriefing sessions. “You won’t believe what I havejust witnessed,” I tell them. “They are going to send the Black kids home!”“Are you serious?” Steve questions, “I mean, this morning, while Tim lec-tured, Deangelo was sitting there covering his ears with his hands and lettingout loud sighs, Dominique was playing with her water bottle, and Jorell wasgoing back and forth from being restless to feigning sleep. Certainly that is


NationalAdvocacy

Organization

Program

FoundingGroup

Members

CampCounselors

LiberalReligious

Camp

NAOStaff and

CurriculumDevelopers and

UGAEvaluators

Figure 4.1. Program Personnel in the Program to Promote Diversity inCivic Life


behavior that needs to be corrected, but it isn’t something that should get akid sent home!” “It’s true,” I exclaim. “It didn’t take them long to offer thatas a solution. But it started with Kelli being concerned with the youths’ behav-ior during the teaching and then somehow it developed into a recruitmentissue and then this.”

The first academic session had just ended that morning in the large room. Theissue prompting Kelli, one of the camp’s associates, to call the meeting wasthe different levels of engagement she had noticed among the 29 high schoolstudents during their first academic session on the history of the first amend-ment in the United States. The students had sat in three semicircles in chairsfacing Tim and Marge, the two instructors, while adults not teaching that daysat around near the back. Three Black teens sat restless, making faces and gig-gling, their heads and faces partially buried under large sweatshirt hoods.Among the students were a small handful of interested participants (a Mus-lim, a Sikh, a Christian, two Universalist Unitarians), who responded to thebulk of the questions, while the rest of the diverse group of students sat qui-etly with ambiguous stares on their faces. “What are they thinking?” I won-dered as I too watched quietly while taking note of the instructors’ relianceon lecture peppered with an occasional question–answer sequence. As theytalked, I had wished they would write down the names of the people andplaces they mentioned, and I wished for visuals, diagrams, timelines, anythingto help me follow what they were talking about. So after the youth had leftand Kelli asked for a quick debriefing because she wouldn’t be able to attendthe formal one, I was hopeful and curious.

Quickly and abruptly, she speaks: “I won’t be here for our debriefing thisafternoon so I want to say this now. There are kids who have separated them-selves from the group, who are not very engaged. What I noticed from us wasa lack of response; no one was reacting to it. The adults in back were chat-ting, which really sets a tone that you don’t have to be listening. It is reallyimportant to model the behaviors we want them to do.” “Good for you,”I think, “they need to address their teaching strategies now.” But Tim reactsdefensively and is evidently agitated, and Marge echoes what he has tried toarticulate: “I think we are all aware that we have certain speakers who standout. If we stop to speak to kids who are not behaving it draws attention.” Andthen it’s lunchtime and the discussion is tabled till that afternoon.

After lunch, minus Kelli, they take up where they left off, and as I think backon it and consider its twists and turns, I realize that the talk they needed tohave about the nature of their collaboration never occurred, and that the talkthey did have would take center stage the rest of the week, but would neverget openly addressed.

Tim starts off by apologizing for reacting so strongly to Kelli, but explains thatif she had come up to him and asked to talk about her concern, he wouldhave been less defensive. Frankie agrees, “Perhaps it would have been more



appropriate for her to approach you rather than the larger group. It was yoursession and you had indeed seen the behavior she was referring to, and so[she] should have asked you about the seating.” I am curious how others willreact to Frankie’s statement, which so clearly contradicts their mantra of openand shared deliberation, but Tim wants to say more. “There was a group ofguys sitting over there and saying I’m not buying into what is going on here. I guess I want to allow more tolerance than Kelli. Although I was personallyannoyed when one of the guys started to imitate what was being presented,and I decided that to go up to him would be very embarrassing to him.” Nirman, a member of the national organization, who had been listening qui-etly, jumps in supportively: “You have far more patience than I do, during acamp I once led I had a kid cry because I called him on it. But then he wasengaged. Some of us have different teaching styles. I didn’t see what Kelli saidas nonconstructive and destructive. I saw it as we could have kept the chat-ter in the back down, we were at fault. We can recognize that we have differ-ent teaching styles and help each other out.”

I watch with hope this attempt to discuss the issue as one of teaching and onethat could be solved collaboratively, but Tim, tensing up, builds instead onNirman’s handling of it as a disciplinary issue. “I would have not embarrassedto the extent that you would. I’ve seen you do it and you can pull it off. But I can’t do that, I come off too strongly.” And Imad, new to the team and amember of the national advocacy organization, offers a suggestion for han-dling the discipline: “Maybe the problem can be resolved through the campcounselors. I suggest that the counselors sit with the students and deal withthat.” And then Frankie voices a concern not yet raised: “Then we have theproblem that the counselors may not have the training, and then there is the issue that they are all White and the children are all Black.”

I groan quietly. “Yes, yes,” I think. Although not the only Black studentsamong the youth, the three teens being discussed are the most economicallylimited, so race and class play a role here, but this is not a disciplinary issue.This is a teaching issue, one of needing pedagogical strategies to reach out todifferent kinds of young people. After all, this is supposed to be a program fordiversity and democracy. Could it be that these inner-city teens had neverbeen to a camp before, or attended a history lecture with 26 other youth theydid not know and who practice religions they may never have heard of? Howwere any of these young people supposed to figure out the norms of such aprogram overnight? But certainly someone in this group is going to see thatand say something. And when Irene, a conflict resolution trainer and educa-tor begins to speak, I listen. She is one of two Black faculty members and hasyears of experience handling discriminatory situations. “I’m listening on somany levels about race and culture. It is so hard as a White person to bringthem in and there has to be a buy in, but I don’t know. Why are they here?”

What Irene is asking is a question about expectations. What were the youths’expectations of this program? But Frankie takes it up as a failure in the adults’



recruitment process and explains that some of the people who were soughtout to identify appropriate teens for the program misinterpreted the point ofthe program and sent teenagers who don’t fit “the model.” Before anyone askswhat this model looks like, the conversation becomes intense and difficult tofollow, but the core of it is clearly an attempt to express solutions that revealtwo opposing pedagogical philosophies: one view articulates a democraticpedagogy requiring adaptation and reaching out to the youth who have muchto teach them; the other favors a transmission approach requiring receptivityfrom the youth who have much to learn.

Advocating for reaching out, Nirman draws on sociocultural theories thatargue that one way to understand the inner-city Black youths’ nonassimila-tory patterns is by understanding how different behaviors are associated withdifferent cultural norms. Frankie, however, who understands learning astransmission, worries that what she sees as asocial behavior will affect thebehaviors of other students who are not acting out. Nellie, an associate withthe camp who had been sitting quietly, tries to shift the focus to the program’sresponsibility toward the youth. “There seems to be two levels to this con-versation. What kind of culture are we as faculty creating? What kind of cul-ture do the youth create? What is the team doing to support the facilitation,what kind of culture do we want together? What kind of modeling do wewant to do?” But her words fall on deaf ears as Tim seems beyond discussion:“Personally, I am not willing to tolerate anyone sabotaging this program. We have invested too much. Now if kids are having doubts that this is goingto be a valuable experience, then ok. But if they are committed that this weekis going to be a real drag, I am not willing to put up with that. So how are we going to deal with this? I’m thinking we need to tell the kids upfront thisis what we think you are doing and we are not going to tolerate it. If this iswhat you are doing, there is an airplane waiting for you.”

There is a quick discussion about behavior and race, some asking for patience,others agreeing with Tim, and then Nellie suggests that they let the youthdevelop their own community standards. “We need a little patience; we needa lot of vigilance. We see this over and over again, sometimes race, sometimesfinancial disparities. That’s why I believe so strongly in community stan-dards.” And Frankie jumps on this as a solution: “If the students create a listof community standards they are going to buy in or they aren’t going to buyin to them. But if they don’t we can say if you are not working within that setof rules, you will be sent home.” That said, the conversation is over, and Ireturn to my cabin where Jude and Steve are waiting.

Discussion

This incident opened up multiple challenges for us as evaluators and ashumans sharing a world with others. We were unsure where to begin orwhat issue needed most addressing. We were ashamed that a group of White



liberals like ourselves could profess to support a diverse and democraticworld and then reveal such classist, racist, and undemocratic perspectives.We understood that many of the statements made during this incident werethe result of emotional and physical duress and fear that the program wouldfail; however, we were surprised by the group’s inability to relate pedagogyto behavior, race, and class in this situation.

As it turned out, the youngsters stayed. Neither they nor any of theother teenagers ever again engaged in any behaviors considered problem-atic. Furthermore, all of the adults reached out to the youth in various inter-active activities that succeeded, along with the youths’ own efforts with eachother, in building community. However, that first day set the pattern for theremainder of the week: each day had some hours of didactic presentationfrom adults, bookended with more interactive activities. Adult decisionmaking likewise was set. Rather than clearly articulate and argue throughtheir differing positions, the adults usually deferred to Frankie’s final deci-sion on the matter under discussion.

Our vignette illustrates the different layers of meanings present in mostcomplex human interactions. In a separate publication (Freeman & Preissle,2010) we pursue the pedagogical rift in the group between those assuminga transmissive model of education and those favoring a more transforma-tive model of education. The rift divided the advocacy organization adults,who favored more didactic approaches to education from the camp adultsand we three evaluators, who favored a more interactive approach. Webelieve that the adults’ insistence on compliance to their transmissive effortsprevented them from more fully achieving their goal of promoting religiousliberty in a democracy. In this chapter, however, we move our focus frompedagogy to the social dynamics in which we were ourselves complicit.These dynamics presented a democratic and inclusive façade that hid a moreautocratic and hierarchical reality.

From the beginning the adults voiced the common goal of empower-ing youth leaders in the United States to defend and support religious free-doms, and they articulated a shared desire to reach out to diversecommunities across the country. What never occurred was a deep consid-eration of how these diverse communities might enculturate youth to dif-ferent models of leadership and how the program might adapt to thesedifferent forms of interaction.

Although deferring to Frankie to get things done, not everyone agreedwith her perspective on whom and what the program was for. In our finalfocus group interviews with youth and adults, we believed we were gettingthe kind of diversity of opinion we needed for our report to push the pro-gram founders to recognize and reflect on their own levels of acceptanceand resistance to the diversity of experience the youth brought. This was ahighlight of our assessment. We were stunned, therefore, to later learn howthe differences in adult perspectives were addressed: the National AdvocacyOrganization separated itself from the Liberal Religious Camp and continued



the program on its own under Frankie’s leadership. In our concluding sec-tions we offer reflections on how our responsiveness could have been morecritical.

Moral Knowledge, or, What We Did, or Failed to Do, and Why

An ongoing issue for naturalistic or responsive evaluators concerns themany roles evaluators find themselves taking in the course of an evaluation.Working with stakeholders is a relational, situational, and personalendeavor and involves sensitivity to the emotional, psychological, and socialwell-being of a diversity of people, while also pressing them to share infor-mation that may be at odds with others with whom they also have personalrelationships. It involves, in other words, moral knowledge. Moral knowl-edge involves our capacity to judge when a response or action is called forin a particular situation, what that response should be, and the conse-quences, positive or negative, of the action or response (Connor, 1999).

One of the strong prejudices we brought to the evaluation is a com-mitment to the value of dialogue. We agree with Abma (2006) that the con-ditions for “good” dialogue involve “the willingness of stakeholders toparticipate, to share power, to change in the process, and to be open andrespectful” (p. 34). What we struggled with was helping ensure that “good”dialogue occurred while not disempowering the group’s process by steppingin and taking it over. In hindsight, we believe that we mistook the locus ofour concern. For example, one reason we did not intervene in the conver-sation was out of respect for the group’s own desire for deliberative dialogueand the false belief that intervening in their process would be disempower-ing to them. We believed in their capacities for development because of howthey did confront each other, speak their own minds, uncover the valuesunderneath them, express anxiety, anger, and frustration one minute andreach out, voicing their care and commitment to each other and to theirshared vision the next. Who were we, we thought, to step in and judge thequality of their interactions or predict their outcomes? Furthermore, whenwe did participate in the dialogue to offer our perspective on what we werehearing, our voices merely blended in as one point of view among 20 oth-ers. Our solution was to work from these deliberations and build in con-structivist procedures to elicit further stakeholder perspectives on practiceswe felt fell short of their ideals of democracy and diversity and recirculatethose perspectives in individual interviews and focus groups. What we real-ize now was that our inability to resolve how to integrate ourselves intotheir deliberative process and intervene in the dialogic encounters them-selves meant that our actions, or inactions in this case, reinforced some ofthe patterns of which we were so critical.

Our biggest failure, therefore, was not directly addressing the issue ofpower and how power, especially Frankie’s, halted deliberation of key issues.Not alone in this view, one founding member described her disappointment



in the inability of the adults to work more collaboratively. “I think there wasa lack of clarity around who was effectively in charge . . . . The way it wasdescribed to me, it was a collaboration between [different groups], and I think [instead] there were power struggles. . . . Everyone who wasinvolved followed Frankie’s lead and I think we followed her lead for dif-ferent reasons, all of them not necessarily admirable.”

Dealing with power inequalities as they were occurring would haverequired someone to call on a stronger democratic process and take the lead-ership in developing and putting into practice a critical pedagogy that wouldinvolve us all. The decision of who would take that leadership need nothave been ours to make, but we were in as good a position as others to rec-ommend such an action. We knew alliances were being formed betweensome members of the group, and we witnessed, just as everyone did, howsome viewpoints were being excluded. Power differentials were neveraddressed by us, or anyone else, during implementation. By not adequatelyresponding to the founding team’s desire for input on their collaborativeprocess and actively informing that process, we failed our stakeholders,especially those who found themselves silenced in the group’s process.

Moral Discourse, or, What We Might Have Done Differently

Moral discourse involves paying attention to how we think, talk, act, andcall on moral frameworks within everyday practice such as evaluation. Asparticipants proceed with their activities, evaluators engage them in con-templative discourse about the priorities and values underlying programpractices and stakeholder perspectives, and analysis of the language usedbecomes a source of information about self and others (MacIntyre, 1984;Schwandt, 1989, 2002). When we collect stakeholder perspectives andexperiences during an evaluation, we are collecting bits and pieces of thismoral discourse—what people connected to a program value about it andwish to achieve. In promoting a collaborative approach, the programfounders tried to keep such a discourse open but failed, as we also failed inassisting them. Although a critical approach does not seek to proffer orimpose a solution, how might we have worked out a better integration ofwhat we could offer this group without taking over their process? Howmight we have integrated such a discourse into our practice?

When we conduct an evaluation, we are putting into play our beliefsabout knowledge generation, learning, and transformation. An ethical con-flict for us was sensing that we were acting in ways contrary to our ownbeliefs about these things. There are two pieces we now consider to beessential to the resolution of the conflicts we experienced. The first is mak-ing clear to our client our position on the issues as they arise for us. If weare to be members of a formative team, we must reflect on and make explicithow our own values are guiding our engagement. The second follows fromthe first. As outsiders to a program we are in a special position to articulate



opposing stakeholder values as we hear and observe them and to inform thegroup of how their practices, whether during development or implementa-tion, are favoring or discriminating against these values.

Using the issues raised in the vignette for reflective purposes, howmight we have acted differently in this and other situations? It is easier tospeak critically of the acts of others than it is to acknowledge how quicklywe too can (and do) fall into racist discourses. Rather than consider reflec-tively how we could assist the founders in strengthening their process, wereacted emotionally to their transitory panic and talk of exclusion. Thisresulted in our focusing primarily on understanding the perspectives thatdivided the group at the expense of probing into their collaboration, or lackthereof. What might have happened if we had pointed out that their reac-tion to the youths’ seeming disengagement was likely being distorted byprevalent stereotypes of Blacks and behavior? How might this conversationhave led them to better understand how their fears about behavior and thepossibility of having their own views challenged coincided with their dis-agreement over whom their program was intended for? Finally, how mightthis conversation help those who preferred a transmissive pedagogy betterunderstand how transformational pedagogies help address issues of engage-ment through teaching rather than discipline? Rather than engage them inbetter understanding the effects of stereotypical thinking and assist them in their efforts to work against it, we too balked, and allowed our shock anddiscomfort to interfere with clear thinking about our responsibilities.

Responsibility for General and Public Welfare, or, Prioritizing Our Commitments

Looking back over our evaluation, we believe there were many areas inwhich we were successful. We developed strong relationships with all theprogram stakeholders, even the ones we disagreed with. We presented theirperspectives on the strengths and weaknesses of the program in a fair andcomprehensive way. We paid attention to how the youth were being treatedand how they benefited from the program and were pleased to be able toreport positively on both accounts. During implementation the adults nevermistreated the teenagers or discriminated against them personally. Althoughthe conflict represented in the vignette could be understood as an expres-sion of the founders’ fears at being ill-prepared for a diversity of youthbroader than they had anticipated, it was also an expression of their under-standing of how they believed their program ought to be designed so as tobenefit society. Our critical retrospection, in that it is critical of the elitistview espoused by some of the founders, is our expression of how we believeprograms and practices such as these ought to benefit society.

What we understand now in retrospect was how the differences in ped-agogical preferences represented differences in how program founders under-stood the responsibility of the program to the client. In our view and the view



of those who professed a transformative pedagogy, the youth were the pro-gram’s client. The program should provide a rich educational experience thatwould transform the youths’ understanding of their own capacities for empa-thy, advocacy, and action and allow them to contribute to their communitiesand society in their own way. Frankie and others who supported her viewbelieved the client to be the mission of the national organization. In this view, you do not turn to youth for understanding and guidance because youalready know what the mission is. The youth are an abstraction. They are thefuture of a nation, and as such it is the nation that the program takes to beits client. Had we reflected more critically on AEA’s principle of responsibil-ity for general and public welfare, we might have been in a better position tohelp the program founders understand that their disagreement was in howthey interpreted their responsibilities to society. Moving the discourse awayfrom pedagogy to one of civil society, a discourse all the founders were wellversed in, might have greatly assisted their process.

Understanding how we interpret our responsibility to society is crucialto understanding how to guide our actions as program evaluators. In theend, we overlooked opportunities for deliberative discourse and we stayedsilent while Frankie enacted a business model of collaboration where oneleader solicits everyone’s input, but then individually makes a final decision,and where leadership itself is assumed to be based on meritorious and elit-ist criteria. We now know we were enchanted by the active and dialogic,albeit ineffectual, engagement of our stakeholders, swayed by our own advo-cacy for their mission, and diverted by our concern for them as individualsfrom our commitment to a democratic society. We hope that by making vis-ible what are very common, but difficult to address, occurrences and con-versations in diversity-oriented programs, our story contributes to theimpressive work in evaluation that supports dialogue, deliberation, diver-sity, and a democratic society, as well as programs like the one we evaluated,that seek ways to foster these values.

Acknowledgments

We would like to thank the youth and adults for their participation in thisevaluation study and for their commitment to creating a better world.Thanks are also due to Kathryn Roulston and Jodi Kaufmann for feedbackon earlier versions of this chapter. Finally, we are grateful for the fundingwe received from the University of Georgia and the unnamed nonprofitorganization.

References

Abma, T. A. (2006). The practice and politics of responsive evaluation. American Jour-nal of Evaluation, 27(1), 31–43.

American Evaluation Association (AEA). (2004). American Evaluation Association guid-ing principles for evaluators. Fairhaven, MA: Author.



Beiner, R. (1983). Political judgment. Chicago: University of Chicago Press.Connor, W. R. (1999). Moral knowledge in the modern university. Ideas, 6(1). Retrieved

October 3, 2007, from http//:nationalhumanitiescenter.org/ideasv6n1/connormoral.htmErickson, F. (1986). Qualitative methods in research on teaching. In M. C. Witrock (Ed.),

Handbook of research on teaching (3rd ed., pp. 119–161). New York: Macmillan.Freeman, M., & Preissle, J. (2010, July-August). Pedagogical ethical dilemmas in a

responsive evaluation of a leadership program for youth. The International Journal ofQualitative Studies in Education, 23(4), 463–478.

Greene, J. C., & Abma, T. A. (Eds.). (2001). Responsive evaluation. New Directions forEvaluation, 92.

Humphreys, M. (2005). Getting personal: Reflexivity and autoethnographic vignettes.Qualitative Inquiry, 11(6), 840–860.

Macbeth, D. (2001). On “reflexivity” in qualitative research: Two readings and a third.Qualitative Inquiry, 7(1), 35–68.

MacIntyre, A. (1984). After virtue: A study in moral theory (2nd ed.). Notre Dame, IN:University of Notre Dame Press.

Polkinghorne, D. E. (1995). Narrative configuration in qualitative analysis. QualitativeStudies in Education, 8(1), 5–23.

Schwandt, T. A. (1989). Recapturing moral discourse in evaluation. EducationalResearcher, 18(8), 11–16.

Schwandt, T. A. (2002). Evaluation practice reconsidered. New York: Peter Lang.Schwandt, T.A. (2007). On the importance of revisiting the study of ethics in evalua-

tion. In S. Kushner & N. Norris (Eds.), Dilemmas of engagement: Evaluation and thenew public management. Advances in program evaluation (Vol. 10, pp. 117–127). Ams-terdam: Elsevier JAI.

Segerholm, C. (2003). Researching evaluation in national (state) politics and adminis-tration: A critical approach. The American Journal of Evaluation, 24(3), 353–372.

MELISSA FREEMAN is associate professor of qualitative research methodologiesin the College of Education at the University of Georgia; her research focuses oncritical, hermeneutic, and relational approaches to educational research andevaluation.

JUDITH PREISSLE is the 2001 Distinguished Aderhold Professor in the College ofEducation, University of Georgia, and an affiliated faculty member of UGA’sInstitute for Women’s Studies. She teaches, researches, and writes in educationalanthropology, qualitative research, feminist studies, and ethics.

STEVEN HAVICK is a doctoral candidate in the University of Georgia’s Depart-ment of Elementary and Social Studies Education and currently teaches U.S. his-tory at Salem High School outside of Atlanta.


Documents

Moral knowledge and responsibilities in evaluation implementation: When critical theory and responsive evaluation collide