The Evaluation Process as a Three-Act Play or post, copy, · be successful. The evaluation process itself generates the plot of the play, which varies from program to program and

17

The Evaluation Process as a Three-Act Play

• Evaluation as a Three-Act Play{{ Act I: Asking the Questions{{ Act II: Answering the Questions{{ Act III: Using the Answers in Decision Making

• Role of the Evaluator • Evaluation in a Cultural Context • Ethical Issues • Evaluation Standards • Summary • List of Terms • Study Questions

2

P erforming a health program evaluation involves more than just the

application of research methods. The evaluation process is composed

of specific steps designed to produce information about a program’s

performance that is relevant and useful for decision makers, managers, program

advocates, health professionals, and other groups. Understanding the steps and

their interconnections is just as fundamental to evaluation as knowledge of the

quantitative and qualitative research methods for assessing program performance.

There are two basic perspectives on the evaluation process. In the first perspective—the

rational-decision-making model—evaluation is a technical activity, in which

research methods from the social sciences are applied in an objective manner to

produce information about program performance for use by decision makers (Faludi,

Copyright ©2016 by SAGE Publications, Inc. This work may not be reproduced or distributed in any form or by any means without express written permission of the publisher.

Do not

copy

, pos

t, or d

istrib

ute

Prologue18

1973; Veney & Kaluzny, 1998). Table 2.1 lists the elements of the rational-decision-

making model, which are derived from systems analysis and general systems theory

(Quade & Boucher, 1968; von Bertalanffy, 1967). The model is a linear sequence of steps

to help decision makers solve problems by learning about the causes of the problems,

analyzing and comparing alternative solutions in light of their potential consequences,

making a rational decision based on that information, and evaluating the actual

consequences. Today, the systematic comparison of alternative approaches in Step 3

is often referred to as health policy analysis, which compares the potential, future

advantages and disadvantages of proposed, alternative policy options to reduce or solve

a health care issue or population health problem (Aday et al., 2004; Begley et al., 2013).

In practice, however, the evaluation of health programs rarely conforms to the

rational-decision-making model. Because politics is how we attach values to facts

in our society, politics and values are inseparable from the evaluation of health

programs (Palumbo, 1987; Weiss, 1972). For instance, the public health value of

“health for everyone” conflicts with the differences in infant mortality rates across

racial/ethnic groups, and politics is the use of values to define whether this difference

is a problem and what, if anything, should be done about it. Consequently, in the

second perspective, evaluations are conducted in a political context in which a variety

of interest groups compete for decisions in their favor. Completing an evaluation

successfully depends greatly on the evaluator’s ability to navigate this political terrain.

This chapter introduces the political nature of the evaluation process, using the

metaphor of the evaluation process as a three-act play. The remaining chapters of the

book are organized around each act of the play. The last two sections of this chapter

address the importance of ethics and cultural context in conducting evaluations and

the role of the evaluator in the evaluation process.

1. Define goals to achieve or problems to solve

2. Identify alternative approaches (programs, policies, or other interventions) that might achieve the goals or solve the problems

3. Systematically assess and compare the future outcomes of the alternative approaches

4. Choose one or more approaches based on the results of the systematic assessment

5. Implement the approach(es)

6. Evaluate the approach(es)

TABLE 2.1 ● Elements of the Rational-Decision-Making Model


Do not

copy

, pos

t, or d

istrib

ute

Chapter 2 • The Evaluation Process as a Three-Act Play 19

Evaluation as a Three-Act Play

Drawing from Chelimsky’s earlier work (1987), I use the metaphor of a “three-act play” to describe the political nature of the evaluation process. The play has a variety of actors and interest groups, each having a role, and each entering and exiting the political “stage” at different points in the evaluation process. Evaluators are one of sev-eral actors in the play, and it is critical for them to understand their role if they are to be successful. The evaluation process itself generates the plot of the play, which varies from program to program and often has moments of conflict, tension, suspense, quiet reflection, and even laughter as the evaluation unfolds.

Table 2.2 presents the three acts of the play, which correspond to the basic steps of the evaluation process (Andersen, 1988; Bensing et al., 2004). The play begins in the political realm with Act I, in which evaluators work with decision makers to define the questions that the evaluation will answer about a program. This is the most important act of the play, for if the questions do not address what decision makers truly want to know about the program, the evaluation and its findings are more likely to have little value and use in decision making. In Act II, the research methods are applied to answer the questions raised in Act I. Finally, in Act III, the answers to the evaluation questions are disseminated in a political context, providing insights that may influ-ence decision making and policy about the program.

Act I: Asking the Questions

In Act I, the evaluation process begins when decision makers, a funding organization, or another group authorize the evaluation of a program. In general, decision mak-ers, funders, program managers, and other groups may want to evaluate a program

ACT I: Asking the Questions

Scene 1: Developing a policy question

Scene 2: Translating the policy question into an evaluation question

ACT II: Answering the Questions

Scene 1: Developing the evaluation design to answer the questions

Scene 2: Developing the methods to carry out the design

Scene 3: Conducting the evaluation

ACT III: Using the Answers in Decision Making

Scene 1: Translating evaluation answers back into policy language

Scene 2: Developing a dissemination plan for evaluation answers

Scene 3: Using the answers in decision making and the policy cycle

TABLE 2.2 ● Evaluation as a Three-Act Play


Do not

copy

, pos

t, or d

istrib

ute

Prologue20

for overt or covert reasons (Rossi et al., 2004; Weiss, 1972, 1998a). Overt reasons are explanations that conform to the rational-decision-making model and are generally accepted by the public (Weiss, 1972, 1998a). In this context, evaluations are conducted to make decisions about whether to

• Continue or discontinue a program • Improve program implementation • Test the merits of a new program idea • Compare the performance of different versions of a program • Add or drop specific program strategies or procedures • Implement similar programs elsewhere • Allocate resources among competing programs

Because Act I occurs in the political arena, covert reasons for conducting evalua-tions also exist (Weiss, 1972, 1998a). Decision makers may launch an evaluation to

• Delay a decision about the program • Escape the political pressures from opposing interest groups, each wanting a

decision about the program favoring its own position • Provide legitimacy to a decision that already has been made • Promote political support for a program by evaluating only the good parts of

the program and avoiding or covering up evidence of program failure

Whether a program is evaluated for overt or covert reasons may depend on the values and interests of the different actors and groups in the play (Rossi et al., 2004; Shortell & Richardson, 1978).

A stakeholder analysis is an approach for identifying and prioritizing the interest groups in the evaluation process and defining each group’s values and inter-ests about the health program, policy, or health system reform and the evaluation (Brugha & Varvasovszky, 2000; Page, 2002; Rossi et al., 2004; Sears & Hogg-Johnson, 2009; Varvasovszky & Brugha, 2000; Weiss, 1998a). The term stakeholder was created by companies to describe non-stockholder interest groups that might influ-ence a company’s performance or survival (Brugha & Varvasovszky, 2000; Patton, 2008). In evaluation, a stakeholder is an individual or a group with a stake—or vested interest—in the health program and the evaluation findings (Patton, 2008). Based on definitions in the literature, a stakeholder is an individual, a group, or an organization that can affect or is affected by the achievement of the health program’s objectives, or the evaluation process or its findings (Bryson et al., 2011; Page, 2002). Stakeholders tend to have two broad types of stakes (Page, 2002). The first type is a stake in an investment in something of value in the health program, such as financial or human resources. For example, funders of the health program and evaluation have stakes in both. The second type is a stake in the activity of the health


Do not

copy

, pos

t, or d

istrib

ute


program; in other words, a stakeholder might be placed at risk or experience harm if the activity were withheld. For instance, providers who receive revenue from deliver-ing a medical treatment to patients have stakes in evaluations of the effectiveness of the treatment. If the evaluations show that the treatment has few health benefits, the treatment may be delivered less often and ultimately lead to a loss in revenue.

A stakeholder analysis provides essential information for planning the evaluation in a political context, including a better understanding of the program’s and the eval-uation’s political context, the identification of common goals and contentious issues among the stakeholders, and the creation of an evaluation plan that addresses stake-holder interests as much as possible (Sears & Hogg-Johnson, 2009). Rossi et al. (2004) suggest the following best practices for stakeholder analysis:

• Identify stakeholders at the outset and prioritize those with high vested interests in the health program and evaluation.

• Involve stakeholders early because their perspectives may influence how the eval-uation is carried out.

• Involve stakeholders continuously and actively through regular meetings, provid-ing assistance with identifying the evaluation questions and addressing study design issues, and requesting comments on draft reports.

• Establish a structure by developing a conceptual framework for the evaluation to build a common understanding of the health program and evaluation, promote focused discussion of evaluation issues, and keep everyone in the evaluation process “on the same page.” (Chapter 3 addresses this conceptual frameworks in detail)

In identifying and prioritizing stakeholders, most, if not all, evaluations of health programs have multiple stakeholders with different interests. Page (2002) suggests prioritizing stakeholders based on (a) their power to influence the health program or evaluation; (b) whether a stakeholder’s actions and perspectives are perceived to be legitimate and, therefore, should be taken into account in the evaluation; and (c) urgency—that is, whether a stakeholder’s interests call for immediate attention in the evaluation. Stakeholders with all three attributes tend to have the highest priority in the stakeholder analysis.

Figure 2.1 presents a power-interest grid, which is a tool for identifying the stakeholders and rating roughly their relative power and interest in the evaluation (Bryson et al., 2011). The grid is a two-by-two matrix, where power, ranging from low to high, is shown in the columns, and interest, also ranging from low to high, is shown in the rows. Power is defined as the ability of stakeholders to pursue their interests (or who has the most or least control over the program or direction of the evaluation), whereas interest refers to having a political stake in the program and evaluation (or who has the most to gain or lose from the evaluation). The goal is to sort each stakeholder into one of the four mutually exclusive cells in the matrix: players, subjects, context setters, and crowd. Players are key stakeholders and potential


Do not

copy

, pos

t, or d

istrib

ute

Prologue22

users of evaluation results, assuming that the questions posed in Act I address at least some or all of those interests. Subjects may become more engaged in the evaluation by adopting a participatory or empowerment approach to advance their interests, as explained later in this chapter. Context setters’ interests may change, depending on the results of the evaluation, and obtaining their buy-in may become essential as the evaluation process unfolds. The spread of stakeholders across the four cells may reveal commonalities among them, which may be used to build stakeholder buy-in and collaboration in the evaluation process.

Evaluations of health programs tend to have similar stakeholders (Rossi et al., 2004; Shortell & Richardson, 1978). Policymakers and decision makers often autho-rize evaluations to supply clear-cut answers to the policy problems they are facing, such as whether to continue, discontinue, expand, or curtail the program (Bensing et al., 2003). The funding agency may want to evaluate a program to determine its cost-effectiveness and discover whether the program has any unintended, harmful effects. Rossi et al. (2004) suggest that the policymakers and evaluation funders are the top stakeholders in the evaluation process. The organization that runs the pro-gram may be interested in an evaluation to demonstrate to interest groups that the program works, to justify past or future expenditures, to gain support for expand-ing the program, or simply to satisfy reporting requirements imposed by the fund-ing agency.

FIGURE 2.1 ● Stakeholder Power Versus Interest Grid

Source: “Working With Evaluation Stakeholders: A Rationale, Step-Wise Approach and Toolkit,” by J. M. Bryson, M. Q. Patton, & R. A. Bowman, 2011, Evaluation and Program Planning, 34(1), p. 5.

Subjects –have a

significantinterest, butlittle power

Players –have a

significantinterest and

substantial power

Context Setters –have substantialpower, but littledirect interest

Crowd – havelittle interest andnot much power

Hig

hLo

wIn

tere

st

HighLowStakeholder Power


Do not

copy

, pos

t, or d

istrib

ute


Program administrators may support an evaluation because it can bring favorable attention to a program that they believe is successful, which may help them earn a promotion later on. Administrators also may use an evaluation as a mechanism for increasing their control over the program, or to gather evidence to justify expanding the program, or to defend the program against attacks from interest groups that want to reduce or abolish it.

Alternatively, contextual stakeholders, or organizations or groups in the immedi-ate environment of the program, which either support or oppose the program, may advocate for an evaluation, with the hope of using “favorable” results to promote their point of view in Act III of the evaluation process. The public and its various interest groups may endorse evaluations for accountability or to ensure that tax dollars are being spent on programs that work. The public also may support evaluations because their findings can be a source of information—in the mass media, on the Internet, in journals, and elsewhere—about the merits of a health program or health system reform, such as the Patient Protection and Affordable Care Act (ACA).

Program evaluators may want to conduct an evaluation for personal reasons, such as to earn an income or to advance their careers. Alternatively, evaluators may sympa-thize with a program’s objectives and see the evaluation as a means toward promoting those objectives. Other evaluators are motivated to evaluate because they want to con-tribute to the discipline’s knowledge by publishing their findings or presenting them at conferences. In addition, the larger evaluation and research community, composed mainly of evaluation professionals, may have interests in the methods and findings of the evaluation.

After the stakeholders are identified and their relative power and interests are defined, a grid is constructed, as shown in Figure 2.2, displaying each stakeholder’s initial support versus opposition to the program and the proposed evaluation. The power-position grid provides information for planning the evaluation, such as devel-oping a strategy for engaging stakeholders in the evaluation (Preskill & Jones, 2009) and taking steps to address explicitly the concerns of supporters and opponents in the evaluation process. Once the evaluation’s findings and recommendations are known, the grid offers information for planning the communication strategy to disseminate evaluation results to stakeholders and the public.

Table 2.3 presents a brief case study of a stakeholder analysis that Sears and Hogg-Johnson (2009) performed for an evaluation of a pilot program in Washington state’s workers’ compensation system, which provides health insurance coverage for workers who are injured on the job. The pilot program was authorized by the Washington state legislature in a contentious political context. Key findings of the stakeholder analysis were the identification of key stakeholders, their values, whether they sup-ported or opposed the pilot program at the outset, and what evaluation questions the stakeholders wanted the evaluation to address.

Act I, “Asking the Questions,” has two parts, or scenes. In Scene 1, evaluators work with decision makers and other groups to develop one or more policy questions about the program, based on findings from the stakeholder analysis (see Chapter 3). A policy


Do not

copy

, pos

t, or d

istrib

ute

Prologue24

In Washington state, if a worker is injured on the job, the worker may apply for health care benefits paid by the Washington state workers’ compensation system. To claim benefits, an injured worker must be seen by an authorized attending physician, who completes and signs the worker’s acci-dent report form and certifies work disability compensation. These documents are submitted to the Washington Department of Labor and Industries (L&I), which functions as the public insurer and administers the claims. In 2004, the Washington state legislature approved Substitute House Bill (SHB) 1691, a 3-year pilot program giving nurse practitioners (NPs) the authority to function as attending physicians. That is, for the first time in the system’s history, NPs were allowed to sign and certify workers’ claims. The legislature approved the pilot program because of concerns about access to health care for injured workers in rural areas, delays in reporting injury claims, and efforts by NPs to expand their scope of practice. Because some stakeholders expressed concerns that authorizing NPs to function as attending physicians would increase the system’s costs (which were covered through employer contributions), the legislature approved an evaluation of the pilot. The evaluation was conducted in a highly political context because its findings would influence whether the pilot would become permanent.

TABLE 2.3 ● Case Study of Stakeholder Analysis: Pilot Program of the Washington State Workers’ Compensation System

FIGURE 2.2 ● Stakeholder Power Versus Support or Opposition Grid

Source: “Working With Evaluation Stakeholders: A Rationale, Step-Wise Approach and Toolkit,” by J. M. Bryson, M. Q. Patton, & R. A. Bowman, 2011, Evaluation and Program Planning, 34(1), p. 9.

WeakSupporters

WeakOpponents

StrongSupporters

StrongOpponents

Supp

ort

Opp

osit

ion

StrongWeakStakeholder Power


Do not

copy

, pos

t, or d

istrib

ute


Source: Adapted from “Enhancing the Policy Impact of Evaluation Research: A Case Study of Nurse Practitioner Role Expansion in a State Workers’ Compensation System,” by J. M. Sears & S. Hogg-Johnson, 2009, Nursing Outlook, 57(2), pp. 99–106.

In the stakeholder analysis, the following key stakeholder groups were identified: an organized nurse practitioner group, an organized physician group, employer organizations, organized labor, and L&I. Semistructured interviews were conducted with representatives from the stakeholder groups to collect information about (a) official and/or unofficial position on the legislation and rationale for that position; (b) perspective on the political process; (c) comments on the bill’s implementation and early impacts; and (d) comments about the evaluation design.

Based on the interviews, the stakeholders’ values and political position on SHB 1691 are summa-rized in the table below. Organized nursing supported the measure, whereas organized medicine opposed the bill because of concerns about quality of care and scope of practice/turf issues. Employer organizations had concerns the bill would increase utilization and costs, and that NPs would be more likely than physicians to endorse workers’ claims. Organized labor supported the bill because it increased access to care for injured workers.

Stakeholder Values and Positions on Expanding Nurse Practitioner (NP) Roles in SHB 1691

Stakeholder (organized groups)

Protect Scope of Practice

Improve Access

Low Costs

Quality Care

System Efficiency

Initial Position

NPs X X X X Support

Physicians X X X Oppose

Employers X X Oppose

Labor X X X X Support

L&I X X X Neutral

A preliminary list of evaluation questions was drafted based on the language in SHB 1691. Interview results were used to refine the preliminary evaluation questions and design. In particular, the number of evaluation questions was expanded to address the stakeholders’ information needs, such as the pilot’s effects on access to care, accident report filing times, and change in number of claims filed. The revised evaluation plan was presented to the advisory committee of the workers’ compensation system, which had representatives from the key stakeholders, to gain further comments and refine the evaluation plan. Efforts were made to involve stakeholders continuously throughout the evaluation process.

question is a general statement indicating what decision makers want to know about the program. Decision makers can include the funding agency, the director of the organization that runs the program, the program’s manager and staff, outside interest groups, and the program’s clients. Together, they constitute the play’s “audience,” and the objective of the evaluation is to produce results that will be used by at least some members of a program’s audience.

Although decision makers may authorize an evaluation of a health program for a variety of reasons, many evaluations are performed to answer two fundamental


Do not

copy

, pos

t, or d

istrib

ute

Prologue26

questions: “Did the program succeed in achieving its objectives?” and “Why is this the case?” For some programs, however, decision makers may want to know more about the program’s implementation than about its success in achieving its objec-tives. For example, questions about achieving objectives may be premature for new programs that are just finding their legs, or when decision makers want to avoid information about the program’s successes and failures, which may generate con-troversy downstream in Act III of the evaluation process. In these and other cases, the basic policy question becomes, “Was the program implemented as intended?” In general, as the number and diversity of decision makers from different interest groups increase, the number of policy questions about the program may increase greatly, which may decrease the likelihood of finding common ground and reaching consensus on the evaluation’s purpose and key questions.

When a program addresses a controversial, political issue, heated debates may arise among decision makers and interest groups about what questions should and should not be asked about the program. Evaluators can play an important role when they facilitate communication among the decision makers and interest groups to help them form a consensus about what policy questions to ask about the program. In addition to moderating the discussions, evaluators also can be active participants and pose their own policy questions for decision makers to consider. If the program is already up and running, evaluators can support the discussions by providing descrip-tive information about program activities that may help decision makers formulate questions or choose among alternative questions.

If the play is to continue, Scene 1 ends with one or more well-defined policy ques-tions endorsed by decision makers and, in some contexts, by at least some interest groups. The play may end in Scene 1, however, if no questions about the program are proposed or if decision makers cannot agree on a policy question or what the program is trying to accomplish (Rossi et al., 2004; Weiss, 1998a). For covert reasons, decision makers may place stringent limits on what questions can and cannot be asked when they want to avoid important issues, or possibly to cover up suspected areas of pro-gram failure. Under these conditions, evaluation findings may have little influence on people’s views of the program, and consequently, there is little value in conducting the evaluation (Weiss, 1998a).

Once one or more policy questions are developed, Scene 2 begins and the policy questions are translated into feasible evaluation questions. In Scene 2, the evaluator is responsible for translating a general policy question, such as “Does the program work?”, into a more specific evaluation question, such as “Did the smoking prevention program reduce cigarette smoking behavior among adolescents between the ages of 13 and 15?” To ensure that the evaluation will produce information that decision makers want in Act III of the play, decision makers should review the evaluation questions and formally approve them before advancing to the next act of the play.

In Scene 2, the evaluator also is responsible for conducting a feasibility assess-ment (Bowen et al., 2009; Centers for Disease Control and Prevention, 1999; Melnyk


Do not

copy

, pos

t, or d

istrib

ute


& Morrison-Beedy, 2012; Rossi et al., 2004; Weiss, 1998a). Before going ahead with the evaluation, the evaluator should confirm that adequate resources, including time and qualified staff (or consultants), exist to conduct the evaluation for the budgeted amount of money. The evaluator should verify that data required for answering the questions are available or can be collected with minimal disruption and that a suf-ficient number of observations will exist for subsequent quantitative or qualitative data analyses (see Chapter 7). For quantitative evaluations, a key issue is whether the projected number of cases will have adequate statistical power. If the evaluation will engage staff in multiple sites, the evaluator should request a letter indicating a site’s agreement to participate in the evaluation. The feasibility assessment also should confirm whether the program has matured and has established, stable routines. Sta-ble programs are preferred because the reasons for program success or failure can be identified more readily than in unstable programs. With stable programs, evaluation findings based on data collected a year ago have a better chance of still being relevant today. In contrast, when a program is changing continually, the findings obtained at the end of the evaluation process may apply to a program that no longer exists. If no insurmountable obstacles are encountered and the evaluation appears to be feasible, the evaluator and the other actors in the play have a “green light” to proceed to the next act of the evaluation process.

Act II: Answering the Questions

In Act II, the evaluation is conducted. Evaluators apply research methods to produce qualitative and quantitative information that answers the evaluation questions raised in Act I.

Evaluations may be prospective or retrospective. In a prospective evaluation, the eval-uation is designed before the program is implemented, as shown in Table 2.4. Prospec-tive evaluations are ideal because a greater number of approaches can be considered for evaluating a program. With greater choice, evaluators have more flexibility to choose an evaluation approach with the greatest strengths and fewest weaknesses. In addition, evaluators have more freedom to specify the information they want to collect about the program and, once the program is implemented, to ensure that the information is actually gathered.

In contrast, retrospective evaluations are designed and conducted after a program has ended, and a smaller number of alternative approaches usually exist for evaluating such programs. Historical information about the program may exist in records and computer files, but the information may not be useful for answering key questions about the pro-gram. In retrospective evaluations, choice of design and availability of information are almost always compromised, which may limit what can be learned about the program.

Causation is an intrinsic feature of prospective and retrospective evaluations. All health programs have an inherent assumption that the program is expected to cause change to achieve its objectives. Program theory refers to the chain of causation,


Do not

copy

, pos

t, or d

istrib

ute

Prologue28

or the pathways or mechanisms, through which a health program is expected to cause change that leads to desired beneficial effects and avoids unintended consequences (Alkin & Christie, 2005; Donaldson, 2007; Weiss, 1995). Program theories are always probabilistic rather than deterministic; few, if any, interventions invariably produce the intended effects (Cook & Campbell, 1979; Shadish et al., 2002).

A health program’s causal assumptions may or may not be based on formal the-ory. Drawing from Merton (1968), Chen (1990), Donaldson (2007), and Krieger (2011), a program’s causal assumptions may be grounded on discipline theory, or what Merton refers to as “grand” theories that are intended to explain a wide range of human behaviors by positing “what causes what.” For example, a health insurance plan may implement an office visit copayment that requires plan members to pay the medical office $10 out of pocket for each medical visit, to reduce health care utilization (Cherkin et al., 1989). Economic theory would suggest that an increase in members’ out-of-pocket costs would likely reduce the members’ utilization (Newhouse et al., 1993).

Alternatively, a program’s causal assumptions may be based on middle-range the-ory (Merton, 1968), or theory designed to explain specific behaviors. For example, the health belief model is an individual-level theory of health behavior change (Glanz et al., 2008). A health promotion program might apply core concepts of the health belief model to increase the number of older adults receiving flu shots by chang-ing their beliefs about their perceived susceptibility and the severity of the illness (Valente, 2002). A vast number of middle-range theories exist in the literature, which may serve as the conceptual foundation for a program’s causal assumptions (for exam-ple, Figures 1.1 and 1.2 may be classified as middle-range theories to explain the pro-cesses and outcomes of health care and public health systems, which may be applied to design and evaluate system interventions). Also, a single health program may be

1. Define the problem

• What is the problem? • Why is it a problem? • What are its attributes? • Target population • Needs assessments

2. Design a program and specify objectives to address the problem

3. Develop the program’s intervention(s)

4. Design the evaluation

5. Implement the program

6. Conduct the evaluation

7. Disseminate findings to decision makers

TABLE 2.4 ● The Prospective Evaluation Process


Do not

copy

, pos

t, or d

istrib

ute


grounded on multiple, interrelated theories, particularly when programs address com-plex social and health problems that are difficult to change. As an illustration, the Parent-Child Assistance Program (P-CAP) conducts paraprofessional home visits over 3 years for extremely high-risk, alcohol- and drug-abusing women to achieve benefi-cial outcomes for the women and their children (Grant et al., 1999; Grant et al., 2005). P-CAP integrates relational theory, motivational interviewing and stages of change theory, and harm reduction theory to construct the goals and protocol of the parapro-fessional intervention.

The social ecological framework also endorses interventions based on multiple the-ories. The framework posits that individual and population health are influenced by social conditions at multiple levels: public policies, institutions, neighborhoods and communities, social relationships, and others (Glanz & Bishop, 2010; Golden & Earp, 2012; National Cancer Institute, 2005). The framework recommends intervening at multiple levels to increase the likelihood of improving individual health or reducing health disparities. Each level of the social ecological model has its own set of behav-ioral theories, and multilevel interventions are developed by integrating the theories across the levels of the framework. For instance, a dietary program to reduce workers’ fat consumption might draw from individual and institutional theories to develop a two-level intervention: (a) disseminating diet information to workers and (b) increas-ing healthy foods in the company cafeteria (National Cancer Institute, 2005).

Theory is important in the evaluation process because programs are more likely to be effective when they are based on theory, compared to those lacking a theoretical base (Finney & Moos, 1989; Glanz & Bishop, 2010). Unfortunately, health program evaluation is plagued generally by a dearth of theory in published evaluations (Fleury & Sidani, 2012). Painter et al. (2005) examined 193 publications in 10 leading health journals from 2000 to 2005. Of those, only about one-third used theory in some way, and less than 20% rigorously applied theory. Conn et al. (2008) reviewed 141 published evaluations of nursing interventions, and although about half mentioned theory, there was little explanation of how the concepts of theories were linked to interventions. Michie, Jochelson, et al. (2009) performed a systematic review of 9,766 publications to identify effective interventions for changing smoking, physical activ-ity, and diet in low-income populations. Thirteen papers met inclusion criteria, and only six studies mentioned theory, with little information about how theory was used or informed intervention development—a problem found in many other interven-tion studies (Michie, Fixen, et al., 2009). Similarly, interventions to improve patient safety in health care, alter contraception use, or reduce unintentional injury are based rarely on theory (Foy et al., 2011; Lopez et al., 2013; Trifiletti et al., 2005). In contrast, theory has been used extensively in some areas, such as the development and testing of HIV-prevention strategies and the prevention of sexually transmitted infections (Albarracin et al., 2005; Aral et al., 2007; Fishbein, 2000).

In short, as Lopez and colleagues (2013) note, the lack of guiding theory for a health program or health system intervention “is akin to having no physiologic basis for a medical intervention” (p. 3). In Act I of the prospective evaluation process, evaluators


Do not

copy

, pos

t, or d

istrib

ute

Prologue30

may improve the quality of the evaluation by advocating for the application of dis-cipline and middle-range theories to develop the content of the intervention and to guide the evaluation methods (French et al., 2012; J. Green, 2000). Taking this action implies that evaluators are educated in at least one discipline theory as well as middle-range theories to guide their work.

In Act II, evaluators develop one or more evaluation designs to answer each evaluation question raised in Act I. There are two basic types of evaluation designs. Impact evaluations (also known as outcome evaluations or summative evaluations) address the first policy question and use experimental or quasi- experimental designs to estimate program effects (see Chapters 4 and 5). Experimental designs use randomization to determine whether observed outcomes are due to the program. In many programs, however, randomization is not possible because laws prohibit excluding groups from the program, logistics prevent random assignment to program and control groups, or the evaluation is performed after the program ends, among other reasons. In these cases, quasi-experimental designs (such as interrupted time series, regression discontinuity analysis, and nonequivalent control group designs) are often used to estimate program effects. In general, the greater the political controversy in Act I, the greater the importance of having a rigorous impact design that can withstand scrutiny when the results of the evaluation are released to the public.

Evaluation of program implementation, which is also known as process evaluation, addresses the second basic policy question and attempts to explain why programs do or do not achieve their objectives by examining how they were imple-mented (see Chapter 6). Implementation evaluations typically are designed to answer the following questions:

• Was the program implemented as intended? • Why did the program work or not work as intended? • Did the program reach its intended target group? • What services did people in the program receive? • Were people satisfied with the program’s services? • What is the average cost for each person who received the program? • Did the program have any unintended consequences?

To answer these and other kinds of questions, implementation evaluations use both quantitative methods (such as surveys and the abstraction of insurance claims) and qualitative methods (such as focus groups, personal observation, and interviews) to gather information about a program. When an impact evaluation is based on a quasi-experimental design, evidence from an implementation evaluation is useful for interpreting the impact results and deciphering whether the program or other forces accounted for program outcomes. Scene 1 culminates in an evaluation plan, which serves as a “blueprint” for organizing and performing the impact and implementation evaluations within a given budget and period of time.


Do not

copy

, pos

t, or d

istrib

ute


In Scene 2, detailed methods are developed to carry out the design (see Chapters 7, 8, and 9). For example, if the evaluation plan requires a telephone survey of sampled program participants, in Scene 2 the sampling protocols would be developed and the telephone instrument would be constructed, pretested, revised, and readied for imple-mentation in the field. Then, in Scene 3, the evaluation is conducted according to the plan. The evaluator collects and analyzes data in ways that produce specific answers to each evaluation question raised in Act I. On the basis of the evidence, the evaluator may formulate one or more recommendations for decision makers and other members of the play’s audience to consider in the final act of the play. Favorable results may lead to rec-ommendations for sustaining, improving, or expanding the program, whereas programs that consistently underperform may be recommended for closure (Eddy & Berry, 2009).

Act III: Using the Answers in Decision Making

In Act III, the findings are disseminated to the evaluation’s audience in a political con-text (see Chapter 10). A central assumption is that evaluations have worth only when decision makers use their results to improve program performance, to formulate new policy, or for other purposes (Patton, 2008; Rossi et al., 2004). Historically, however, this is often not the case (Rossi & Freeman, 1993; Weiss, 1972, 1998a). To encourage the use of evaluation findings, evaluators must translate the answers to the evaluation questions back into policy language for each policy question raised by decision makers and interest groups in Act I. Then, evaluators can develop formal dissemination and implementation plans to ensure that each interest group receives the information it wants about the program in a timely manner.

In the end, decision makers and other groups in the audience are more likely to use the answers if they played an active role in creating the policy and evaluation questions in Act I, reinforcing the circular nature of the play and the notion that upstream events can have downstream consequences for decision making (Shortell & Richardson, 1978). As the play comes to a close, the results of an evaluation are not the final determination of a pro-gram’s worth, which is ultimately a political decision based on the combination of facts and values. Findings can, however, provide public evidence about a program to reduce uncertainties and clarify the gains and losses that decisions might cause (Weiss, 1972).

In health program evaluations, the “play” has three important features. First, although the evaluation process is portrayed as a linear flow from Act I to Act III, the process is often circular between the acts. For example, the feasibility of an evaluation question in Act I may not be known fully until Act II, when the evaluation is designed in detail and the evaluator discovers that the question cannot be answered because of incomplete data or for other reasons. In this case, the evaluator must return to Act I and reformulate the question or replace it with a new question. Similarly, reactions from the audience in Act III may indicate that data analyses in Act II missed important aspects of the program, and the evaluator must circle back to Act II to remedy the omission.

Second, evaluation is a “practice” and not a “cookbook.” Just as physicians practice medicine in different ways, so do evaluators apply research methods in different ways to


Do not

copy

, pos

t, or d

istrib

ute

Prologue32

assess program performance. For any given program, many good designs may exist, but no perfect ones (Cronbach, 1982; Rossi & Freeman, 1993). Through daily practice, each evaluator develops his or her own “evaluation style” for working with other actors in the play, choosing among alternative designs, and traversing the three acts of the play.

Third, the three-act play is a general framework of the evaluation process, and different, customized versions of it can be found in public health. For instance, there are three core functions of public health practice: assessment, policy develop-ment, and assurance, where evaluation is a component of the assurance function (see Table 2.5; Institute of Medicine, 1988). The Centers for Disease Control and Preven-tion (1999) has developed a six-step framework for evaluating public health programs that contains the elements of the three-act play:

Step 1: Engage stakeholders

Step 2: Describe the program

Step 3: Focus the evaluation design

Step 4: Gather credible evidence

Step 5: Justify conclusions

Step 6: Ensure use and share lessons learned

Assessment

1. Health status monitoring and needs assessment

2. Investigation of adverse health effects and health hazards in the community

3. Analysis of determinants of identified health needs

Policy Development

4. Identification of community resources with which to build constituencies to advocate for health

5. Priority setting among health needs

6. Development of plans and policies to address priority health needs

Assurance

7. Management of resources and development of organizational structure

8. Program implementation

9. Program evaluation

10. Provision of health information and education

TABLE 2.5 ● Core Functions of Public Health

Source: The Future of Public Health, by Institute of Medicine, 1988, Washington, DC: National Academies Press.


Do not

copy

, pos

t, or d

istrib

ute


Similarly, Green and colleagues (L. W. Green & Kreuter, 1999; L. W. Green & Lewis, 1986) have developed a customized framework for evaluating health education and promotion programs that is based on the precede-proceed model, which contains the elements of the three-act play. Finally, Glasgow, Vogt, and Boles (1999) propose their own RE-AIM framework for evaluating the public health impact of health promotion interventions in a population of individuals. The framework contains both impact and implementation components of the three-act play, as well as an assess-ment of impacts on all members of the program’s target population, and the frame-work is being applied increasingly in health program evaluation (Gaglio et al., 2013).

Roles of the Evaluator

Just like all the other actors, the evaluator has a role to play that can greatly influence both the process and the outcomes of the evaluation. The role of the evaluator is not fixed and can vary from one health program to another. Although a variety of roles may exist in practice, four basic roles of the evaluator can be defined (Rossi et al., 2004).

In the three-act play described in the previous section, the evaluator has a par-ticipatory role, working closely with decision makers and other groups throughout the evaluation process (Christie & Alkin, 2003; Cousins, 2003; Cousins & Whitmore, 1998; Mertens & Wilson, 2012). When the evaluator chooses to be an active partici-pant, he or she also assumes responsibility for engaging and guiding decision makers and other groups through the evaluation process, where interest group members are involved directly in defining the program’s theory and the production of evaluation knowledge. The key advantage of the participatory role is that engaging decision makers in Acts I and II increases the likelihood that the evaluation will produce answers that decision makers want downstream in Act III.

The participatory role also has its disadvantages. The key danger is that evaluators may be coopted by management and conduct studies that are too narrow and avoid important issues (O’Sullivan et al., 1985). Management cooptation may not be inten-tional; it may arise when administrators fear what may be found from the evaluation. As a consequence, the problem itself may be poorly defined, or the questions narrowly cast in Act I, to avoid exposing management’s own poor performance in Act III.

O’Sullivan and colleagues (1985) illustrate this point using a case in which a North Carolina state agency ended its contracts to transport patients for renal dialysis. In response to the outcry from patient advocates, an evaluation was conducted to deter-mine whether the cutback in transportation services had harmful consequences. The findings showed that patients continued to receive treatment and that death rates did not increase. The evaluators concluded that patients were not harmed seriously, and the agency’s interest in the issue disappeared.

The evaluation, however, failed to examine the indirect impacts of the cutback (O’Sullivan et al., 1985). Because many patients had low incomes, they paid for trans-portation by cutting back on their food budgets and not buying needed medications. Because few patients drove themselves, family members had to take time off from


Do not

copy

, pos

t, or d

istrib

ute

Prologue34

work, reducing family earnings and increasing the emotional costs of being dependent on others. In short, the evaluation would have reached different conclusions if ques-tions about changes in economic and emotional status also had been asked in Act I.

Evaluators also may play an objective role (Mertens & Wilson, 2012). The tradi-tional role of the evaluator is to be an outside researcher who applies scientific research methods to produce an objective evaluation of the program. The evaluator values neutrality and consciously avoids becoming biased by the views of decision makers and other interest groups in the play. Some evaluators enjoy this role because of the autonomy, power, and authority that it gives them in the play. The chief advantage of the role is most visible when decision makers seek objective, unbiased information about a program, which they can use to address the competing demands from interest groups for decisions in their favor.

The detached role of the evaluator, however, also has its disadvantages. There is no consensus among evaluators and scientists about whether the application of research methods can ever be truly objective (Mertens & Wilson, 2012). Evaluators are subject to the influence of their own beliefs and values as well as the social context of pro-grams, which may influence what is and what is not observed about a program. Even though perfect objectivity is always an elusive goal, the objective evaluator can still strive for neutrality—that is, making unbiased observations about program perfor-mance that favor no individual or group.

By playing the role of outside researcher, the evaluator may become too “distant” from the program and fail to learn about what truly goes on in the program on a day-to-day basis. By emphasizing objectivity, the evaluator may rely too heavily on quantitative measures of program performance and miss qualitative information yielding different and valuable insights about the program. The role itself may create adversarial relation-ships with program staff, who may prefer working with evaluators who share their com-mitment to improving the program. Just as in the participatory role, if the objective researcher is responsive solely to questions raised by decision makers, he or she may be coopted and miss important aspects of program implementation and performance.

In stark contrast to the objective researcher, evaluators also can play an advocacy role (Greene, 1997; Mertens & Wilson, 2012). A basic premise of the role is that eval-uators cannot be value neutral; therefore, advocacy is an inevitable part of their role. For evaluators, the question is not whether to take sides but whose side to be on. In the advocacy role, the evaluator explicitly identifies his or her commitment to a set of values in the evaluation. Thus, if evaluators conclude that a health program improves health outcomes, which is consistent with their values, they are free to take a stance and persuade others to adopt it, provided they do not suppress evidence inconsistent with their stance. The advantage of the role is that it increases the likelihood that rec-ommendations from an evaluation will in fact be implemented. The disadvantage of the role is the possibility that evaluators will no longer be impartial in framing ques-tions, in collecting evidence, and in being fair judges of the worth of the program.

In health program evaluation, evaluators either implicitly or explicitly play an advo-cacy role, because most health professionals share a value commitment to protecting


Do not

copy

, pos

t, or d

istrib

ute


the health of the public, promoting access to health care, delivering cost-effective ser-vices, and so forth (Mertens & Wilson, 2012). As a common practice, therefore, advo-cacy evaluators should declare their value commitments and evaluate programs from a stance that promotes those values. By doing so, evaluators advocate for value positions and not for specific programs, which can avoid at least some of the disadvantages of advocacy evaluation.

Finally, evaluators play a coaching role for evaluations conducted by community members (Fetterman et al., 1996; Mertens & Wilson, 2012; Wandersman, 2009). In this type of evaluation, known as an empowerment evaluation, community members assume lead roles in planning, implementing, and evaluating their own programs. As a coach, the evaluator takes the role of offering help, advice, and guidance, and of promot-ing community ownership of the evaluation. The advantages of the role are found mainly in community empowerment and self-determination, which may increase the likelihood that the evaluation addresses community interests. Its major disadvantage is that com-munity members may lack the knowledge and experience to conduct the evaluation, even with help from the coach, which may lead to inaccurate or incomplete findings.

One example of empowerment evaluation is a public health program funded by the Centers for Disease Control and Prevention (CDC) to establish Urban Research Centers in Detroit, New York City, and Seattle in 1995–2003 (Israel et al., 2006; Metzler et al., 2003). The overall aim of the project was to improve the health and quality of life of inner-city, impoverished, and racially/ethnically diverse populations. At each site, partnerships were created among the community, public health agencies, health systems, and academia. A participatory research process was followed to identify problems affecting the health of inner-city communities and to design, implement, and evaluate solutions to them. Community members prioritized local health prob-lems, worked with the partners to identify what could be done about them, and were engaged in all phases of the evaluation process. The sites have addressed diverse social and public health issues such as asthma prevention, access and quality in health care, family violence, HIV prevention, immunizations, and racism.

Complementing the four basic types of evaluation roles, Skolits et al. (2009) iden-tify a greater number of evaluator roles based on the activities that evaluators perform in the evaluation process. Similar to defining the evaluation process as a three-act play, Skolits et al. divide the evaluation process into three phases (pre-evaluation, active evaluation, and post-evaluation), define the main activities in each phase, and artic-ulate the evaluator’s primary role in each phase, as shown in Table 2.6. An eleventh role, evaluator as manager/coordinator, occurs throughout the evaluation process, which is similar to our metaphor of the evaluator’s being the director of the play. By defining primary and secondary roles in greater detail, an evaluator may be able to prepare for and guide a future evaluation with greater precision, including what competencies should be present on the evaluation team and how evaluation activities should be allocated to team members.

Performing evaluation roles and activities has an emotional dimension. Skolits et al. (2009) recognize that role conflicts are inevitable in performing evaluation


Do not

copy

, pos

t, or d

istrib

ute

Prologue36

activities. Evaluators also may have conflicts with stakeholders, who may be unco-operative or resistant or who may become hostile toward the evaluation. An adverse emotional climate in an evaluation may have negative consequences for the evalua-tion, such as blocked access to data, intentional reporting of false information, and lack of cooperation. In short, excessive evaluation anxiety (XEA) may emerge and potentially undermine the evaluation process (Donaldson et al., 2002). Eval-uators also should have competencies in handling psychological and behavioral issues that may emerge, particularly in politically charged contexts. Donaldson et al. (2002) recommend strategies for recognizing and managing XEA in practice.

In summary, evaluators can choose different roles to play in the evaluation process. The roles are not mutually exclusive, and an evaluator can shift between roles across the three acts in Table 2.1. The evaluator’s choice of role typically is influenced by the fund-ing source of the evaluation. When decision makers hire an evaluator from an outside agency to examine a health program, the decision maker may want the evaluator to play the role of the objective researcher. Similarly, evaluators in academia doing large-scale evaluations of health programs or health systems funded by government or foundations also may want to play this role. In fact, the Robert Wood Johnson Foundation histor-ically has mandated that evaluations of its demonstration projects be performed by

Evaluation Phase and Activities Primary Evaluator Role

Pre-Evaluation Phase (Act I)

Preparation to conduct the evaluation

Initial contact

Evaluation planning

Evaluation contracting

Manger

Detective

Designer

Negotiator

Active Evaluation Phase (Act II)

Initial implementation/trust building

Evaluation data collection/analysis

Evaluation judgment

Evaluation reporting

Diplomat

Researcher

Judge

Reporter

Post-Evaluation Phase (Act III)

Promoting evaluation use

Evaluation reflection

Use advocate

Learner

TABLE 2.6 ● Three Phases of the Evaluation Process, Evaluation Activities, and Primary Evaluator Roles

Source: “Reconceptualizing Evaluator Roles,” by G. J. Skolits, J. A. Morrow, & E. M. Burr, 2009, American Journal of Evaluation, 30(3), pp. 275–295.


Do not

copy

, pos

t, or d

istrib

ute


objective professionals who are not responsible for project implementation. In contrast, large health agencies with sufficient resources may hire their own evaluators to assess their health programs, and the agency may shape the role of the evaluator and, there-fore, her or his relationships with other actors in the play.

Evaluation in a Cultural Context

In the evaluation process, evaluators frequently perform their myriad roles within a cultural context. This is particularly the case for interventions for which reducing health inequalities is a major goal (U.S. Department of Health and Human Services, 2014). In response, public and private organizations have launched a variety of pro-grams and health system reforms to reduce health disparities across diverse cultural groups (Lee & Estes, 1994), defined by race/ethnicity, country of origin, religion, socioeconomic status, geographic location, mental health, disability, age, gender, and sexual orientation or gender identity (Braveman, 2006; Lima & Schust, 1997). Just as health programs and health systems must be customized to health beliefs, values, and practices of each culture, so must evaluations also consider these factors in each act of the play.

Evaluation in a cultural context is becoming the norm rather than the exception in the United States because of the changing racial/ethnic composition in the U.S. population. The U.S. Census Bureau estimates that, by 2025, 40% of all Americans will be ethnic minorities, and by 2050, non-Whites will become the majority (Dunaway et al., 2012). Today, a majority of U.S. births are to non-Whites (Tavernise, 2012). These U.S. trends, plus calls for mandatory evaluation of health policies on a global scale (Oxman et al., 2010), reinforce that cultural competency is a core element of evaluation.

Cross and colleagues (1989) have developed a framework of cultural competency that offers guidance for evaluators. Although over 160 definitions of culture exist (Barrera et al., 2013; Guzman, 2003), culture refers to the “integrated pattern of human behavior that includes the language, thoughts, communications, actions, cus-toms, beliefs, values, and institutions of a racial, ethnic, religious, or social group” (Cross et al., 1989, p. 7). Cultural competency is a set of congruent behaviors, attitudes, and policies that come together in a system or organization that enables that organi-zation and individuals to work together effectively in cross-cultural situations (Cross et al., 1989). For evaluators, cultural competency means the application of cultural knowledge, behaviors, as well as interpersonal and evaluation methods that enhance the evaluator’s effectiveness in managing the three-act evaluation process. In other words, evaluators are culturally competent when they perform their roles well in cul-turally diverse settings and can complete their evaluation responsibilities successfully. Guzman (2003) notes the implications of cultural competency for evaluators:

An individual has one culture, or identifies with one culture (dominant U.S. culture), but can incorporate or understand the behaviors of another cultural group in relation


Do not

copy

, pos

t, or d

istrib

ute

Prologue38

to the cultural rules of that culture rather than the dominant culture. . . . Becoming culturally competent cannot take on a cookbook approach. . . . It involves individuals who are willing to be actively engaged with the evaluation participants and the evalu-ation process. (pp. 179–180)

Their cultural competency framework also stipulates that organizations and institutions be culturally competent in designing and delivering health inter-ventions and overseeing the evaluation. Cultural competency may be viewed as a continuum, with evaluators and their organizations striving continually to move up the continuum over time toward the highest levels of cultural competency, as shown below:

• Destructiveness (lowest cultural competency): fosters attitudes, policies, and practices that are destructive to cultures and, consequently, to the individuals within those cultures

• Incapacity: does not intentionally seek to be culturally destructive, but lacks the capacity to help minority clients or communities

• Blindness: provides services with the express philosophy of being unbiased; functions with the belief that color or culture make no difference and that all people are the same

• Pre-competence: realizes program weaknesses exist in serving minorities; attempts to improve some aspects of their services to a population—e.g., hiring minority staff

• Competence: accepts and respects differences; continually assesses and attempts to expand knowledge and resources; adapts service models to better meet the needs of minority populations

• Proficiency (highest cultural competency): holds culture in high esteem; con-ducts research to add to the knowledge base of practicing cultural competency; develops new therapeutic approaches based on culture; publishes and dissemi-nates results of demonstration projects

Recognizing the centrality of cultural competency in evaluation, the American Evaluation Association (2011) has developed essential principles of cultural compe-tence for professionals and organizations (see American Evaluation Association, 2011, for the complete statement):

• Acknowledge the complexity of cultural identity. Culturally competent evaluators recognize, respond to, and work to reconcile differences between and within cultures and subcultures.

• Recognize the dynamics of power. Evaluators recognize that social groups are ascribed differential status and power, are aware of marginalization, and use their power to promote equality and self-determination, and avoid reinforcing cultural stereotypes and prejudice in their work.


Do not

copy

, pos

t, or d

istrib

ute


• Recognize and eliminate bias in social relations. Culturally competent evaluators are thoughtful and deliberate in their use of language and other social relations in order to reduce bias when conducting evaluations.

• Employ culturally congruent epistemologies, theories, and methods. Culturally com-petent evaluators seek to understand how constructs are defined by cultures and are aware of the many ways epistemologies and theories can be utilized; how data can be collected, analyzed, and interpreted; and the diversity of con-texts in which findings can be disseminated.

The essential principles are related to what Tervalon and Murray-Garcia (1998) have called cultural humility, or the notion that evaluators (or anyone else) can-not be truly competent in another’s culture (Minkler, 2005). Humility requires a high degree of self-awareness and self-assessment, the lifelong process of becoming “reflec-tive practitioners” engaged in checking power imbalances and maintaining mutually respectful relationships with marginalized groups. Harris-Haywood et al. (2014) and Dunaway et al. (2012) have developed self-report instruments that evaluators may use to assess their levels of cultural competency and identify areas of improvement. Beach et al. (2005) report evidence that cultural competence training generally improves the knowledge, attitudes, and skills of health professionals.

The importance of culture in evaluation may be illustrated by attempts to reduce cig-arette smoking among Native Americans, who have the highest rate of smoking (about 32%) among most racial/ethnic groups in the United States (Office on Smoking and Health, 2014). Reducing cigarette smoking among Native Americans is difficult because of their historical regard of tobacco as a sacred gift and its central place in Indian cul-ture (National Public Radio, 1998). Programs to reduce smoking or achieve other health objectives in a cultural group demand evaluations that reflect and respect the group’s culture and build-in opportunities for collaboration by (Duran et al., 2008):

• Including members of the cultural group as full participants in all acts of the play • Ensuring that the values and traditions of the culture are taken into account in

the design of the program (e.g., through cultural adaptation of evidence-based interventions; Barrera et al., 2013) as well as the evaluation

• Considering cultural factors in the choice of measures and data collection protocols

• Having members of the cultural group interpret the findings of the evaluation and develop recommendations for decision makers

On the basis of the American Evaluation Association’s principles of cultural com-petency, evaluators have a responsibility to understand the role of culture in evalua-tion if they are to practice in an ethical manner (SenGupta et al., 2004). One strategy for practicing evaluation in a culturally competent manner is to follow the principles of community-based participatory research (CBPR), where evaluators collab-orate hand-in-hand with community members (Israel et al., 2008; Jones et al., 2008;


Do not

copy

, pos

t, or d

istrib

ute

Prologue40

Minkler & Wallerstein, 2008). A major advantage of CBPR is that community-generated and -owned approaches to improving community health and reducing health dispari-ties are more likely to have impact and be sustainable.

Ethical Issues

In conducting evaluations in political and cultural contexts, evaluators often face ethical issues (M. Morris, 2008). Consider the following scenarios:

• The evaluator proposes a randomized design to determine the impacts of the program, but the director asks her to choose a less rigorous, quasi-experimental design to satisfy the demands of interest groups that are worried that some needy people will be denied the program in the randomized design (M. Morris, 1998).

• Faced with the threat of severe budget cuts, the leaders of a health program want to launch an evaluation that will generate a “Big Number” to convince funders to support the program and secure future revenue.

• In an impact evaluation, the evaluator reports that a health program has no beneficial outcomes for program participants, and the director of the program wants to suppress the report because he believes it may damage the program’s future funding and harm staff morale.

• In an implementation evaluation, staff report they have little say in program decisions, and the evaluator concludes that the program is too centralized. The director of the program refuses to distribute the final report unless the conclu-sion is changed (Palumbo, 1987, p. 26).

• An evaluator conducts a pilot study of a health program to obtain preliminary results for a large grant application for a comprehensive evaluation of the health program. In the pilot study, the evaluator examines five outcomes, and the pro-gram has positive effects for two of the five outcomes. The evaluator excludes from the grant application the outcomes that were not statistically significant.

To help evaluators address these and other important dilemmas in their work, professional associations have developed guiding principles to promote the ethical conduct of evaluations by their members. AcademyHealth, the professional asso-ciation for health services researchers, and the American Evaluation Association have produced guidelines for the ethical conduct of evaluation research.

AcademyHealth (2014) has developed ethical guidelines for managing conflicts of interest in health services research. Conflicts of interest are inevitable in evalua-tion and arise through the multiple roles that an evaluator plays in an evaluation. The evaluator’s primary role is to conduct the evaluation with reliable and unbiased meth-ods. Based on the evaluator’s role, the evaluator’s primary interest is to generate and disseminate valid and reliable findings that inform policy and practice and to ensure integrity in the process. Because evaluators also play other roles, such as advocacy, they have secondary interests as well, such as promoting changes in health policy. Other


Do not

copy

, pos

t, or d

istrib

ute


secondary interests may include evaluation funding (received or pending), employer obligations, personal financial interests (such as consultancies, payment for lectures or travel, stock or share ownership, or patents), or nonfinancial support. Conflicts of interest occur when secondary interests distort the integrity of the evaluator’s judg-ments about the primary interests.

Conflicts of interest (competing interests) may be actual or perceived (Academy-Health, 2004; American Journal of Public Health, 2007). On the one hand, constitu-tional rights to freedom of speech protect advocacy roles for evaluators. On the other hand, outside observers may be uncertain about whether an evaluator’s advocacy has influenced negatively the integrity of the evaluation methods. Similar concerns also may arise when evaluators receive consulting fees from entities that have a stake in the results of the evaluation; when a company, foundation, or other sponsor that created a new health program or medical treatment is financing the evaluation of the treatment; or when other, similar arrangements exist. Perceived conflicts of interest arise when a reasonable person could perceive that a secondary interest might unduly affect the integrity of a judgment about the primary interest. Perceptions of conflicts of interest by reasonable observers are possible in process and impact evaluations because evaluators must often make numerous judgments about complex methods, which create opportunities for actual and perceived conflicts to arise.

AcademyHealth (2004) has developed 14 ethical guidelines for managing conflicts of interest in each act of the evaluation process (see Table 2.7). The American Evalua-tion Association (2004) also has advanced five similar Guiding Principles for Evaluators, as summarized below:

• Systematic inquiry: Evaluators should conduct systematic, data-based inquiries. • Competence: Evaluators should provide competent performance to stakeholders. • Integrity/honesty: Evaluators display honesty and integrity in their own behavior

and attempt to ensure the honesty and integrity of the entire evaluation process. • Respect for people: Evaluators respect the security, dignity, and self-worth of

respondents, program participants, clients, and other evaluation stakeholders. • Responsibilities for general and public welfare: Evaluators articulate and take into

account the diversity of general and public interests and values that may be related to the evaluation.

In summary, conflicts between the evaluator and the other actors can emerge in all three acts and contribute to the tension and drama of the play. Courage and character are personal qualities that can help evaluators uphold the ideals and values of their profes-sion, and the guiding principles provide them with an important resource for doing so.

Evaluation Standards

The evaluation process is a complex activity that may or may not be implemented suc-cessfully. All phases of the evaluation process in the three-act play must be addressed.


Do not

copy

, pos

t, or d

istrib

ute

Prologue42

Initiating Research • Researchers should pursue research maintaining openness to unanticipated results; do not

pursue research intended to reach a predetermined conclusion. • Researchers and sponsors should come to a mutual agreement on the research objectives and

data sources at the outset. • Researchers should make any sponsor-imposed limitations on the research explicit in the

contract prospectively, and not pursue research if the limitations are not acceptable to the researcher.

• Sponsors should assume responsibility to promote ethical research standards by setting reasonable objectives and deliverables as well as giving the researcher the freedom to analyze the data and report the conclusions as the researcher sees fit.

Conducting Research • There should be no fraud or fabrication in research, regardless of the purposes for which the

research will be used. • Researchers should maintain objectivity in conducting research and analysis; whenever possible,

use measures and methods that are widely accepted and reproducible. • Researchers should record the methods and measures of research and analysis, making this

information available to legitimate requests, such as confirming or replicating the reported results.

Reporting Research • Authors should be free to report data and results within 2 months (but no longer than 6 months)

after the research has concluded; any restrictions on this timing must be disclosed. • Any sponsor-imposed changes in the reporting of results that distort or misrepresent the

research findings are unethical and should not be accepted by the researcher. • Differences in opinion between the researcher and the sponsor about the interpretation of data

should be appropriately managed. • Researchers should endeavor to make their methods transparent and available to peers for

replication or confirmation of results. • All reporting of research results should specify who sponsored and financed the research, as

well as the role and involvement of the researcher at any point. • Work originally intended for advocacy or similar purposes (rather than as research) should be

identifiable as such. • Researchers must be explicit about their other roles and interests when reporting results and

must not conceal the possible limitations of the research.

TABLE 2.7 ● AcademyHealth Ethical Guidelines for Managing Conflicts of Interest in Health Services Research

Source: AcademyHealth. Ethical Guidelines for Managing Conflicts of Interest in Health Services Research. Washington, DC; 2004.

Stakeholder interests and cultural factors must be addressed in ways that comply with ethical guidelines. Evaluation methods must be applied rigorously to answer the eval-uation questions. Findings must be disseminated strategically in a political context to increase their use in policymaking and decision making.


Do not

copy

, pos

t, or d

istrib

ute


In response, the CDC Framework for Program Evaluation in Public Health (1999) has prescribed evaluation standards that evaluators can use to improve the quality of their evaluations. The standards are essentially guidelines to follow in carrying out the evaluation in the three acts of the evaluation process. Users of evaluations also may apply the standards to assess the quality of the work performed by evaluators. The CDC framework’s standards are adapted from the evaluation standards developed by the Joint Committee on Standards for Educational Evaluation (JCSEE) in the 1970s and updated periodically (Yarbrough et al., 2011).

Table 2.8 presents an overview of the evaluation standards from the CDC and the JCSEE. The CDC (1999) standards are grouped into four categories that contain a total of 30 specific standards. The JCSEE presents five categories containing 30 specific standards. To a great degree, the standards offer guidance for designing the evaluation process and assessing its success, but they say less about what methods should be applied to answer specific evaluation questions and how those methods should be carried out.

Centers for Disease Control and Prevention Joint Committee on Standards for Educational Evaluation

• Utility: ensure that the information needs of intended users are satisfied

• Feasibility: conduct evaluations that are viable and pragmatic (realistic, prudent, diplomatic, and frugal)

• Propriety: behave legally, ethically, and with regard for the welfare of those involved in the program and those affected by the program

• Accuracy: reveal and convey accurate information for determining the merits of the program

• Utility: increase the extent that stakeholders find evaluation processes and products valuable in meeting their needs

• Feasibility: increase the evaluation’s effectiveness and efficiency

• Propriety: support what is proper, fair, legal, right, and just in evaluations

• Accuracy: increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality

• Accountability: encourage adequate documentation of evaluations and a meta-evaluation perspective focused on improvement and accountability for evaluation processes and products

TABLE 2.8 ● Evaluation Standards From the Centers for Disease Control and Prevention and the Joint Committee on Standards for Educational Evaluation

Source: Adapted from “Framework for Program Evaluation in Public Health,” by Centers for Disease Control and Prevention, 1999, Morbidity and Mortality Weekly Report, 48(No. RR-11; September 17); and The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3d ed.), by D. B. Yarbrough, L. M. Shula, R. K. Hopson, & F. A. Caruthers, 2011, Thousand Oaks, CA: Sage.


Do not

copy

, pos

t, or d

istrib

ute

Prologue44

List of TermsAdvocacy role 34

Chain of causation 27

Coaching role 35

Community-based

participatory research

(CBPR) 39

Conflicts of interest 40

Core functions of public

health 32

Cultural competency 37

Cultural context 37

Cultural humility 39

Empowerment evaluation 35

Ethical conduct of

evaluations 40

Evaluation designs 30

Evaluation process 17

Excessive evaluation

anxiety 36

Feasibility assessment 26

Impact evaluation 30

Objective role 34

Outcome evaluation 30

Participatory role 33

Precede-proceed model 33

Process evaluation 30

Program implementation 30

Program theory 27

Rational-decision-making

model 17

RE-AIM framework 33

Roles of the evaluator 33

Stakeholder analysis 20

Stakeholders 20

Summative evaluation 30

Theory 28

Study Questions

1. What are the three basic steps of the evaluation process, or the three-act play? How does politics influence

the evaluation process? What are some of the overt and covert reasons for conducting evaluations? Which

actors in the play may desire an evaluation, and for what reasons? How would you find this out?

2. What are the two basic types of evaluation? How are they interrelated with each other?

3. An evaluator can adopt different roles in the play. What are these roles, and can an evaluator play more

than one role at the same time?

4. How would you conduct a stakeholder analysis for an evaluation being conducted in a cultural context?

5. What ethical dilemmas often confront evaluators? How can the guiding principles and evaluation standards

for evaluators help resolve these dilemmas in the field?

SummaryThe evaluation process consists of three basic steps.

The first step occurs in the political realm, where

evaluators work with decision makers and other

groups to define the questions that the evaluation

will answer about the program. In the second step,

the evaluation is conducted to produce information

that answers the questions. In the third step, the

evaluation findings are disseminated to decision

makers, interest groups, and other constituents in a

political context. The metaphor of a three-act play

is used to illustrate how political forces shape the

evaluation process and the role of the evaluator.

Ethical issues can arise in all acts of the play, and

to meet these challenges, the American Evaluation

Association and other professional groups have

developed guiding principles and evaluation

standards to promote the ethical conduct of

high-quality evaluations. Because health programs

often are created to reduce health and health care

disparities across cultural groups, evaluations also

must consider cultural factors in their design and

conduct. Act I of the play begins in the

next chapter, which describes how to develop

evaluation questions.


Do not

copy

, pos

t, or d

istrib

ute

Documents

The Evaluation Process as a Three-Act Play or post, copy, · be successful. The evaluation process itself generates the plot of the play, which varies from program to program and