Unit 10 Evaluation and Policy Change Check your devices and make sure you can see & hear Have your syllabus handy in the event you have any questions You

Unit 10Evaluation and Policy Change

• Check your devices and make sure you can see & hear

• Have your syllabus handy in the event you have any questions

• You can always contact me by email, schedule an appointment for office hours or schedule a telephone conference

Unit 9 Evaluation and Policy Change

1


2

Units 9 & 10Evaluation and Policy Change


3


4

Overview: Evaluation and Policy Change

• Introduction• Problems in evaluating public programs

– goal specification and goal change– measurement– targets– efficiency and effectiveness– values and evaluation– politics– increasing requirements for evaluation

• Policy change


5

Introduction

• The final stage of the policy process is to assess what has occurred as a result of policy selection and implementation and to make appropriate policy changes:– Evaluating policies is a difficult and highly political process. – Policy change is also difficult, as public-sector organizations

have many methods that they can use to resist policy change.

• We should not assume that government organizations are always resistant to change:– Most organizations resist change to some degree, but most

organizations want to correct their weaknesses.– Many of the obstacles to public-sector change are external to

public organizations—stemming from Congress and the public.


6

Problems in Evaluating Public Programs

• Evaluating public programs involves:– cataloging the goals of the program– measuring the extent to which goals are

achieved– suggesting changes that may bring

performance more in line with the goals of the program

– difficulty of producing unambiguous measurements of performance in a public organization


7

Goal Specification and Goal Change

• The legislation establishing a program should be a source of goal statements. Remember that:– Legislation is frequently written in vague language.– It is often difficult to attach readily quantifiable goals.– Goals specified in legislation may be impossible or

contradictory.– Internal political dynamics can become particularly

important in the absence of clear goals.– Contradictions in goals may also exist across

organizations in government (tobacco subsidies versus antismoking campaigns).


8


• Internal political dynamics may modify existing goals:– Roles can be expanded in positive ways:

• The Army Corps of Engineers transformed its image from one of environmental disregard to one of environmental sensitivity.

– Goal changes can also be negative, as with the capture of regulatory bodies by regulated industries.

– The “displacement of goals” by members of an organization is more common than capture:

• Individuals in the organization may become focused on personal survival and aggrandizement rather than the mission of the organization.

• The goals of the organization may become more focused on maintenance and survival than on its mission.

• Anthony Downs describes the “life cycle” of organizations, beginning as advocates of social causes but over time emphasizing on personal survival and the maintenance of budgets.

• In these instances, the operating goals of the program deteriorate.


9


• There are several types of goal change:– displacement: individuals exhibit transformation; reflexive transformation– empire building: organizations exhibit transformation; reflexive

transformation– street-level bureaucracy: individuals exhibit transformation; operational

transformation– adaptation: organizations exhibit transformation; operational

transformation• When transformations are exhibited individually, the focus of

evaluation should be correcting individual behavior, whereas organizational transformations should call evaluative attention to the policies of the organization.

• Efforts to minimize or avoid goal displacement include:– making managers more directly responsible for organizational

performance– instituting performance pay schemes at lower levels– making the public sector more consumer driven


10


• Goals may be impractical:– The Preamble to Constitution expresses many goals but does so

in vague language.– The Employment Act of 1946 pledges the government to

maintain “full employment,” but the government plays the “numbers game” to attempt to prove that the goals have been reached.

• Most public organizations service multiple constituencies:– The Comprehensive Employment and Training Act (CETA)

performed different functions for different groups in society.– The process of evaluation requires that analysts ask whose

goals are being achieved.• Goals may be either straitjackets or opportunities—the

organization is told both what to do and what not to do.


11

Measurement

• The efficiency and effectiveness of government is limited by the absence of a ready means to judge the value of what is being produced.

• Activity measures are often substituted for output measures in order to evaluate public sector performance; this serves the interests of existing organizations:– It shields organizations from stringent evaluations on

nonprocedural criteria.– Action becomes equated with success.– It may also have the less obvious effect of keeping clients

attached to a program even when they no longer need the program.


12

Factors that Inhibit Adequate Measurement

• Several factors inhibit adequate measurement of government performance. These are:– The time-span over which benefits of programs occur:

• Lester Salamon’s analysis of “sleeper” effects of the New Deal programs: initially viewed as failures but significant results were apparent thirty years after the programs were terminated.

• the “flip side” of the time problem:– the effects of a program should be durable– example: Head Start does not maintain high performance without

reinforcement in later years, reflecting the effects of program decay

• the time element may produce political difficulties– individuals have little time, and must produce results quickly (especially

elected officials)– policy process to focus on short-term successes– bureaucrats may observe a more long-term perspective


13


• Additional factors affecting the population:– It is difficult to isolate the effects of different programs.– It is difficult to hold constant all social and economic

factors other than the one being examined.• The history of the program and the individuals

involved:– Existing programs may jeopardize the success of a

new program.– Clients and administrators may become cynical about

the likely success of a program.– All policy areas seem to be confounded by almost

constant cycles of change and contradiction.


14


• An organizational basis of evaluation, which may limit the scope of the inquiry– Different organizations will measure different aspects

of a program.– A given agency may not evaluate the unintended

effects of a program.– For example, interstate highway system projects are

evaluated differently by engineers and mayors:• Engineers probably regard the highways to be a success in

terms of saved lives and fuel, whereas • Mayors of large cities may see the highways as a failure

because they facilitated “white flight” and reduced city tax bases.


15


• Reactive effects of experimental evaluation, which may influence results:– People may alter their behavior other under the

conditions of a policy experiment.– Individuals may work to produce results from

programs that favor their interests.– If an experimental method is not used, unmeasured

social and economic factors, not the program, may be the cause of any observed effects on the target population.

– Problems of research design reduce analysts’ ability to make definitive statements about the real worth of a policy.


16

Targets• Evaluators need to know not only the goal of the program but also

whom it is intended to affect:– Medicare—although the health of the elderly population has improved in

general, the health of the neediest portion of the elderly population has not commensurately improved

– Head Start—has not been used by the poorest families because it is a part-day program

• Participation in many programs is voluntary, and this may create problems for organizations seeking to serve particular target audiences.

• A program may create a false sense of success by “creaming” the segment of the population served by a program:– Programs may select clients who need little help in order to report more

success, and this creates a problem of generalizing effects for an entire population.

• Selecting target audiences is inherently political, as programs may seek to broaden audiences to build political support:– Broadened audiences make program evaluation more difficult.


17

Efficiency and Effectiveness• Evaluating efficiency requires assessing the ratio of the

cost of the efforts to the results:– In measuring efficiency, it is difficult to determine results and

assign costs.• Similar problems arise in measuring effectiveness

because surrogate measures of intended results are often required.

• Due to these problems, much of the assessment of performance in government depends on the evaluation of procedural efficiency:– Proceduralism may cause goals to be displaced, shifting the

focus of evaluation to the process itself rather than the services that the process is intended to produce.

– Concern with measuring efficiency through procedures may actually reduce the efficiency of the policy process because procedural steps and red tape exacerbate service delivery problems.


18

Values and Evaluation• The analyst who performs an evaluation requires a value system to

enable him or her to assign valuations to outcomes:– Value systems are by no means constant across the population or

across time.• There is difficulty in determining the proper valuation and weighting

of the outcomes of a program, as values vary across persons and over time:– This problem may be particularly relevant when a program has

significant unintended effects.• The analyst brings his or her own values to the evaluation process:

– Such values may be manifested through the policy process.– They may also have substantial impact on the final evaluation of

outcomes.• There are also other sources of values:

– organizations and professions, and – these values may affect the policy process

• Values may become the real battleground while rational argumentation and policy analysis are merely the ammunition.


19

Political Context

• Evaluations of public programs are performed in a political context.

• Political leaders may be interested only in those benefits created for their constituents and may disregard overall program effectiveness.

• A program may be evaluated only to validate decisions that have already been made—evaluators are often called upon to do “quick and dirty” evaluations.

• Institutional evaluators are therefore important because they are relatively impartial and stable in their evaluations.


20

Increasing Requirements for Evaluation

• There is a more recent focus on outputs (effects) rather than inputs (budget and staff) of the government.

• Congress passed the 1993 Government Performance and Results Act (GPRA), with the basic idea of appraising government organizations on the basis of their strategic plans and on quantitative measures:– The danger is that Congress will focus on only a few simple quantitative

measures and fail to understand the complexities involved.– The Gore Report was similar to GPRA in its emphasis on evaluation.– Continuing regulatory reviews have also required additional economic

evaluations, even if the evaluations have at times been a bit superficial.– Educational quality is increasingly subject to evaluation, although it is

still being measured primarily by standardized tests.– The George W. Bush’s administration’s focus on effort to punish poorly

performing schools are based on rather simplistic measures of progress; creating additional risks.


21

Policy Change

• After evaluation, the next stage in the process is policy change.

• Rarely are policies maintained in exactly the same form over time; instead, they are constantly evolving:– Change is often the result of:

• changes in the socioeconomic or political environment;• learning on the part of the personnel administering the program; or• simple elaboration of existing structures or ideas

• Policymaking in industrialized countries is often the result of policy change rather than the rise of new issues:– Most policy areas are already populated by a number of

programs and policies.• Policy succession—the replacement of one policy by

another—is therefore an important concept now.


22

Outcomes of Policy Evaluation

• Policy Maintenance: occurs only rarely as a matter of choice—it may result from a failure to make decisions:– This is an unlikely outcome of evaluation,

however, because politicians make “names” for themselves by advocating new legislation, not maintenance.

– Original policy decisions are generally flawed in some way, so it is likely that potential improvements will be identified.


23

Outcomes of Policy Evaluation

• Policy termination is also unlikely. – Programs acquire a life of their own—they develop

organizations, hire personnel, and develop a clientele.– Programs that build a “stock” of expected future benefits are

particularly difficult to terminate:• Social Security created a “stock” of future benefits for its clientele. A

reduction in benefits would create hardships for its clientele.• Welfare and food stamps do not involve planning by recipients, so

there is no “stock” element; makes it easier to reduce or terminate programs.

– The Reagan administration was unable to make sweeping program reductions. The George H.W. Bush administration wanted to eliminate the Department of Education but found that promise difficult to keep.

– While public programs are not immortal, relatively few are fully terminated.


24

Outcomes of Policy Evaluation• Policy succession is the most likely outcome of an existing program.• There are several forms of policy succession:

– linear: the direct replacement of one program or policy by another, or the simple change of location of an existing program:

• Example: the replacement of the Aid to Families with Dependent Children welfare program by the Personal Responsibility and Work Opportunity Reconciliation Act of 1996.

– consolidation: placing several programs that have existed independently into a single program:

• Example: rolling together a number of categorical health and welfare programs into a few block grants during the Reagan era.

– splitting: dividing programs into two or more individual components in a succession:

• Example: the 1974 split of the Atomic Energy Commission into the Nuclear Regulatory Agency and the Energy Research Development Agency.

– nonlinear: complex successions of policies and organizations, involving elements of other types of successions

• Example: the multiple changes involved in the creation of the Department of Energy.


25

Policy Succession• Policy Succession: The process is somewhat distinctive even

though it may entail processes like the following:• Agenda setting; not as difficult as in policy initiation stage:

– The broad issue has already been accepted as an agenda item and is now returned to a particular institutional agenda.

• Legitimation and formulation:– Client and producer interests are threatened by many proposed policy

changes, especially in cases of likely policy consolidation.– Existing clients are likely to fight to maintain the status quo in many

cases.• Some policy successions are internally generated:

– External political forces may help effectuate, and even help sponsor, internally generated policy change.

– Some program managers are risk takers and are willing to bet on the benefits of policy change.

– Some organizations will gladly shrink the size of overextended programs such that organizational “heartlands” are preserved.


26

Policy Succession• Clientele groups may seek to split a program from a larger

organization in order to develop a clearer target for their political activities.

• Coalitions for policy change:– Forming a coalition requires attention to the commitments of individual

members of Congress to particular interests and ongoing programs.– It requires use of mechanisms of partisan analysis, logrolling, and pork-

barrel legislation.– Building broadly interested coalitions for policy change may threaten the

existence of the policy, as termination may be the only alternative on which the coalition can agree.

• Implementing a policy succession may be the most difficult portion of policy process. This may be true for several reasons:– Field staffs may not respond to nominal changes in policy; this may

result from inertia or a lack of understanding.– Modifications in one policy may adversely affect other policies and

organizations.– Changes may occur that go beyond the intended policy modifications.


27

Policy Succession• “New governance” via third-party and indirect mechanisms means

that connecting programs with outputs is more difficult; this may further inhibit policy succession.

• Policy succession takes massive political efforts, yet cannot show results for some time—it may therefore create disappointment in the new program.

• Policy successions may generate multiple effects on entrenched interests, resulting in more change than was intended.

• Because policy succession is difficult, it is appropriate to think about how to facilitate ongoing policy change when designing organizations and programs.

• Policies are increasingly being designed with built-in “terminators”:– Sunset laws and the declining sense of entitlement may encourage

continual change.– It is not possible to reverse history and add terminators to existing

programs that are already functioning without them.


28

Course Conclusion

• Unit 10 completes the course

• Follow-up in Discussion

• Make sure your Final Project is submitted

• I hope you enjoyed the course and will find it useful in future studies and work

• Good Luck and Good Night


29


30

Documents

Unit 10 Evaluation and Policy Change Check your devices and make sure you can see & hear Have your syllabus handy in the event you have any questions You