The case study evaluation: A means for managing organizational and political tensions

Evaluation and Program Planning, Vol. I, pp. 253-266, 1984

Printed in the USA. All rights reserved.

0149-7189184 $3.00 + .OO

Copyright Q 1985 Pergamon Press Ltd

THE CASE STUDY EVALUATION

A Means for Managing Organizational and Political Tensions

CATHERINE MARSHALL

University of Pennsylvania

ABSTRACT

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policymaking for California’s Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.

The Early Childhood Education program in California (ECE) was a state initiated plan for school reform. Earlier evaluations of ECE had become political tools in the struggles between the legislature and the state department of education. Legislators had demanded to see evidence of effectiveness while ECE was being implemented in local schools. The political stakes of ECE evaluations were high; previous evaluations focused on achievement score outcomes rather than on processes that might improve the program.

This article describes an evaluation that, because of its design and management, was used for policy and program decisions, even though it was conducted in a politically tense environment. The Interim Report, a

case study evaluation of ECE, was different from other evaluations of ECE. Previous evaluations of ECE had been attacked by politicians as nearly useless for policy deliberation or program improvement. The Interim Report was indeed used by legislators, by the State Department of Education (SEA) and local school districts (LEAS) in their policymaking.

This paper analyzes why this particular form of evaluation succeeded in influencing policy decisions at multiple levels when other evaluations had failed. Because ECE was an exceedingly complex, state-mandated but locally controlled reform strategy, there were ensuing questions for evaluation design.

DESIGN AND UTILIZATION QUESTIONS

When evaluations are to be used in policymaking, the 2. What research methodologies are preferred, how design and management must address the following important is the timing and method of reporting? questions: 3. How can researchers stay attuned to the shifting in- 1. What factors determine whether research and eval- formation needs of policymakers?

uations are used for policymaking? 4. Who should determine the relevant questions, ap-

The analysis was part of a larger research project that focused on the politics of evaluation (see Marshall, 1981). The author was an indepen-

dent researcher, conducting research on evaluation and policymaking and focusing on the Early Childhood Education Program (ECE) evaluation utilization. Her research was supported by the Bush Foundation Training Program in Child Development and Public Policy at the Univer- sity of California, Los Angeles. She was never a participant in any of the evaluations of ECE.

Requests for reprints should be sent to Catherine Marshall, PhD, Assistant Professor of Education, University of Pennsylvania, Graduate

School of Education Cl, 3700 Walnut Street, Philadelphia, PA 19104.

253

254 CATHERINE MARSHALL

5.

6.

7.

propriate methodologies, and the policy implications: the researchers or the policymakers? Must researchers educate policymakers about methodology, relevance, and limitations of research? How can case study evaluation be useful for policy and how can quality control and policy relevance be addressed in the design? How can evaluations be reported when findings may damage agencies and careers?

Experienced evaluators have faced these questions. For example, the California Education Management and Evaluation Commission noted that evaluations fall short if evaluators promise too much. “Evaluators . . . often (a) fail to deliver the right questions, (b) promise more answers than they can deliver, (c) produce reports late in a decision cycle, and (d) measure program outcomes too narrowly” (1978, p. 2). In addition, policymakers sometimes make unreasonable demands for go/no go decisions. The commission noted: “Policymakers prefer hard information that can reduce uncertainty, couched in unequivocal language” (1978, p. 4). Patton (1978) stated that a utilization crisis exists because of the time and money wasted on non-utilized evaluations. His research indicated that evaluations are most useful when “the guy who was asking the question was the guy who was going to make use of the answer” (p. 67). He recommended structuring high communications, involvement of salient persons from the beginning, getting the best agency’s commitment, ensuring that the evaluation is aimed at a felt need, knowing how the evaluation will be used, and taking a decision-oriented approach.

The National Institute of Education (NIE) evaluation of Title I structured the participation of policymakers in the selection of questions to be investigated. Paul Hill, the study director, described himself as:

Serving a select clientele, specifically the majority and minority staffs of the House and Senate Education Com- mittees. Thus, the primary planning negotiations were between the NIE and Congressional staffs in order to determine what information both the majority and minority staffs could agree would be important in the pro- spective debate. (Hill, 1979)

Alkin, Daillak, and White (1979) found that evaluation utilization is affected by (a) pre-existing evaluation bounds, (b) orientation of the users, (c) evaluator’s approach, (d) evaluator credibility, (e) organizational factors, (f) extra-organizational factors, (g) information content and reporting, and (h) administrator style.

After studying policymakers’ perceptions of research utilization, Weiss (1979) concluded that:

If they are truly committed to affecting organizational decisionmaking through research, social scientists may

have to become involved in the rough and tumble of organizational decisionmaking. It is not enough to do research; they may have to become advocates for the research and its conclusions. They will have to make a strong case for preferable policy alternatives, and become engaged in the confrontations and controversies that advocacy entails. (p. 18)

Realities of politics and agency needs complicate evaluations. Controversy and confrontation ac- celerate when research and evaluation are used as political instruments. Where intergovernmental rela- tions are affected, where goals include social action and reform, where money and careers are at stake, evaluation may be utilized more for political reasons than for informing decisionmaking. Cohen (1970) illustrated this phenomenon with Follow Through, McLaughlin (1975) with Title I, and Zigler and Valen- tine (1979) with Head Start.

Researchers and evaluators have puzzled over ways to enhance evaluation utilization, noting the following:

1.

2.

3.

evaluators and legislators are two communities who have different languages, timing needs, and dissemination modes (Dickey, 1980); information needs are different and fluctuating depending upon the timing in decisionmaking; evaluators must stay attuned to shifting policy concerns (Mitchell, 1979; Hill, 1979; Patton, 1978); bureaucrats and policymakers seek to influence evaluations; a state department of education will naturally have a self-interest and a role in education evaluation management (Smith, 1982; Cohen, 1970; Sroufe, 1977).

Thus, evaluation management must be attuned to political and organizational tensions.

Qualitative evaluation has some particular utility, as Parlett and Hamilton (1976) and LeCompte and Goetz (1982) have delineated. Qualitative evaluation of social interventions in complex systems like schools may allow for needed design flexibility, for more description of processes, and a variety of measures. An ex- ploratory approach allows for discovery of what is working and where problems exist in implementation.

These insights from previous evaluators and researchers show that evaluation management in a political setting must face the following fundamental dilemmas:

1.

2. 3. 4.

What is the precise policy or management question? Who are the audiences? What purpose will the evaluation serve? What research design is appropriate for the question?

Case Study Evaluation 255

5.

6.

7. 8.

9.

What are the biases, intrigues of politics, and constraints that affect the potential audience? Are there adequate and appropriate human and monetary resources available for the study? How should the findings be reported? How much should evaluators reach into policymaking- should they take forceful stands to ensure that their findings be incorporated in policymaking? What constitutes utilization in evaluation?

Such a list might outrage evaluation purists- the idea of letting themselves be affected by political intrigues would be horrible. Nonetheless, the story of this “successful” evaluation suggests that evaluation managers must face these questions.

This paper uses insights and questions about evaluation utilization to guide the analysis of a case study evaluation of an innovative, controversial state education program. The analysis identifies strategies for aligning the evaluation questions with policymakers’

needs; for negotiating a compromise research design that satisfied the needs of conflicting groups; for addressing the concerns of those who are unused to qualitative research; and for reporting in a way that reduces the political impact of negative findings, allows the sponsoring agency to avoid crises, and ap- plies directly to policy debates. The success of this evaluation shows the importance of maintaining organizational/political support. At the same time, it shows a mode for managing a case study evaluation in the midst of organizational and political tensions.

The next section provides the context of The Interim Report evaluation, including the educational and political controversies and the history of previous evaluations of ECE. Next, the evaluation controversies are described showing the ways they were managed in The Interim Report evaluation. The final section analyzes The Interim Report’s management to draw lessons for future evaluations of politically controversial programs.

THE INTERIM REPORT IN THE CONTEXT OF ECE POLITICS AND EVALUATION

What was ECE? The answer depended on who you asked and when. California State Superintendent Wilson Riles had made election campaign promises about reforming and revitalizing education, raising reading and math achievement, and focusing on very young children. ECE policy incorporated these goals along with a “blue ribbon commission” of experts and concerned citizens’ recommendations for addressing socioemotional needs of children, individualizing instruction, and developing children’s self-concepts. ECE also involved restructuring the educational system, making decisionmaking attuned to local needs, and including school site decisionmaking and school advisory committees. Legislators’ concerns about controlling the purse strings, raising achievement scores in low achieving schools, and maintaining legislative control over the State Education Agency (SEA) would be met through yearly evaluations. These mixed, conflicting, ambiguous, and changing goals and the information needs for political decisionmaking created a number of evaluation dilemmas, however. The who, how, and what of the evaluation procedure was not established.

A quote serves to illustrate the intersect of politics and evaluation, as an SEA evaluator reflected on the problems of evaluating ECE and dealing with legislative demands:

The conservative political philosophy is one that emphasizes outcome, bank for the buck, cost-effectiveness, are we getting what we are paying for. Every bill that comes up has some proviso for evaluation. Implicit in ECE was the goal of bringing kids up to par. We tried, unsuccessfully, to buy off any scrutiny of test scores per se as indicative of success of the program. We got two

years to prove its worth. There’s no such thing as a longitudinal study of a program in government. Every- body is intent on pulling it up and looking at the roots to see if it’s growing. The phenomenon of ECE and test scores has absolutely nothing to do with evaluation. It was an arguable focus by adversaries of the program which was used to elicit legislative argument and not to go/no go with the program. That was never the issue. The issue was to find some lever that could be used.

This section describes ECE evaluation history to illustrate the dilemmas.

Initially, evaluation was managed by standard operating procedure: The SEA evaluated ECE by com- piling aggregate measures of outcomes, and where data were available, printed a report-&r& Child- hood Education: First Annual Evaluation Report, 1974-75 (1975). The report, however, was compiled in a hurry and with data from schools that had just begun to implement ECE. Superintendent Riles, in his in- troduction to the report, proclaimed that ECE “re- sulted in significant improvement in school achievement” (p. iv).

When Riles requested funds for ECE expansion in 1975, legislators, the Department of Finance, the media, and academicians zeroed in on the evaluation report. Critics of the evaluation noted that only 43% of the ECE students were represented in the data, that there was no effort to link achievement gains to ECE processes, that there was no description of how ECE worked, and that it was inappropriate for SEA to evaluate its own program. Legislators who were being asked to expand the funding for ECE had inadequate information upon which to act. ECE had its own con- stituency, however. One senator who attacked ECE


was deluged by phone calls from parents and school people who supported ECE. Legislators began to see that Riles was a charismatic leader with his own con- stituency and was emotionally and politically committed to seeing ECE expand. They approved ECE expansion, but mandated an independent evaluation of ECE.

But, early in 1977, Superintendent Riles and SEA began to show a new openness-a willingness to face implementation problems, seek solutions, and form new policy. The department’s response to the following finding of the Evaluation of ECE, ESEA Title I and EDY, 1975-76, was an indication of this new stance. The evaluation findings said:

To answer the information demands of legislators and other groups, the independent evaluation had to include control groups, norm-referenced tests, and information on SEA’s management of parent involvement, staff development, and instruction variables. These varied information demands, the limited funding, and delays in transporting achievement test scores complicated the evaluation. Then the pressure from the SEA to send an executive summary before the evaluation was completed led to a loss of credibility when the premature executive summary and the final evaluation report itself did not coincide. Further, legislators still had no description of how ECE worked. When the evaluation director reported to legislators, she reiterated her cautions about drawing inferences, repeating the report’s statement that “our study cannot provide unequivocal guidance for legislative or school decisionmaking because evaluation and methodological concerns were subordinated by program implementation decisions” (Baker, 1976, p. 9). The evaluation left legislators angry that evaluations could not, or would not, make clear, definitive policy recommendations for them. (Patton (1978) had emphasized the importance of focused policy recommendations.)

In schools whose entering students averaged between the 21st and 99th percentile on the 1973-74 Entry Level Test, grade three reading achievement improved markedly beyond predicted levels after three years in ECE. . . . In schools whose entering students averaged below the 20th percentile on the ELT, grade three reading achievement declined relative to prediction after three years of ECE. (SEA, 1977, p. 36)

The indication of problems with ECE implementation and outcomes was clear. The SEA’s response was:

When we noted the decline, we generated possible reasons and sent it to [an SEA official] who said, “here’s a problem; we should be honest about it and then go out and find the problem source.” There was no attempt to sup- press the finding. (An SEA evaluator)’

By 1977, ECE evaluations had been attacked and politically manipulated. To that date, SEA’s reactions to criticism had included belligerent attacks on the credibility and expertise of the critics (Marshall, 1981). No evaluation had successfully described ECE or provided findings relevant to the information needs of policymakers, program managers, or participants in ECE. Political controversies had undermined attempts to use evaluation to improve ECE decisionmaking.

This quote indicates a problem-solving, rather than a defensive politicking mode. SEA realized that it must look at the relationship between its processes and the change processes in schools. It saw that regression analyses on selected variables and achievement scores were not helping program managers (state and local) or legislators to improve ECE or to make ECE politically stable.

After 4 years of ECE, information needs were high. There had been no evaluation that had been accepted as credible by all of the evaluation audiences, no description of processes and insufficient information for policymaking. The various evaluation audiences had raised questions and complaints - some expressed outrage about low scores in ECE schools; the stage was set for a new type of evaluation.

A NEW APPROACH TO EVALUATION

When the SEA reported the negative findings to the legislature, it made a commitment to explore the reasons for the decline in achievement test scores. The Senate Finance Committee mandated a study, entrust- ing to the SEA the task of selecting the appropriate evaluation unit, personnel, and design. It demanded, however, the answer to one question: “What are the characteristics of the ECE schools with low entry-level tests and declining achievement scores and what processes are causing the decline?” No demands were added, no assumptions as to cause were built into the question, and no evaluation measures or methodology were specified. The opportunity, indeed the demand for a new approach to evaluation was set. Flexibility in design was possible and the audience for ECE evalua-

tions was anxious for explanations of the negative findings.

Previous ECE evaluations had been attempts to

‘Quotes are identified by role only in order to guard anonymity. They are derived from an extensive study of ECE evaluation (Mar-

shall, 1981). Many quotes in this paper are from members of The In-

terim Reporf team, but the larger case study providing the context of

ECE evaluation had originally identified the degree and variety of utilization of this evaluation. This Znferim Report evaluation stood out as an exception to an otherwise dismal history of ECE evalua-

tion being caught up in political conflict. The larger study of ECE

decisionmaking was conducted in 1979-80 and included interviewing of more than 40 people in various state, county, local, evaluation, and academic positions, focusing on their involvement in ECE, par-

ticipant observation, and analysis of policy documents.


answer the wrong questions and a mismatch between evaluators’ and audience agendas. The new evaluation design had to address the fundamental dilemmas if policymakers were to be receptive to their evaluation (see pp. 254-255). The Interim Report evaluators found ways to manage these basic dilemmas. Their answers and the stances they chose enabled them to produce a useful evaluation. This section describes how they addressed each of the dilemmas. Later the description is analyzed to cull out evaluation basics in the policy arena.

Identifying a Precise Policy and Management Question For The Interim Report evaluators, a focused question to frame the evaluation emerged directly from the policy deliberations, as shown previously. The low achievement, declining-score schools had been identified, but the processes and problems had not been described in previous evaluations. Unlike previous studies, this evaluation was not encumbered by a laun- dry list mix of questions. The mandate for the evaluation was focused on one policy question and there was no stipulation of a research design for answering the question.

Identifying the Evaluation’s Purpose, Audiences, and the Biases, Politics and Constraints and Needs of Audiences The audience included the SEA as ECE program manager and as political unit -politicos. It included interest groups, legislators, county and local district program managers, ECE participants, and the media. (See Figure 1.) Each of these audiences had its own agenda, which an ECE evaluation could potentially serve. Their biases, intrigues, and constraints are illustrated in the following descriptions of the audiences.

The Locals against State Control. Local educators were against a state-imposed system and SEA monitoring and evaluations. They used symbolic action, ap- peal to legislators, and their interaction with SEA politicos and program managers to prevent SEA control. United Teachers of Los Angeles hired a truck to carry, to the legislature, the reams of forms which they saw as a symbol of over-control, overtesting, and burdens from programs like ECE. A senator, repre- senting some local people, had criticized the SEA teams and told Superintendent Riles to keep SEA people out of his district. This audience wanted to ferret out the mismanagement and political control they saw in ECE. Earlier ECE evaluation history had only dem- onstrated mismanagement and political tensions. This group wanted evaluations to provide insight on processes or information about how to eliminate the management and political problems in ECE implementation.

Legislators. Legislators’ experience with previous evaluations made them suspicious and frustrated. There was suspicion because of previous program pro- motions exaggerating favorable findings and attacks on ECE critics. Any SEA evaluation had to confront and rise above this suspicion. In addition, legislators were frustrated by long reports which detailed meas- urement procedures. They wanted information that could systematically describe and substantiate the suc- cesses and problems with ECE and make policy recommendations. This would enable them to alter educational policy, answer constituents’ complaints, and alter resource allocations to maximize effectiveness. Also, the legislative audience wanted evaluations to help them keep their control of educational policy. Hill (1979), Rosenthal (1978), and Mitchell (1979) delineate legislative concerns for timely evaluation attuned to shifting policy concerns. Patton (1978)

Legislators

SEA Politicos

SEA Office of Pro- gram Evaluation

and Research (OPER)

Figure 1. The ECE Evaluation Audience.

Interest Groups

ECE Participants

L ,

Evaluators and Academics

1 Media 1


shows the importance of focused policy recommendations.

The SEA, The audience of people who needed information to make ECE function smoothly consisted of SEA politicos, program managers, evaluators, LEA participants, and SEA field assistance people. There were tensions between SEA program managers and the SEA Office of Program Evaluation and Research (OPER). One program manager explained:

Evaluators are defensive about their purity. But evaluators d&i understand the program so they can’t measure it or describe it appropriately. OPER’s numbers [statistics] don’t do it. They were always off on their description. Their numbers were fine but their conception was off-it was not in the flavor of ECE. (An SEA program manager)

Program managers relied more on the information network with locals and participants. More often than not, program managers had changed policies long before an evaluation report came up with a finding that said that policy should be changed. Their information network did not run smoothly, however. The dissemination of information on policy changes led to awkward situations, as a county ECE director described:

Our role at County is to continue being supportive of SEA when they’ve blown it. It is difficult, like when they send letters promising documents and then don’t come through, and when state people gossip so they leak the fact that a requirement wit1 be phased out before it is official and I’m there inservicing people on how to fulfill that requirement. (A county ECE director)

When SEA field assistance teams went on the road they faced school people who needed interpretations of unclear SEA guidelines. They faced unbelieving school administrators who recalled having followed an SEA policy only to find that the policy interpretation from one SEA person differed from that of another SEA person. Policy decisions made by SEA’s politicos did not reach the field assistance people immediately. In addition, the field assistance personnel- the SEA representatives who were most visible in the field - were looked upon as the least competent SEA employees.

The program manager audience needed evaluations that would see ECE as a whole program and look at implementation issues. Their present mode of keeping close contact with school people had helped manage many problems in implementation, but they needed more systematic and more timely program evaluation. They, and SEA politicos, needed to know which ECE processes were being implemented and what processes were effective. Previous evaluations had not served their needs: “From an evaluation standpoint, there was nothing in the way we collected data or in the evalua-

tion questions that would give us a clue about the problems” (An SEA politico). Previous evaluations had provided outcome data with no real clues about processes. ECE evaluations had failed to answer the LEA implementors’ needs too. They had emphasized state-level concerns for management and account- ability, but as Alkin et al. (1979) have shown, this sort of emphasis is not useful to LEAS. Thus local ECE managers gained no understanding of what was and was not working in their programs.

Tensions between the previous SEA’s OPER and SEA politicos had complicated the SEA’s past evaluation utilization attempts. OPER took pride in doing objective, scientific studies. The practice of sending all evaluation reports to SEA politicos for editing caused consternation. An OPER member lamented: “If [a top politico] says no, the evaluation won’t go. She’ll say it doesn’t have the right tone; it doesn’t tell how the program works; it’s not positive enough” (An SEA evaluator). Just prior to the initiation of The interim Report evaluation, however, a new, politically astute manager took a high position in OPER. He was not tied to tra- ditional evaluation methodology and he had links with program managers and politicos.

[The new man] is slick, cool, really going somewhere. . . . He was part of the Executive Cabinet and that says you’re somebody. (A IegisIative analyst)

[The new man] was trusted by both sides, program and OPER. [His predecessor] was more the professional evaluator; he was not policy-minded, didn’t play games . . . but [the new man] sees the bigger picture of what evaluations have to do. (An SEA evaluator)

He would serve as a link between factions of SEA. OPER took on new leadership perspective that would be invaluable in the development of The Interim Report. An OPER staff member reflected the change in perspective:

I used to be rigid, a pure evaluator, upset about program managers altering our descriptions. Now I’ve seen what can happen to a good program. Evaluation should be to identify problem areas, to give honest answers. They can’t when they’re used to destroy a good program. (An SEA evaluator)

Such attitudes allow a new sort of evaluation design and reporting.

The SEA, as both audience and evaluator, had conflicting agency needs, information needs, and political needs. Managers needed information focusing on problems in ECE processes. Politicos needed that too, so that ECE problems could be solved, but politicos also needed to reaiign the SEA Program and OPER to cooperate and to defend ECE. SEA politicos, program managers, and evaluators could benefit from a new


type of evaluation that emphasized processes, focused on one question, and used a case-study approach.

Media. A final audience for ECE evaluation was the press. In the past, the reporting by the Los Angeles Times served to mobilize critiques of SEA evaluations. The superintendent had used television spots to pub- licly interpret evaluation findings and promote ECE. The press would benefit from evaluation findings that touched on areas of political conflict.

This description of the ECE evaluation audiences illustrates their needs, biases, and the constraints and possibilities inherent in ECE evaluations. Any 1977 ECE evaluation could be attacked, supported, used, or abused by these audiences depending on the evaluation management. The Interim Report evaluation could serve legislators if it was attuned to their biases and frustrations. It could serve program managers if it got above the details of outcomes and looked at the whole process of ECE implementation. The evaluation could serve the public, through their legislators and through the media, by raising questions to improve schools.

Identifying the Appropriate Research Design and the Essential Human and Monetary Resources The Senate Finance Committee had requested the evaluation, but had not appropriated funds. Existing SEA personnel and resources had to be used, there- fore. The usual pattern would be to give the task to OPER evaluators; SEA program people would continue their regular activities, then later read the findings and attack them as irrelevant to the real program. The Interim Report deviated from this pattern. There were special arrangements developed for management, personnel, design, and reporting of this evaluation.

First, the evaluation question was focused on processes and the mandate for the evaluation allowed design flexibility. Thus, it would be important to gather qualitative data on processes of implementation. Second, the new leadership in OPER was open to cross-unit innovation. They maintained connections with program managers and politicos and built bridges with the legislature. Thus, The Interim Report team consisted not only of OPER evaluators but also of several program staff people. One expressed his orientation to evaluation as “not fixed on measuring objec- tives” (an Interim Report team member), and one was a valued advisor to a key politico. He “had her ear” (an SEA evaluator). This politico reportedly “never reads anything but she relies heavily on what a few trusted individuals tell her” (an SEA evaluator). Thus, several people served as strong links between the evaluation team and the politicos and program managers.

Program management was also shifting at this time and ripe for innovation. The program director’s management style, appropriate for initially operationalizing an amorphous reform program and defending ECE,

was no longer seen as appropriate. Now the SEA policy stance allowed controlled criticism and increased ECE program flexibility. As more and more people were brought into the discussion of ECE evaluation, the director’s control began to dissolve.

A new program staff person brought expertise and advocacy for case study design for the evaluation. She also emphasized the importance of building team commitment to the project by team involvement in the planning. She was naive with regard to SEA organizational patterns and was not socialized to keep her ideas quiet or allow politicos to moderate them. She recalled:

I had a lot of money then and I didn’t want to just do a schlocky job. I’m not sure that now I could do it, having been socialized. I wouldn’t have yelled as loud or fought as many battles. (An Interim Report team member)

The new people in program and in OPER coalesced as The Znterim Report team and began the task of arguing an evaluation design into existence.

The personnel issue was resolved, but leadership and management issues loomed. Fundamental issues were decided at a meeting in the SEA board room with evaluation team members, program managers, and politicos. The setting and the actors indicated high- level decisionmaking. The agenda issues on the surface were the following: (a) Will the evaluation be strictly research (as the team wanted) or will it be combined with field assistance (as the ECE program director wanted)?; (b) Will the program managers or OPER manage the evaluation? But underlying issues were: Will the program director be undermined? How much high-level support exists for this evaluation? If it has negative findings which make SEA look bad, who will defend the study? If the personnel costs of collabora- tion rise, will the study be scuttled? The team asserted that if they mixed assistance with evaluation it would divert them from intense in-depth data collection, that the stance of evaluator and the stance of skills of SEA technical assistance person were different. They also wanted to ensure that the evaluation was designed and conducted without the interference of intraorganizational politics. One new OPER member recalled the meeting that helped answer these questions:

[A key pohtico] set it up ahead of time. She phoned the superintendent and told [the program director] and me to talk to him (the superintendent) to resolve the clash over the study. It was 40% staged. (An Interim Report team member)

The meeting ended with the program director’s ideas defeated; with the new leadership in OPER managing, promoting, and supporting the team’s case study design; but with the understanding that this unique team would essentially manage itself. Thus, there was


a commitment of strong support from top politicos, including Superintendent Riles.

A meeting between a key politico, several team members, and the OPER staffer was called:

We [met] about using achievement data. We talked about the utility of summative evaluation- how it doesn’t help Program people or schools, it’s demoralizing. It does no good to evaluate bad schools a third time. Instead we need to help them learn something about the nature of ECE. The theme was “do something different.” (An Interim Report team member)

Politicos and team members became gradually committed to a case study design. As can be seen, this design decision was a decision derived in the midst of a power struggle among units, personalities, and philos- ophies within the SEA for the evaluation depended as much upon resolution of the power struggle as it depended upon agreement on the design.

Sampling questions still needed resolution. Despite the commitment of politicos to the new evaluation format, agendas did clash and the team had to delineate the appropriate case study design and establish and defend its agenda. Politicos wanted to include a larger sample. Program people still wanted to provide technical assistance mixed with evaluation. The team needed to maintain program and politico support to get access to schools. If they expanded the sample they would sacrifice the strength of case studies. But they asserted the need to spend time identifying and describing processes in a small number of intensive case studies.

Other design questions had to be managed. Qualitative research must face questions about reliability, validity, control against bias, and generalizability of findings. One key OPER staff person pulled out in protest over the case study design:

He felt he couldn’t be part of anything that was called evaluation but was so unstructured. I kept saying it isn’t unstructured but structured a different way. If you didn’t have an instrument he wouldn’t be part of it. (An SEA evaluator)

The team devised controls and validity checks. Con- trols against bias were maintained by team training and structure. Because of common training, the team had consistent views of the purpose of the evaluation. At the same time, the variety of expertise within the group built in a questioning attitude and a tension which were the control for validity and reliability. The common understanding provided a research framework for the case studies and the different interests raised important questions. The team’s developing pride and defensiveness about the study built in additional determination to do credible work: “We had to fight hard to do the study originally so the commitment was strong.

We had a collective defensive attitude. . . .” (An Interim Report team member). The struggle to obtain and maintain high-level support for the study created team energy and determination to do a piece of useful, significant work: “We were thinking we were prima don- nas even if nobody else did and we wanted to do something that was useful. Otherwise, why do it?” (An In- terim Report team member). This sense of pride and a “we’ll show them” attitude were an informal quality control.

The designed structure of school visits provided built-in controls. Two team members spent 2 days looking for what, why, and how the school was a “decreaser” (in achievement scores) or an “increaser.” Team members went to the schools with their observation skills but without assumptions. They looked for 2 days for processes that contributed to decreases or in- creases. They made independent hypotheses, discussed them during the intervening week, devised appropriate tests, then spent the final 2 days checking hypotheses. Each team wrote a case report for each school describing the processes identified and predicting whether or not the school would change its achievement score pattern. The team members were a quality control and bias check on each other.

An OPER-based team member described the con- siderations of the case study approach:

It was high-risk, high-gain. If we failed, it was a reflection on us, not on the organization. We were not a part of the main process, nor was the study. . . . We saw case study as appropriate for explanation, not just for description. Not “what,” but “why and how.” The old way made inferences for “what” without the whys and hows. That’s not effective. (An Interim Report team member)

The team had agreed on the value of the case-study approach, designed data collection strategies, and also determined approaches for analyzing qualitative data. They knew that most people expected statistical analyses and they would have to analyze and report in a way that met those expectations halfway. A team member explained.

It’s from the triangulation approach in evaluation. It’s another way of gathering data. In the past, we’d put all our eggs in the basket of stats. The first study had chart after chart of variables saying what happens if you cross parent involvement with curriculum planning, etc., then making generalizations based on these variables. This [case study] seemed a way of combining statistical data from test scores with more qualitative data and putting in some human judgments. (An Interim Report team member)

The analysis and the mode of reporting retained the qualities of in-depth exploration of cases while, at the


same time, displaying relevant statistics to reassure evidence justifying these patterns emerged from data those who mistrust qualitative research.2 analysis.

Thus, The Interim Report team could defend their procedures and they earned and nurtured high-level politico and program manager support. The unique self-management, cross-department staffing and high- level support was settled, along with the commitment to the case-study methodology. The team had ham- mered out the design through arguing and through training sessions, but their approach was not assured until the politico, program manager, and OPER questions and needs were addressed. The team argued itself to consensus on these issues:

Politicos pressured to include in the report a section on statistics. A member recalled resisting that section: “That’s Chapter II. The statistics part was weak. . . . It says nothing. We all hedged about putting it in. . . . The study stands alone, without stats” (An Interim Report member). However, an OPER team member put together statistics showing background variables of the increaser and decreaser schools to meet SEA politico demands by showing the data that had identified the problem of ECE and declining scores in familiar form.

1.

2.

3

Case studies would provide answers and could with- stand questions of limits of generalizability, validity, and reliability; The case study approach was expensive in person- hours and required special expertise, but the high- level support would allow for it; technical assistance could follow the evaluation, but would not be mixed with evaluation. The team should be the managers, designers, evaluators, and reporters of the evaluation. They must be prepared to explain and defend their methods for this new type of evaluation.

Managing the Reporting of Findings The request for The Interim Report had come directly from policymakers as had the focused evaluation question. After months of planning, training, and data gathering, The Interim Report team faced reporting issues. A report including all the case studies would be too long, would breach confidentiality, and would make no recommendations. Yet a report that failed to include descriptions of what was happening in ECE would eliminate one of the strengths of the case study approach.

Reporting issues were resolved to deal with analysis, data display, and ECE politics. A writing team analyzed all of the cases. They looked for patterns among the cases and sought policy or program-related problems in ECE processes in the cases. These patterns were to be incorporated in the “Findings” and “Implications” sections of the report. As noted earlier, the team had built in controls against bias by their training and by building an inter-unit interdisciplinary team. Thus, as they collected data and analyzed the cases, they continuously challenged each other’s interpretations. Through this process the strong patterns and the

*There are many arguments to be made in favor of combining the case-study approach with quantitative approaches, as argued in

Cook and Reichardt’s volume, Qualitative and Quantitative Methods in Evaluation Research (1979).

Tension over reporting findings had existed from the beginning. The team was determined to ensure that their work was used; the initial high-level support and the continuous personal communication with politicos set the stage. The question remained, however, about whether politicos would allow dissemination of findings that implied poor communication to schools and inadequate implementation assistance from SEA or findings that showed that a major component of ECE guidelines-the diagnostic-prescriptive approach to improve instruction-had contributed to mechanistic teaching, more paperwork, and less teacher-pupil interaction. SEA evaluation reports had never been so blunt about shortcomings, especially a program which was called “Riles’ baby.”

The writing team wanted to be hard-hitting and frank, yet they worked with the realization that politicos could modify their writing. One member recalled:

We had continuing debate about how strong the statements could be. It came down to how strong the person felt it was essential. This wus tempered by SEA norms. . . . These implications statements were probably the strongest statements ever to come out of an SEA written report. (An Interim Report member)

Through negotiations with the SEA legislative liaison, a compromise format emerged. The report would give the rationale, strengths, weaknesses, and limitations of case studies, including a statement that case studies have “less control over the emergence of an incorrect explanation” (State Department of Education, 1977, p. 3). Chapter II would satisfy the objections of those who expected statistics,

In addition, the final report used 6 of its 35 pages to explain the case study approach. This methodology was new to many in the evaluation audience. The team, determined to ensure that their findings were considered valid and significant, knew that they needed to educate their audiences. Chapter III included the findings from the analysis of the patterns in the case studies. They showed inadequate communication of the intent of ECE, the need to recognize the impor-


tance of leadership, teachers misinterpreting the ECE goal of individualization and lowering expectations for low performing students, poor planning for staff development, and inadequate in service for improving staff capacities to serve ethnic and language minorities. The findings showed that some teachers interpreted the diagnostic/prescriptive mode to mean self-teaching with commercially prepared programs, that new ECE- funded reading programs were not integrated into the school curriculum, that school people were confused by the various evaluative techniques. Thus the report identified how the inadequate attention to implementation processes were effectively thwarting progress toward ECE goals.

After each section on findings, the team placed an “Implications” section. These were policy recommendations - hard-hitting ones. For example:

- Inservice training should help adapt new programs and skills to the pre-existing practices rather than merely en- forcing compliance to a new program. (p. 25)

-Within the reading curriculum itself, a balanced variety of reading skills should be taught. Scores declined when there was over-emphasis on decoding and phonetic analysis and underemphasis on integrated skills such as com- prehension and story writing. (p. 30)

-The strategic implementation parts of ECE- planning, implementation, internal and external program quality review . . . , and evaluation-need to be more explicitly connected to each other. The ongoing interaction among these parts has been overlooked in some schools. (State Department of Education, 1977, p. 32)

Such statements clearly and concisely stated the fundamental problems with ECE’s initial planning and guidelines and with field assistance.

Politicos allowed the report although they knew that SEA staff morale and ECE participants could be hurt. They could be open to the critical findings and policy implications because of Chapter IV, entitled “Depart- ment Action,” which gave SEA the chance to explain its proactive “ongoing effort to seek appropriate ways of improving departmental, district, and school programs” (State Department of Education, 1977, p. 33).

By the time The Interim Report was printed, the SEA had undertaken intensive re-examination of ECE policies in preparation for new school improvement policy. Chapter IV outlined these efforts and emphasized that The Interim Report study design did not allow for estimates of how many ECE schools were having the observed problems. The chapter answered the anticipated questions from legislators and school people who would demand that SEA do something about the problems. It offered promises of improvement which could convince doubting legislators and school people that the program should be salvaged, despite the problems. The report-writing controversies were settled.

Evaluators Participating in Policymaking to Ensure Utilization Questions remained about the team’s role in reporting and disseminating findings. They were determined to ensure that their work was significant. The timing was perfect.

It was a time of decisionmaking and there were in- dicators of an openness to re-examination and refor- mulation. SEA politicos had structured “half-day sessions” for that purpose. Participants included people from the field assistance unit, Program, OPER, and politicos. It was an ad hoc unit, convened to formulate new policy for school reform, for formulating what eventually became part of legislation - Assembly Bill 65 -designing the new School Improvement Program. One participant explained:

The timing was great; we finished [The Interim Report] in June and Assembly Bill 65 was coming out. [SEA politicos were] wanting to get our level of people- the people who are the consultants and the people who are in the districts all the time- together and say, “now we’ve got a chance to rethink and redo ECE.” We had a whole week of half-day sessions at a hotel. They weren’t purposefully pulling together the people who were on the team but those of us who were there were loud enough that the stuff from the study became the stuff that laid the ground- work for getting Assembly Bill 65. (An Interim Report team member)

The half-day sessions coincided with the report writing. An Interim Report team member recalled: “We were pulling together and using our findings at the half-day sessions. . . . We talked about the study to everyone, in every form we could. I’m sure it was proclaimed in more emotional ways than it was in print.” Another member recalled the sessions as: “Like pitched battles at first but later more collegial. Under [an SEA politico’s] leadership they were at a high conceptual and integrative level, yet they were very concrete about what goes on in schools.” The evaluation findings fed directly into this unique structure for intense decisionmaking through perfect timing, through a team member’s access to the key politicos, and because team members were determined to see that their work had impact. In addition, this decisionmaking required information on school processes and ECE implementation. The Interim Report findings were providing this essential information in-depth and in personal reporting with rich data from people who had been immersed in the schools.

Clearly, no distinction was made between designer, evaluator, evaluation reporter, and policymaker. The team planned the design, pushed it through for ap- proval, carried out the study, vociferously reported the findings, cultivated relationships with politicos to ensure that the findings would be used, wrote the report with clear, hard-hitting policy implications that could


not be soft-pedaled, and then actively proclaimed and promoted their findings and the implications as participants in a key polic~aking arena. A team member experienced in team de~isionmaking had planned for this:

One of the things that was so important to how it turned out was that the group that were the deliverers of it were the doers of it. . . . What they typically do is get two “stars” to design something and then they train people for three hours and then do it. . . . We fought to pick who we wanted on the team. A key thing is ownership. We wanted their time and the only way to have the team own it was to have them there on the planning.

Thus, in-house utilization of the evaluation was assured. The team’s determination, its access to the half-day sessions, and the timing of their reporting combined to ensure utilization in major policy decisions. The SEA was preparing legislative proposals and in-house program alterations in the half-day sessions. The team had continuously informed the SEA politicos as their findings emerged. They had remained attuned to SEA concerns.

The Interim Report was used by legislators, school district ECE managers, and evaluators too. The team had designed their report to highlight policy recommendations for legislators, to show SEA and school district ECE managers the implementation problems in rich detail in the case study data, and to provide background statistical data and an explanation of case study methodology so that the evaluation design would be understood.

The written report progressed slowly. It finally went to the legislature as an “interim report” in Fall 1977, and later the findings were incorporated in the yearly Consolidated Application Evaluation Report (State Department of Education, 1978). Team members heard indirectly that it was well received by legislators and legislative staff. The SEA printed 1,000 copies and the supply was soon exhausted. One team member per- sonally disseminated the report to school people; SEA program staff used the report in SEA workshops in training local evaluators, and at an Evaluators Re- search Association to illustrate an approach to evaluation that looked for effects and for explanations of why schools differed.

The Interim Report evaluation was utilized, there- fore, in the following ways:

1.

2.

for helping SEA and local program managers ex- amine processes of implementation of ECE, to understand how ECE processes interacted with other school processes; as a training, communication, and technical assistance tool for SEA program managers to use with LEAS;

3.

4.

5.

6.

7.

8.

for informing SEA policy deliberations, including policy formulation for new legislation; for providing legislators with a description of ECE implementation problems; for providing SEA politicos with an opportunity to demonstrate to concerned educators and legislators that they were actively working on ECE problems, thus reducing tensions between the SEA and the legislature and between the SEA and the LEAS; for creating a collegial team that bridged the gap between evaluators and program managers; for explaining and demonstrating (to other evaluators, policymakers, and administrators) the design and the utility of case study evaluation; for validating the decision to design more SEA evaluations as small, focused studies using a variety of methodologies and examining processes as well as outcomes.

Although it is difficult to substantiate cause-effect relationships, The Interim Report contributed to a new climate of understanding among legislators, legislative staff, program managers, and evaluators. These people had seen the political battles and the low utilization of previous state-level, outcomes-based evaluations. The Interim Report helped them see that programs like ECE actually restructured education systems. Thus, the evaluations must look at how the program affects, and is affected by, school traditions, communications, governance, staffing, and other process and structure factors. The Interim Report, using the case study methodology, allowed the emergence of rich descriptions of ECE in real schools. It was a welcome change to ECE participants who had seen previous evaluations that focused on achievement score outcomes.

“It isn’t the number-crunchers versus the fuzzy- heads anymore,” an SEA politico commented as he described the acceptance of the small, focused, qualitative evaluations. This mode of evaluation gained with acceptance:

We’re now turning out a whole variety of short studies that are appealing. [Several assemblymen] like what we’re doing on staff development and proficiency and school improvement. . . . We try to tailor the reports to be responsive to the management concerns within the SEA. We ask, where are the glitches, what are the barriers to implementation, what things facilitate. . . . As the reports are gone over and are translated by the legislative analyst and legislative staff, there is enough congruence with legisiators’ preconceptions and enough that isenlightening to legislators. The lnierim Report was the start of our special studies. (An SEA evaluator)

Documents from the Auditor General’s office, the California Education Evaluation and Management Commission, and interviews verify this emergence of a new orientation and acceptance of small evaluations that focused on processes and implementation with a


variety of data sources. New legislation in 1978- 5. evaluation seeking assessment of implementation, Assembly Bill 2506-called for an emphasis on ex- not just outcomes; and amining the nature, extent, and quality of program im- 6. LEA district administrators being involved more in plementation and the new School Improvement Pro- School Improvement processes and decisionmaking. gram legislation (A.B. 65) included an emphasis on this approach to evaluation. In addition, the planning These thrusts can be traced back to the “half-day ses- for School Improvement (for the legislation, the man- sions” where Interim Report team members, fresh agement and implementation, and the evaluation) in- from their observations in schools, described the real corporated recommendations from The Interim Report, problems that LEAS experienced as they coped with including the following: SEA programs.

The evaluators’ personal commitment to the process and the product of The Interim Report study guaran-

1. reducing the emphasis on records-keeping; teed that their audiences would take notice. One team 2. examining the modes of communication to improve member strongly believed in the appropriateness of

SEA-LEA relationships and to incorporate LEA evaluation to describe and explain processes in concerns in SEA planning; schools, another strongly believed that ownership of

3. providing staff development designed for the users; one’s work ensured high quality work. Both put their 4. SEA guidelines allowing greater flexibility; beliefs into action and wanted to let the world know.

ANALYSIS AND IMPLICATIONS

This account of The Interim Report details crucial strategies for managing organizational/political tension while conducting a credible, usable evaluation. Evaluations utilized for program improvement and policymaking are unique (Boruch & Wortman, 1979). Such evaluations merit attention and analysis.

The Interim Report evaluation differed from other evaluations of ECE in its question-framing, its methodology, its personnel, its political support, its organizational support, the “ownership” of the study, and the evaluators’ role vis a vis policymaking. These elements combined to create a salubrious climate for evaluation utilization. The unique elements can be summarized as follows:

1. The study focused on one question, a question that legislators, SEA managers, and school people needed answered. The call for the study contained no laun- dry list mix of questions, no prescription of methodology, no assumptions about relevant variables.

2. The case study design allowed for examination of processes, of what had happened in the course of ECE implementation. The design allowed for in- corporation of information on the values and un- derstandings of ECE participants, including legisls- tors, ECE Program managers, SEA politicos, local administrators. These processes, values, and under- standings are key variables in program implementation (Sproull, 1977; Weatherly & Lipsky, 1977; Pat- ton, 1978). The case study design answered the information needs of the various audiences, including description of problems in implementation and recommendations for policy alteration.

3. The Interim Report evaluators themselves owned the evaluation. The evaluation team conceptualized, designed, negotiated and defended their decisions:

-They thrashed out a case study design and developed training and structures for explaining and defending the methodology.

-They maintained high level support. They remained attuned to political and organizational tensions.

-They were a special team, formed for and by this project and were committed to its success.

-They used the findings in intensive discussion in SEA policy meetings.

-They used the evaluation in the field. 4. The information in The Interim Report was

presented in a format similar to the informal assess- ments typically made by legislators and program managers. The case studies focused on problems and searched for alterable variables; this is similar to the trouble shooting of legislative hearings, school visits, and program manager monitoring. The problems were described and illustrated, then the implications for policy alterations were delineated. Wilson (1979) and Parlett and Hamilton (1976) have suggested that case study or ethnographic evaluation, if managed to address policy concerns, is a design that optimizes the chance for utilization.

The Interim Report was planned around a policy related, focused question which the various evaluation audiences agreed upon. This was good fortune, in part, but the evaluation team selected and defended a problem definition which focused on organizational and interorganizational processes and, from there, selected the appropriate case study methodology.

The evaluation team came together at a time of SEA organizational change-roles were altering, unit func- tions were shifting. The team’s ability to recruit per-


sonnel from various SEA units and to manage themselves enabled them to create a salubrious environment for team work. Situational leadership theory en- courages such practices, where mature professionals work out their own structures for task division, time management, goal setting, recruitment, and training (Hershey & Blanchard, 1977). This team earned this flexibility by staying attuned to organizational/political needs for information, and by frequent informal reporting to SEA politicos. Additionally, the constant reporting prepared SEA politicos to create a constructive defense for any negative evaiuation findings.

Cross-unit commitment to the utility and validity of the case-study evaluation was a positive outcome of the cross-unit recruiting for the evaluation team. The usual intraorganizational tensions were alleviated and the evaluation incorporated all of their perspectives.

The report was not a polished product-it was mim- eographed and stapled. However, it was appropriately brief, it contained a clear explanation of the methodology which answered the anticipated concerns for generalizability, validity, quality control, and fears of legislators who had learned to trust only statistics. Enough description was included to show the logic of conclusions and implications, but not so much as to make the report tedious. Finally, the report’s policy implications were direct, specific, and aimed at all levels of the complex education policy system. They identified malleable variables in the system’s processes and they provided suggestions for change. Their statements of “implications” were negotiated to avoid organizational resistance.

How did all of these key elements get interwoven to produce an evaluation that avoided political and organizational tensions and answered needs of various audiences? -Luck, or at least taking advantage of a lucky setup. The SEA’s constructive stance, the legislators’ identifying a specific evaluation question, the SEA readying for new policy initiatives, SEA units realigning-created the opportune time and openness for this evaluation. Intraorganizational blockages loosened so that people with access, power, and expertise could work together. The SEA, legislators, locals and OPER were open to evaluation innovations. (See Figure 2.)

This analysis of one evaluation shows key elements that can be structured into evaluation management to facilitate evaluation utilization. Evaluators can devise efficient and effective techniques for politically sen- sitive evaluation management. The case study design, the emphasis on process and implementation variables, and the constant communication with policymakers facilitated evaluation utilization. This description and analysis emphasizes and illustrates the importance of maintaining organizational/political support. It illustrates too that organizational/political support is a complex, interwoven process in education policy systems. Evaluation managers must respond to this. They may find that the case study methodology, emphasis on processes, gross-organizational perspectives, and descriptive findings with clear policy implications will help them manage organizational and political tensions in evaluation.

Agreement on focused question

Policy related question-framing

Evaluator involvement in reporting and dissemination

Reporting and implications focus for change on maileable variables

POLITICAL

SUPPORT

\

Access to informal and formal policymaking structures

\ Flexibility for innovative professional work

ORGANIZATIONAL

SUPPORT

Cross-unit recruitment of personnel

Allocation of appropriate personnel and resources

Politically attuned and timed reporting and dissemination Methodology appropriate to the

evaluation question

Case study lent face validity

Figure 2. The Intermixing of Crucial Elements for Facilitating Successful Evaluation Utilization.


REFERENCES

ALKIN, M., DAILLAK, R., & WHITE, P. (1979). Using evaluations. Beverly Hills, CA: Sage Publications.

BAKER, E. (1976). Evaluation of the California Early Childhood Education Program, Volumes I-IV. Los Angeles: Center for the

Study of Evaluation, UCLA.

BORUCH, R., & WORTMAN, P. (1979). Implications of educa-

tional evaluation for evaluation policy. In David Berliner (Ed.),

Review and research in education. Washington, DC: American

Educational Research Association.

CALIFORNIA EDUCATIONAL MANAGEMENT AND EVAL-

UATION COMMISSION. (1978, December). The uses and limits of educational evaluation at the state level. Sacramento, CA: State

Department of Education.

COHEN, D. K. (1970). Politics and research: Evaluation of social

action programs in education. Review of Educational Research, 40, 213-238.

COOK, T. D., & REICHARDT, C. S. (1979). Qualitativeand quantitative methods in evaluation research. Beverly Hills, CA: Sage

Publications.

DICKEY, B. (1980). Utilization of evaluations of small-scale in-

novative educational projects. Educational Evaluation and Policy Analysis, 2(6), 65-77.

HERSHEY, P., & BLANCHARD, K. H. (1977). Management of organizational behavior: Utilizing human resources. Englewood

Cliffs, NJ: Prentice-Hall.

HILL, P. (1979, November). Managing the evaluation of Title I. Paper presented at the University of California at Los Angeles, Los

Angeles, CA.

LECOMPTE, M., & GOETZ, J. (1982). Ethnographic data

collection in evaluation research. Educational Evaluation and Policy Analysis, 4,(2), 387-400.

MCLAUGHLIN, M. W. (1975). Evaluation and reform. Cam-

bridge, MA: Ballinger.

MARSHALL, C. (1981). Evaluation as apolitical tool: A casestudy of California Early Childhood Development Study. Paper presented

AERA Annual Conference, Los Angeles.

MITCHELL, D. E. (1979). Social science utilization in state legislative policymaking, Final report (NIE-G-76-0104). Washington,

DC: National Institute of Education.

PARLETT, M., & HAMILTON, D. (1976). Evaluation as illumina-

tion: A new approach to the study of innovatory programs. In

G. Glass (Ed.), Evaluation studies review annual, Vol. 1. Beverly Hills, CA: Sage Publications.

PATTON, M. (1978). Utilization-focused evaluation. Beverly Hills,

CA: Sage Publications.

ROSENTHAL, A. (1978). Utilization of evaluations in legislative settings. Paper presented at Evaluation Network Conference,

Aspen, CO.

SMITH, N. (1982). The context of evaluation in state departments

of education. Educational Evaluation and Policy Analysis, 4(3), 373-386.

SPROULL, L. (1977). Managerial attention in new educationalpro- grams: A micro behavorial study of program implementation. Un-

published doctoral dissertation, Stanford University.

SROUFE, G. E. (1977). Evaluation and politics. In J. Scribner

(Ed.), The politics of education. Seventy-seventh yearbook of

N.S.S.E.

STATE DEPARTMENT OF EDUCATION, CALIFORNIA. (1975). Early childhood education: First annual report, 1974-1975. Sacramento, CA.

STATE DEPARTMENT OF EDUCATION, CALIFORNIA.

(1977). Interim report on the special study of selected ECE schools with increasing and decreasing reading scores. Sacramento, CA.

STATE DEPARTMENT OF EDUCATION, CALIFORNIA.

(1978). Consolidated application evaluation report. Sacramento, CA.

WEATHERLY, R., & LIPSKY, M. (1977). Street-level bureaucrats and educational innovation. Harvard Educational Review, 47,

171-197.

WEISS, C. (1979, March). Uses of social service research in organizations: The constrained repertoire theory. Paper presented at

the Conference on Theory and Research on Human Service Organi-

zations, Palo Alto, CA.

WILSON, S. (1979). Explorations of the usefulness of case study

evaluations. Evaluation Quaterly, 3(3), 446-459.

ZIGLER, E., & VALENTINE, J. (1979). Project Head Start: A legacy of the War on Poverty. New York: The Free Press.