ENVIRONMENTAL MONITORING; A CASE OF LEARNING THROUGH EVALUATION?

Policy Studies Review, November 7986, Vol. 6 , No. 2

Alison Craig

ENVIRONMENTAL MONITORING: A CASE OF LEARNING THROUGH EVALUATION?

Two aspects of policy analysis that have attracted considerable attention in recent 4 ears are evaluation and implementation. Program evaluations are nothing new. However, many changes have occurred over the last 20 years in attempting to make evaluations more effective (Schneider, 1986). The emergence of policy implementation studies during the late 1960s and 1970s has had a significant impact on evaluation research, moving evaluation beyond an activity to see if a program is working to becoming an integral part of policy design (Browne & Wildavsky, 1983). The a i m of both evaluation and implementation research is improved policymaking. The objective is not simply to evaluate, but to assess room for improvement given what is known about the institutional structure (Mead, 1985).

Despite the attention given to evaluation and implementation during the 1970s and early 1980s, the research has not produced a more predictable relationship between policy objectives and program outcomes. This can partly be attributed to implementors, like evaluators before them, trying to prevent failure before it occurs rather than learning from problems as they are encountered. The focus of implementation has been on designing policies that would anticipate and avoid problems, rather than on developing workable policy objectives. Integrating evaluation with implementation means measuring a program not by what should be done but by what can be done. This requires having the ability to learn from the experience gained during implementation and being able to alter program objectives so they contribute to a more effective policy (Elmore, 1982; Browne .% Wildav- sky, 1983).

This paper looks at the role evaluation has played in developing a national environmental monitoring system. Environmental monitoring programs are designed to provide information useful in enforcing existing environmental statutes and in helping to detect and prevent future problems. The need for an effective national environmental monitoring system that can adequately address these goals is well recognized. Despite the considerable number of monitoring programs which have been conducted, there is still a lack of reliable data that can be used to support environmental policymaking. Several reports have been written criticizing past programs. A point of fundamental concern is how well these evaluations have been incorporated into the current development of an effective nationwide monitoring system (National Environmental Monitoring Hearings, 1984).

This report examines one program in particular, the ?'National Status and Trends Program" administered by the Department of Commerce, Na- tional Oceanic and Atmospheric Association (NOAA) . Some background information on environmental monitoring programs is provided, followed by a discussion of evaluations of past programs and recommendations made for

I would like to thank Dr. Joseph Rees and Dr . Lauriston King for critical review of this manuscript.

366

Craig: Environmental Monitor ing: A Case of Learning Through Evaluat ion? 367

improvements. First, 1'11 look at the "Status and Trends" program and h m well it addresses past recommendations. Then I will consider whether the experience gained during implementation of earlier programs has in- fluenced the objectives of environmental monitoring programs. I conclude by focusing on whether or not environmental monitoring in general, and ttStatus and Trends" in particular, demonstrates a case of learning by evaluation.

BACKGROUND ON ENVIRONMENTAL MONITORING

Over the last several decades, an increasing amount of attention has been given to the problems associated with chemical contamination of the marine environment. In 1962, Rachel Carson, an ex-employee of the U.S. Fish and Wildlife Service, wrote The Silent Spring warning that the ltChemical Age" and the growth in variety and volume of chemicals would expose us to serious environmental problems. Since Carson's warning, several national and Presidential commissions have expressed the need for developing an effective monitoring system that would provide scientifically valid information about the effect of pollutants on the environment (Nation- al Envirnomental Monitoring Hearings, 1983).

In 1970, President Nixon directed the Council on Environmental Quality to make a study of ocean disposal for waste materials. The councilfs report cited a "critical need for a national policy on ocean dumping" and provided the basis for the Marine Protection, Research and Sanctuaries Act of 1972, commonly called the Ocean Dumping Act. The Ocean Dumping Act, amended in 1977, prohibits the dumping of material harmful to the marine environment and designates NOAA as the lead agency in carrying out a comprehensive and continuing program of monitoring marine environmental quality (U. S. Congress, House Report 95-325).

EVALUATIONS OF MONITORING PROGRAMS

Several problems have been encountered in trying to implement a nationwide environmental monitoring system. The nation has committed approximately $50 billion per year to environmental management but the results obtained have been small (National Environmental Monitoring Hear- ings, 1983). How to make these monitoring programs more effective has been the subject of several reports and committee hearings.

In 1977, the National Research Council (NRC) conducted a study of the capabilities of the then existing environmental monitoring programs. The council reported that the monitoring programs currently being conducted did not provide information useful in implementing national policies for environmental management. The real problem outlined in the NRC report seemed to be that although there have been several reports and commissions to study existing monitoring systems and the databases generated, there had been no way to fit these programs together into a national environmental monitoring network with an extensive nationwide database. The NRC made several recommendations for improvement in environmental monitoring programs. First, new monitoring programs should begin with ltprototypelf studies--studies to gain operating experience about the many aspects of carrying on a comprehensive long-term program. A prototype study would provide experience with operational aspects such as deciding where to sample, how to sample and what to sample for. The study would also aid in developing the best methods for collecting, storing and analyzing data so that the knowledge gained could be made available as useful

368

information. A second recommendation by the council was to rely more on existing programs as a way to reduce costs and encourage program coordination. Third, a nationwide program should be scientifically planned and conducted. The program should be developed by technical experts in consultation with agencies currently conducting monitoring programs and with potential users of monitoring data (Blodgett , 1977).

Since the 1977 report by the NRC there have been several additional reports showing deficiencies in monitoring programs. In 1983, and again in 1985, the House Subcommittee on Natural Resources, Agriculture Research and Environment held hearings on the condition of the nation's environmental monitoring programs. Several of the witnesses called during the hearings repeated the need for improved scientific leadership and more coordination of existing programs. Mr . Walter Lyon , an engineering consultant and adjunct professor of civil engineering at the University of Pennsylvania, stated that monitoring by federal agencies represented only a small share of the total national environmental monitoring effort, with state, local and private groups investing three to four times the effort of the federal government. In addition, federal monitoring efforts were scattered throughout many programs administered by several different agencies - Without coordination within and between these agencies, information could not be put together in a form useful at the state and local level. Mr. Lyon stated that setting up a national monitoring program would require determining the extent of existing programs, agreeing upon the goals for a national program, and bringing together the existing efforts under one program aimed at serving the needs of those who use the information. Testimony by D r . John Farrington, director of the Coastal Research Center and senior chemist at Woods Hole Oceanographic Institute, expressed the need for close interaction between research activities and monitoring programs so that the data generated by the programs could be accurately interpreted. Distinctions must be made between man- induced environmental changes and natural variability. Carrying out an effective monitoring program requires understanding how to measure the effects of man's activities as well as determining what the effects are (National Environmental Monitoring Hearings, 1983).

The major concern of both the 1983 and the 1985 hearings was that despite all the monitoring programs being conducted, information on the quality of the nation's environment remains poor and is of little use in supporting environmental policymaking. Coordination of existing monitoring programs is seen as the mechanism for delivering reliable information to decisionmakers. The need for a coordinated nationwide monitoring network is expressed repeatedly, yet programs remain fragmented. Atten- tion needs to be given to why coordination is so difficult.

The a i m of coordination is to produce mutually supportive programs. Yet different programs serve different purposes. Coordination requires choosing which goals are most important. There can be no coordination without first agreeing on objectives. The real problem becomes one of sorting out and choosing among vague, multiple and often conflicting objectives (Pressman 8 Wildavsky, 1973).

Focusing on coordination as the answer to improved environmental monitoring programs ignores the real problem of agreeing on what can or should be expected from such programs. Conducting a monitoring program requires first agreeing on what should be accomplished and then deciding what to sample, where, and how often. No one set of measurements can answer all the questions about environmental quality. Selecting what measurements will be made means many concerns will be left out. What is

P o l i c y Studies Review, November 1986, 6:2


most important to measure will vary considerably with different locations and different usages. In addition, coordinating among different federal, state, local and private monitoring programs requires establishing what the appropriate federal role should be. Is it simply to coordinate monitoring activities of other programs and then establish environmental trends on a national scale, or should the federal role be deciding what is needed to measure environmental trends and then funding programs accordingly? Should the federal role be to fill in the gaps in existing monitoring programs or to develop an extensive nationwide program? (National Environ- mental Hearings, 1984).

Developing an improved environmental monitoring system requires a deliberate altering of objectives so they more closely resemble what can be accomplished. When programs fail to meet the initial objectives stated, faulty implementation is often cited. But implementation problems are only one possibility. Program failure could be because initial expectations were too high (Pressman & Wildavsky, 1973). Instead of focusing on lack of scientific leadership and lack of coordination as explanations for why monitoring programs have failed to produce information useful to environmental policymakers, perhaps we should also ask i f too much was expected. If so, have the expectations about what information monitoring programs can provide changed with the experience gained through previous implementation?

THE "STATUS AND TRENDS" PROGRAM

The objective of this paper is to explore whether environmental monitoring demonstrates a case of learning through evaluations by focusing on the development of one program in particular, NOAA's "Status and Trends" program. Status and Trends is a national program administered by the Ocean Assessment Division of NOAA's National Ocean Service at a total cost of over $5 million per year. The overall goal of the program is to docu- ment the status and long-term changes in the quality of coastal environ- ments. Because toxic chemicals pose some of the greatest threats, the emphasis of the program is on measuring levels of these chemicals in fish, mussels, oysters and sediments collected from the Atlantic, Pacific and Gulf coasts and observing effects these chemicals may cause (Ehler & Calder, 1986).

Status and Trends is based on experience gained during several of the previous monitoring programs, particularly the "National Musselwatch Program" carried out by the Environmental Protection Agency from 1976- 1978. The goal was not so much to measure the effects of chemical contaminants on the environment, but rather to determine how to measure these effects (National Environmental Monitoring Hearings, 1983). The program used mussels and oysters to monitor pollution because these animals are widely distributed, they concentrate chemicals in their tissues by factors much greater than the surrounding seawater making it easier to detect trace levels, and they have relatively stable local populations extensive enough to support the repeated samplings necessary for collecting data on short- and long-term changes. In addition they are a commercially valuable seafood, so measuring chemical contaminants is of interest for public health reasons (Texas A&M Research Foundation, 1984).

In 1982, an international workshop was held to re-assess the Mussel- watch concept and to evaluate the accomplishments and deficiencies of the program. One of the major accomplishments was an extensive database

Musselwatch was considered a prototype program.

370 Policy Studies Review, November 1986, 6:2

against which future chemical changes could be measured. Among the problems were: data management was inadequate so information was not promptly available, a statistical design was not established before sampling and analysis began, and only limited numbers of chemical contaminants were measured because of analytical limitations.

In 1984, NOAA began developing Status and Trends as a program that would build upon and improve the Musselwatch program and other environmental monitoring programs carried out during the 1970s. Two planning workshops went into deciding what the program should contain. Some state agencies, experts at academic institutions and federal agencies were contacted about the type of information they wanted the program to provide. There was considerable agreement among everyone involved. The main emphasis was on developing a long-term, nationwide program that would provide a more uniform and extensive database on levels of chemicals in the marine environment. The information gathered would be geared primarily towards environmental managers and politicians (Calder , personal communication).

Approximately 150 sites, 50 sites on each coastline, were selected by NOAA for sampling. The sites were carefully chosen to integrate chemicals contaminants from a broad area. Monitoring contaminant accumulations from specific sources is not a goal of the program, so sites dominated by single sources of contaminants were avoided. The scientific work is being conducted through NOAA's National Marine Fisheries laboratories, a con- sortium of private f i r m s , and the Texas A&M Research Foundation. Scien- tists from NOAA's Ocean Assessment Division (OAD) are responsible for: (1) collecting and analyzing historical data, ( 2 ) maintaining and updating the database with information collected by the various groups and, ( 3 ) evaluating the data and producing assessment reports dealing with national marine environmental quality issues. An issue of significant concern in the program is the ability of comparing data collected by various groups over a period of time. To promote comparability, established standardized procedures are followed and regular interlaborntory comparison exercises are conducted using reference materials prepared specifically for the program. In addition to the samples collected for immediate analysis, samples from approximately 20 percent of the sites are being collected for long-term storage. NOAA has contracted the National Bureau of Standards to set up a specimen bank for storage and retrospective analysis. A s new contaminants are recognized and become important concerns, these carefully stored samples will serve as benchmarks against which contemporary pollution levels can be measured.

All the data collected by the program participants are returned to NOAA's OAD for storage in a computer-based system. The data will be analyzed by NOAA's scientists and will be used to answer questions such as whether general environmental conditions are getting better or worse, how existing conditions among different coastal areas compare, and whether these conditions are approaching or exceeding harmful levels (Ehler & Calder, 1986).

"STATUS AND TRENDS" - LEARNING BY EVALUATION?

A considerable number of evaluations have been made on the state of environmental monitoring programs. A good deal of attention has been focused on how to develop an effective national monitoring system, such as Status arid Trends, that would provide information useful in environmental management decisions. Whether o r not Status and Trends demonstrates

Craig: Environmental Monitor ing: A Case of Learning Through Evaluat ion? 37 1

learning through evaluation requires looking first at how past criticisms have been addressed and then whether program expectations have evolved in view of what can be accomplished.

Basically, three recommendations keep resurfacing in evaluations made of monitoring programs: (1) the need for a more scientific approach to planning, collecting and interpreting the data, ( 2 ) more reliance on, and coordination of, existing monitoring programs, and ( 3 ) consideration of how to make the data more useful and effective in management decisions.

With respect to the first recommendation, the Status and Trends program has a strong scientific approach to environmental monitoring. The program goes well beyond simply monitoring levels of chemical contaminants for the purpose of enforcing existing environmental regulations. Status and Trends is designed as a research experiment that will provide information about changes in the levels of chemical contamiriants and the effects these changes are having on the marine environment. In developing the Status and Trends program, NOAA has incorporated many of the sugges- tions for improving scientific leadership that were brought out in evaluations of earlier programs.

First, Status and Trends began with a "prototypet1 study, the Mussel- watch program, that provided experience with the many different aspects of operating a nationwide program. Musselwatch established methods for how to measure the effects of chemical contaminants and provided insight into problems the program might encounter--such as differentiating between man-induced and natural variations, the limitations of analyzing just a few of the many chemical contaminants being discharged into coastal waters, and ensuring the data are easily available and presented in a usable form.

Another improvement in planning the Status and Trends program was that scientists were included in the beginning stages of program development. The statistical design, collection methods, and analytical techniques were established before sampling began. Workshops were held to determine which chemicals were to be measured and sample locations were carefully chosen. NOAA's long-term commitment to the program (funding for 5, potentially 10 years), will allow Status and Trends to produce a much more extensive and scientifically valid database than was available through earlier programs.

The second point of concern is whether Status and Trends shows more reliance on and coordination with existing monitoring programs. The Status and Trends program, to some extent, addresses this concern. The program includes the database generated by Musselwatch and many of the current sampling sites are near to sites used in Musselwatch. Status and Trends is coordinating the monitoring conducted on all three coasts and will produce one nationwide database. Part of NOAA's marine monitoring activities include developing an inventory of existing local, state, federal and private monitoring programs. The inventory includes the purpose of the program, type of data collected, data quality and format, and data availability (National Environmental Monitoring Hearings, 1984). Part of the problem in integrating data from existing monitoring programs with Status and Trends is that different programs collect different data, follow different procedures, and use different laboratory techniques. It is difficult to modify existing programs because of the diversity of interests they serve. The institution of quality control seen in Status and Trends is an important step towards coordinating different data sets collected.

Status and Trends was developed with an awareness of the past problems in data management. Several steps have been taken to ensure the

372 Policy Stud ies Review, November 1986, 6:2

data is readily available. Data will be stored in a computer system designed for easy access, there will be one uniform database produced, and the data will be analyzed using a program designed to answer specific questions useful in management decisions.

The third area of concern is how effective the Status and Trends program will be in turfling the data obtained into usable information. Several of the earlier recommendations for improving data management were included in the program design. The program was developed with a specific goal: gaining information on the status and long term changes of chemicals in the marine environment. In developing the program, NOAA contacted several potential users of the data and held regional workshops to see what has been done and what type of information is needed. Prin- cipally, state agencies were contacted since they are the primary decisionmakers. By identifying the potential users as environmental managers and politicians, the data can be put in a form most useful to them.

It is too early to tell how decisionmakers will use the data since the program is still in the preliminary data reporting states. The initial data reports from Status and Trends provide a broad description of the nation's coastlines showing where anamolies lie. The reports indicate what to be alert for and what areas are relatively clean. How this information will be used in environmental management decisions depends to a large extent on how successful state and local programs are in tieing into this data base. Although there has been a good deal of talk about modifying state and local programs to fit in with Status and Trends, there has been little progress (Butler, personal communication).

Perhaps one way to make Status and Trends more useful to state and local decisionmakers would be to include them during sampling. This could be done by contacting representatives of relevant state or local agencies when sampling is to be done in their area. Such representatives were contacted initially during site selection. However, more contact during actual data collection would allow those who will be making management decisions to become more aware of the program's objectives and expectations and to identify changes that could help make the program more useful to them. It could also make sampling more successful. Including information and observations of people familiar with the area could make the program more alert to specific events that might happen between the annual sampling trips.

CONCLUSION

The objective of this paper has been to determine whether Status and Trends demonstrates a case of learning through evaluation. To some extent it does. Status and Trends builds on experience gained during earlier monitoring programs. The program incorporates several of the recommendations made for improving monitoring efforts. Furthermore, it is designed to evolve as we learn more about environmental quality and our ability to measure i t . One change will be in moving from measuring levels of chemical contaminants to measuring biological response to these chemicals (Butler, personal communication).

Designing a program in light of what can be done requires the flexibil- ity to alter initial program objectives. It is difficult to say what changes have occurred in the expectations for environmental monitoring programs. The objectives of environmental monitoring-are still vague. Yet, examining the design of the Status and Trends program does show some subtle changes. The aim is to provide an understanding of environmental quality


on a national scale. Coordination of existing programs is to be achieved by providing a nationwide database for state and local programs to tie into rather than to restructure existing programs. The data is being collected with specific users in mind. The ability of those using the information to adapt the program to their needs will be important in helping Status and Trends contribute to more effective environmental management decisions.

Learning through evaluation requires continuous evaluating from learning as you go that will contribute to the gradual, cummulative improvement in programs and policies (Browne & Wildavsky, 1983). The key to the success of the program is in how well the program continues to evolve and learn.

REFERENCES

Blodgett, J. E. (1977, July 18). Environmental monitoring: Needs and opportunit ies (pp. 166-167). The Library of Congress. Congressional Research Service.

Browne, A. & Wildavsky, A. (1983). What should evaluation mean to implementation? In Implementation. Berkeley: University of California Press.

Butler, L. (1986, July 31). Personal communication, NOAA, Ocean As- sessments Division.

Calder, J . (1986, April 3). Personal communication. Program manager of "Status and Trends , I 1 NOAA, Ocean Assessment Division.

Ehler, C. N . , & Calder, J . A . (1986, April). Monitoring environmental quality. Sea Technology, 32-37.

Elmore, R . F. (1982). Backward mapping: Implementation research and policy decisions. In Studying implementation, methodological and ad- ministrat ive issues. Chatham, NJ: Chatham House Publishers, Inc.

Mead, L.M. (1985). Policy studies and political science. Policy Studies Review, 5(2), 319-335.

National Environmental Monitoring. (1983, May 19, 20, June 2) . Hearings before the Subcommittee on Natural Resources, Agriculture Research and Environment of the Committee on Science and Technology. U . S . House of Representatives. 98th Congress, 1st session. No. 70.

National Environmental Monitoring. (1984, March 28). Hearings before the Subcommittee on Natural Resources, Agriculture Research and Environ- ment of the Committee on Science and Technology. U.S. House of Representatives. 98th Congress, 2nd session. No. 138.

Pressman, J . , & Wildavsky, A. (1973). Implementation. Berkeley: University of California Press.

Schneider, A.L. (1986). The evolution of a policy orientation for evaluation research: A guide to practice. Public Administration Review, 46(4), 356-363.

Texas AeM Research Foundation for Geochemical and Environmental Re- search. (1984, December). Technical proposal for analysis of bivalves and sediments for organic chemicals and trace elements (RFP No. 84- DGS-00219 (CZM)) (unpublished). Submitted to U. S. Department of Commerce, NOAA.

U . S. Congress. House miscellaneous reports on public bi l ls. House Reports Nos. 292-345 with exceptions. 95th Congress, 1st session. Report 95-325.

Environment and Natural Resources Policy Division.

Documents

ENVIRONMENTAL MONITORING; A CASE OF LEARNING THROUGH EVALUATION?