Lessons Learned from the 316b Choice Question Development

Lessons Learned from the 316b

Choice Question Development

November 1, 2011Presented by: Erik Helm

with: Elena Besedin, Robert Johnston, Julie

Hewitt, and Ryan Stapler

Outline

• Multi-attribute choice question design under three constraints:– Policy issues to be measured– What can be measured environmentally– Measures people can comprehend in a choice question

format• How focus groups informed choice question design• Preliminary results from the Northeast Region survey

(still in the field)• Conclusions

Background on 316b

Water being used by 316b facilities passes through “trash racks” where larger fish unable to swim away are impinged against the screen and usually die.Smaller organisms which pass through the screens are entrained in the cooling system where they are heated and subjected to abrasion and turbulence and also die.

§316(b) of the Clean Water Act requires that the location, design, construction and capacity of cooling water intake structures reflect the best technology available for minimizing adverse environmental impact.

Background on Survey Development

• Survey development began in 2003• Paperwork Reduction Act and Information Collection

Requests:– OMB approved 1 streamlined and 3 full ICRs– Various stages of survey have been available for public

comment a total of 180 days– EPA received about 700 pages of comments

• Survey was externally peer reviewed in 2006• EPA conducted 16 focus groups and 28 cognitive

interviews, producing over 1,600 pages of documentation.

Lessons Learned through the 316b Development Process

• Start with an assessment of what is required to properly inform policy

• Look for measureable environmental outcomes that can be directly linked to policy needs

• Don’t start with oversimplified environmental measures and use FGs to test for the correct level of complexity of environmental information

• Use focus groups to qualitatively test for the ability of participants to separate and assess environmental attributes

Policy and Environmental Outcomes Important in the 316b Survey Design

• EPA has defined adverse environmental impact under successive 316b regulations as being individual fish saved from impingement and entrainment by cooling water systems

• Some stakeholders have stressed that only fish that are commercially or recreationally caught are important

• Other stakeholders have indicated that only population impacts are important

• The use of recirculating cooling systems has the co-benefit of reducing thermal discharge at facilities

• Additional groups have stressed that I&E and thermal issues can impact species assemblages and local ecosystem balance

A Choice Question from 2005

Attribute List in 2009Effect of Policy What It Means

Fish Saved (per Year)

The number of fish that are saved: 0 to 100 score showing the number of young adult fish saved per year. A score of 100 means that no fish are lost in cooling water intakes (all fish are saved). A score of 0 means no additional fish are saved. The current score in Northeast waters is 0.

Catchable Fish

The effect on fish populations that are caught by humans: 0 to 100 score showing the sustainability of fish populations used by humans. It is calculated by the US National Marine Fisheries Service based on the size of fish populations compared to current catch. The current score in the is 65.

Fish Health (All Fish)

The effect on the health of all fish: 0 to 100 score showing the condition of all fish, also called a biotic integrity score. It is calculated from measurements including the number and diversity of fish, the different ways that fish eat, the number of fish living on the bottom, the presence of sensitive fish, the number of diseased fish, and other factors. The current score in Northeast coastal waters is 67.

Aquatic Ecosystems

The effect on coastal ecosystems and food webs: 0 to 100 score showing the ecological condition of affected areas, compared to the most natural waters in the Northeast. Higher scores mean that the area is more natural (see details on next page). The current score in Northeast coastal waters is 44.

$Cost per Year

How much the project will cost your household, in unavoidable price increases for products you buy.

Additional Material on the Aquatic Ecosystem Score that was Removed

Measurements Combined to Form the Aquatic

Condition ScoreWhat Each Measure Means

Water Quality ScoreThe score indicates whether the water is suitable for recreation and aquatic life. It is based on measures such as dissolved oxygen, nitrogen and phosphorus pollution, water clarity, and algae.

Sediment Quality ScoreBased on the survival of bottom dwellers, level of sediment pollution and excessive organic material on the bottom.

Benthic Score (bay, river or lake bottom)

The health of species that live on the bottom (e.g., mussels, crayfish). Measures such things as the number of all species and presence of rare species.

Coastal Habitat Score Based on average historical changes of wetland areas compared to changes in recent years.

Fish Tissue Contamination

Contaminants measured in fish tissue are indicators of contaminants present in surface waters.

The Aquatic Ecosystems Score is a 0 - 100 score showing the effects of policies on the ecological condition of affected areas.

It measures how close affected Northeast waters are to the most natural, undisturbed condition that is possible. Higher scores mean the area is more natural.

The following information is combined to make the final score:

2011 Table Explaining Choice Question Attributes (in Northeast Survey)

Effect of Policy What It Means

Commercial Fish Populations

(Fish Used by People)

A score between 0 and 100 percent showing the overall health of commercial and recreational fish populations. Higher scores mean more fish and greater fishing potential. A score of 100 means that these fish populations are at a size that maximizes long-term harvest; 0 means no harvest. The current score in Northeast waters is 42.

Fish Populations(All Fish)

A score between 0 and 100 percent showing the estimated size of all fish populations compared to natural levels without human influence. A score of 100 means that populations are the largest natural size possible; 0 means no fish. The current score in Northeast waters is 26.

Fish Saved(per Year)

A score between 0 and 100 percent showing the reduction in young fish lost compared to current levels. A score of 100 would mean that no fish are lost in cooling water intakes (all fish would be saved because of the new policy). The current score in Northeast waters is 0. This represents the status quo (no policy) with about 12% of plants already using advanced cooling systems.

Condition ofAquatic

Ecosystems

A score between 0 to 100 percent showing the ecological condition of affected areas, compared to the most natural waters in the Northeast. The score is determined by many factors including water quality and temperature, the health of aquatic species, and habitat conditions. Higher scores mean the area is more natural. The current score in Northeast waters is 50.

$Cost per Year

How much the policy will cost your household, in unavoidable price increases for products and services you buy, including electricity and common household products.

2011 Choice QuestionPolicy EffectNE Waters

Current Situation(No policy)

Option ANE Waters

Option BNE Waters

Commercial Fish Populations(in 3-5 Years)

42%(100% is populations that

allow for maximum harvest)





Fish Populations(all fish)

(in 3-5 Years)

26%(100% is populations without

human influence)


human influence)


human influence)

Fish Saved per Year

(Out of 1.1 billion fish lost in water

intakes)

0%No change in status quo

50%0.6 billion fish saved

95%0.8 billion fish saved

Condition of Aquatic

Ecosystems (in 3-5 Years)

50%(100% is pristine condition)



$Increase in Cost of Living for Your

Household

$0No cost increase

$60 per year

($6 per month)

$72 per year

($5 per month)

HOW WOULD YOU VOTE?

(CHOOSE ONE ONLY)

I would vote for NO POLICY

I would vote for

OPTION A

I would vote for

OPTION B

Focus Group Finding with Regard to Choice Attributes

• Participants generally understood the ecological scores and differences between the attributes– Excluding early details which people found confusing,

participants appeared to understand the ecological concepts being addressed

• Participants expressed preferences and values for these different ecological outcomes– Some people expressed preference for a single attribute or

subset of attributes. – Others said they considered all attributes or keyed in on the

attribute with the greatest change.

Information on Northeast Pilot Survey Results

• EPA started fielding the survey August 30, 2011• Total surveys mailed: 1,440• Completed surveys received: 399• Undeliverables: 117 (8%)• Response rate: 28% (relative to 1,440), 30% (total

minus undeliverables) • Non-response work will begin shortly • Statistical information presented today is based on

just the 330 surveys that have been coded.

Question 3

Not Important

Somewhat Important

Very Important

1. Effect on commercial fish populations 1 2 3 4 5

2. Effect on the fish populations 1 2 3 4 5

3. Effect on fish saved 1 2 3 4 5

4. Effect on the condition of aquatic ecosystems 1 2 3 4 5

5. Effect on cost to my household 1 2 3 4 5

Question 3. When considering policies that affect how facilities use cooling water, how important to you are effects on each of the following scores? Check one box for each. (For reminders of what the scores mean, please see page 7).

Results of t-test on Question 3

Table 1: P-Values from Paired t-testing (two tailed) for Question 3 on the Importance of Ecosystem Scores and Cost

EffectCommercial Fish Populations

Fish Populations (for all fish) Fish Saved

Condition of Aquatic Ecosystems

Cost on My Household

Commercial Fish Populations - < 0.01 < 0.01 < 0.01 0.57Fish Populations (for all fish) < 0.01 - <0.10 < 0.01 < 0.01

Fish Saved < 0.01 <0.10 - < 0.01 <0.05Condition of Aquatic Ecosystems < 0.01 < 0.01 < 0.01 - < 0.01Cost on My Household 0.57 < 0.01 <0.05 < 0.01 -

Average Response 3.97 4.19 4.14 4.34 3.93

P-values indicate that people have different preferences across attributes. Respondents can differentiate between the different outcomes of reducing I&E mortality.

Policy Question 1 Policy Question 2 Policy Question 3

-10%

0%

10%

20%

30%

40%

50%

60%

Figure 1. Percent of Responses by Policy Option

Status QuoOption AOption B

Percent of Options Selected Across Choice Questions

Strongly Disagree

Disagree Neutral Agree Strongly Agree-10%

0%

10%

20%

30%

40%

50%

60%

Figure 2. The survey provided enough in-formation for me to make informed

choices

Strongly Disagree


0%

10%

20%

30%

40%

50%

60%

Figure 3. I feel confident in my answers

Strongly Disagree


0%

10%

20%

30%

40%

50%

60%

Figure 4. Information in the survey was easy for me to understand

Strongly Disagree


0%

10%

20%

30%

40%

50%

60%

Figure 5. Questions were easy for me to answer

Selected Follow-up Question Results

Lessons Learned

• Attributes must be designed to assess policy requirements

• Construct attributes that have clearly measurable ecological end points directly linked to policy needs

• Ecological outcomes of policies don’t have to be “dumbed down” and oversimplified for survey respondents

• People are able to differentiate between well defined ecological measures and express values– Qualitatively in focus groups and quantitatively in surveys

Documents

Lessons Learned from the 316b Choice Question Development