#1

Environmental Management:Risk Assessment, Multi-Criteria Decision

Analysis, and Adaptive Management Techniques for Addressing Model Uncertainty and

Reliability

Igor Linkov & Pat DelimanUS Army Engineer Research and Development Center

Environmental LaboratoryIgor.Linkov@usace.army.mil, Phone: 617-233-9869

Patrick.N.Deliman@usace.army.mil, Phone: 601-634-3623

#2

Adaptive Risk-Based Planning

Ad hoc Process· Include / Exclude?· Detailed / Vague?· Certain / Uncertain?· Consensus / Fragmented?· Rigid / Unstructed?

Scoping

Remedial Investigation

Feasibility Study

Remedy Selection & Record of Decision

Remedial Design/ Remedial Action

Five-Year Review

Record of Decision Modification

Stakeholder's Opinions and

Values

Risk Analysis

Cost

Modelling/ Monitoring

Political / Litigation Pressure

Decision

Decision Analysis Framework

Scoping

Remedial Investigation

Feasibility StudyMonitor System

Response

Data Interpretation, Model Analysis

and Improvement

Implement Management

Strategy

Model Prediction and Management Plan Improvement

Stakeholder's Opinions and

Values

· Agency-Relevant· Currently Available Software· Variety of Structuring Techniques· Iteration / Reflection Encouraged· Facilities Stakeholder Input

Risk AnalysisCost

Modelling/ Monitoring

Monitoring Results

Modelling Results

a) Ad hoc Decision Process

b) Adaptive Decision Analysis Framework

#3

• The Government Performance and Results Act (GPRA) – “provide for the establishment of strategic planning and

performance measurement in the Federal Government” (OMB, 1993).

– embodied a push for better planning, greater accountability, and straightforward performance evaluation in government.

• OMB Program Assessment Rating Tool (PART) – rates the performance of a program through a series of yes/no

questions– Scores on four primary areas: program purpose & design,

strategic planning, management, and results & accountability. – performance metrics used by the program are essential to

PART.

Agencies need to relate response to the mission goals and track the progress and performance

Risk-based Planning: Top-down Drivers

#4

• For stakeholders, the root issue is: fear of becoming a victim to (uncompensated) loss– Layperson: Risk = Hazard x Perception– Expert: Risk = Hazard x Exposure x Consequence

• Core concerns tend to be: trust, control, process, information and timing

Local communities need to understand actions by the Agenciesand like to see their values accounted for

Risk-based Planning: Bottom-up Drivers

#5

• What are the flood and storm threats to coastal Louisiana/Mississippi?

• What do we have to lose and how vulnerable are we?

• What should be our planning timeframe?

• What can be done to reduce risks for our planning timeframe?

• How is the public, agencies, and others involved?

• For taking action at varying scales, what is the cost and risk reduction?

• For taking action at varying scales, what are the adverse impacts to significant resources?

• How do you decide what actions to take?

• How much information is necessary to make decisions?

Coastal Louisiana Restoration Planning:What Questions are We Trying to Answer?

After E. Russo

Risk and/or Uncertainty elements are present almost in every question

#6

Information and Planning/Decision CyclesInformation gathering and

decision-making are two separate cycles in environmental

management

Modeling/Software/GIS…Technology-based Fix in Information Age

Integration – Need for Revolutionary

Changes

After Roman, 1996

#7

Main Points

• Risks and benefits associated with alternative management strategies are difficult to quantify.

• Model, Parameters and Scenario uncertainty and variability associated with predicting efficiency of management options as well as stakeholder value judgment are important to consider

• Challenges of risk assessment and planning for situations with a limited knowledge base and high uncertainty and variability require coupling traditional risk assessment and planning with multi-criteria decision analysis (MCDA) to support regulatory decision making

#8

• Risk-based Planning: Top-down and Bottom-up Drivers

• Risk and Uncertainty– Traditional Way of Dealing with Uncertainty – Need for Formal Decision Analysis

• MCDA -Summary

• Example:– MCDA Use to Select Performance Metrics for Oil Spill

Response Planning– RA/MCDA Application for Sediment Management

• Conclusion

• References

Presentation - Overview

#9

AD HOC Process

Quantitative? Qualitative?

Current Decision-Making Processes

Decision-Maker(s)

Include/Exclude?•Detailed/Vague?

•Certain/Uncertain?•Consensus/Fragmented?

• Iterative?• Rigid/unstructured?

Risk Analysis

Modeling / Monitoring

Stakeholders’ Opinion

Cost or BenefitsTools

Challenge: Multiple & Uncertain Criteria

#10

Challenges to Complex Decision-making

• “Humans are quite bad at making complex, unaided decisions” (Slovic et al., 1977).

• Individuals respond to complex challenges by using intuition and/or personal experience to find the easiest solution.

• At best, groups can do about as well as a well-informed individuals if the group has some natural systems thinkers within it.

• Groups can devolve into entrenched positions resistant to compromise

• “There is a temptation to think that honesty and common sense will suffice” (IWR-Drought Study p.vi)

#11

• Model Uncertainty– Differences in model structure resulting from:

model objectives computational capabilities data availability knowledge and technical expertise of the

group– Can be addressed by

considering alternative model structures weighting and combining models Eliciting expert judgment

Problem: Model UncertaintyLinear Model

y = 3x - 0.6667

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4

X

Y

Polynomial Modely = 2x2 - 5x + 6

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4

X

Y

Mechanistic models for environmental risk assessment are very uncertain and expert judgment is required

#12

• Parameter Uncertainty– Uncertainty and variability in model

parameters resulting from data availability expert judgment empirical distributions

– Can be addressed by Probabilistic Simulations (Monte-

Carlo) Analytical techniques (uncertainty

propagation) Expert estimates

Problem: Parameter Uncertainty

Many parameters and factors important for risk assessment are not well known, reported ranges are large and often

unquantifiable

Mean+SDMean-SD

Mean+SEMean-SE

Mean

Oil and Grease in Sediment

Co

nc

en

tra

tio

n (

pp

m)

49

51

53

55

57

59

VAR1

VAR155.00050.00057.000

#13

Subjective Interpretation of the Problem at Hand

What is the relative influence of modeler perception on model predictions?

Problem: “Modeler/Scenario Uncertainty”

#14

Subjective Interpretation of the Problem at Hand

What is the relative influence of modeler perception on model predictions?

Problem: “Modeler/Scenario Uncertainty”

#15

Multi-Criteria Decision Analysis and Tools

• Multi-Criteria Decision Analysis (MCDA) methods:– Evolved as a response to the observed inability of people to

effectively analyze multiple streams of dissimilar information– Many different MCDA approaches based on different theoretical

foundations (or combinations)

• MCDA methods provide a means of integrating various inputs with stakeholder/technical expert values

• MCDA methods provide a means of communicating model/monitoring outputs for regulation, planning and stakeholder understanding

• Risk-based MCDA offers an approach for organizing and integrating varied types of information to perform rankings and to better inform decisions

#16

Risk Analysis

Modeling / Monitoring

Stakeholders’ Opinion

Cost

Decision Analytical Frameworks• Agency-relevant/Stakeholder-selected

• Currently available software•Variety of structuring techniques • Iteration/reflection encouraged

•Identify areas for discussion/compromise

Decision-Maker(s)

Sharing Data,Concepts and Opinions

Evolving Decision-Making Processes

Tool Integration

Decision Integration

#17

Simplified Decision Matrix

Plan Cost EcoHealth

Human Health

A 100 10 5

B 100 5 10

C 150 10 10

D 150 10 15

#18

Criteria 1 Criteria 2 Criteria 3 Criteria 4

Alt. 1 Monitoring Results

Stakeholder Preference

Economic Cost Non-monetary benefit

Alt. 2 Monitoring Results

Stakeholder Preference

Economic Cost Non-monetary benefit

Alt. 3 Monitoring Results

Stakeholder Preference

Economic Cost Non-monetary benefit

Alt. 4 Monitoring Results

Stakeholder Preference

Economic Cost Non-monetary benefit

How to interpret these results?

How to combine these criteria?H

ow

to

co

mp

are

th

ese

alt

ern

ativ

es?

Example Decision Matrix

#19

Decision Analysis Methods and Tools

Elements of Decision Process

Ad Hoc Decision-Making Comparative Risk Assessment Multi-Criteria Decision Analysis

Define problems Stakeholder input limited or non-existent. Therefore, stakeholder concerns may not be addressed by alternatives.

Stakeholder input collected after the problem is defined by decision-makers and experts. Problem definition is possibly refined based on stakeholder input.

Stakholder input incorporated at beginning of problem formulation stage. Often provides higher stakeholder agreement on problem definition. Thus, proposed solutions have a better chance at satisfying all stakeholders.

Generate alternatives Alternatives are chosen by decision-maker usually from pre-existing choices with some expert input.

Alternatives are generated through formal involvement of experts in more site-specific manner.

Alternatives are generated through involvement of all stakeholders including experts. Involvement of all stakeholders increases likelihood of novel alternative generation.

Formulate criteria by which to judge alternatives

Criteria by which to judge alternatives are often not explicitly considered and defined.

Criteria and subcriteria are often defined.

Criteria and subcriteria hierarchies are developed based on expert and stakeholder judgment.

Gather value judgments on relative importance of criteria

Non-quantitative criteria valuation weighted by decision maker

Quantitative criteria weights are sometimes formulated by the decision maker, but in a poorly justified manner.

Quantitative criteria weights are obtained from decision makers and stakeholders.

Rank/select final alternatives

Alternative often chosen based on implicit weights in an opaque manner.

Alternative chosen by aggregation of criteria scores through weight of evidence discussions or qualitative considerations.

Alternative chosen by systematic, well-defined algorithms using criteria scores and weights.

#20

Problems

Alternatives

Criteria

Weights

Synthesis

Decision

Decision Matrix

Evaluation

RA

MCDAFeedsRA

MCDARAFeedsMCDA

AdaptiveManagement

Linking RA, AM and MCDA

#21

Specify Problems& Opportunities

Inventory & ForecastConditions

FormulateAlternative Plans

Evaluate Effects ofAlternative Plans

CompareAlternative Plans

SelectRecommended Plan

Corps Planning

Problems

Alternatives

Criteria

Evaluation

Decision Matrix

Weights

Risk and DecisionAnalysis Framework

Synthesis

Decision

Decision Analysis ToolsMAUT

Risk Analysis ModelsWave/Storm SurgeInfrastructure ModelsEcosystem ModelsEconomic Models

Figure 2. RIDF

Scenario Analysis

Risk Informed Decision Framework: Restoration Planning for Coastal LA and MS

#22

Example 1: Performance Metrics for Oil Spills Response Planning*

• Framework for selecting metrics– Multiple stakeholders

Agencies (federal, state, local) Responsible parties Local residents NGOs (business, environmental, etc.)

– Integrate deliberation and science to link goals, objectives, metrics, and measures

– Compatible with existing planning, decision-making, and assessment processes

– Completed as part of preparedness planning*based on Linkov, Seager, Figueira, Tkachuk, Levchenko, Trevonnen (2007), funding provided by

NOAA through CRRC, UNH.

#23

Examples (oil spills response)

• Endpoint – Miles of shoreline impacted or cleaned vs. areas protected (e.g., by redirecting or

containing oil). – Number of fish, birds or other wildlife killed or injured (per unit search area). – Number of “appropriate” (not exotics) animals rehabilitated and released. – Degree of change to beaches and sandbars from clean-up actions. – Types of animals and vegetation present after spill cleanup.

• Process – Did getting required permits delay response action? – Rate of bird handling at rehabilitation center. – Time to deploy booming and double-booming in sensitive areas.

• Resource – Amount of oil containment boom deployed. – Number of volunteers deployed.– Number sandbags deployed.

#24

Challenges

• Challenges to defining “good” metrics– What is most relevant may be very difficult to measure.– Metrics may be indirect measures of what people really care about.– What is easy to measure may not be relevant to what people care

about.– There can be disagreements about thresholds to differentiate

“good” versus “bad.”– Accuracy and reliability of data recording is a challenge.– Paucity of baseline data.– Timing of measurement can affect assessment of performance.– Difficult to communicate to the public – or at least that is

managers’ perception.– Weightings and aggregation.

#25

Characteristics of Good Measures

– scientifically verifiable– cost-effective– easy to communicate to a wide audience– relevant to what people care about– decision or action relevant– credible– scalable over an appropriate time period and geographic

region – sensitive to change

#26

Oil Spill Response Metrics Taxonomy by Type of Information Measured

Economic Thermodynamic Environmental Ecological Human Health Socio-Political

Clean up costs.Property & eco-

system damage.

Ecosystem damages or lost services.

Lost marginal profits.Volunteer

opportunity costs.

Volume of oil spilled, recovered, des-troyed, or contained.

Slick area and thickness.

Mass of clean up wastes generated.

Volume cleaning agent deployed.

Chemical concen-tration & toxicity.

Habitat suitability, e.g., acres shellfish bed.

Length of oiled shoreline.

Degradation rates.Residual Risk

Wildlife deaths or populations, fecundity and recovery rates.

Biodiversity.Catch sizes.Plantings, seedings.Habitat Suitability

Threatened pop-ulation

Quality-adjusted-life-years (QALYS)

Disability-adjusted-life-years (DALYS)

Life-expectancyInjuries

Newspaper column inches, minutes TV coverage, web hits.

Volunteerism.Public meeting

attendance.Critical sites pro-

tectedHistoric sites pro-

tected

#27

Assessment Criteria

#28

Metric Assessment by Criteria

#29

Criteria Weight

#30

Rank Acceptability Analysis

#31

Pairwise Metrics Domination

#32

Sensitivity Analysis

COST Most Important

ENVIRONMENTAL Relevance Most Important

#33

Example 2: NY/NJ Harbor

#34

Issues• Harbor among most

polluted in U.S.

• >106 yd3 fail regional criteria for ocean disposal

• Existing disposal site closed 1 Sep. 97

• Proposed deepening

Example: NY/NJ Harbor

#35

Example: Decision Methodology

• Proof of Concept Study

• Objectives– Integrate comparative risk assessment results with cost and stakeholder

decision criteria– Use decision criteria/performance measures from published data and

proposed costs– Test decision tools, methodology and results

• Set contaminated sediment management options

• Set decision criteria/performance measures

• Software - Criterium DecisionPlus

• Stakeholder Values / Expert Surveys – USACE/EPA dredged material managers meetings (New Orleans 2004)– SRA/USACE/Contaminated Sediments Meeting (Palm Beach 2004)

#36

Conceptual Illustration of Disposal Alternatives

Landfill Upland CDF Nearshore CDF CAD Pit No-Action Island CDF

Water Line

In-place Sediment

Dredged Material

Effluent

Manufactured Liner

Dike Wall

Cap

Standard Landfill Waste

KEY:

In-place Soil

Kane Driscoll, S.B., W.T. Wickwire, J.J. Cura, D.J. Vorhees, C.L. Butler, D.W. Moore, T.S. Bridges. 2002. A comparative screening-level ecological and human health risk assessment for dredged material management alternatives in New York/New Jersey Harbor. International Journal of Human and Ecological Risk Assessment 8: 603-626.

Manufactured SoilCement Lock

#37

$ / Cubic Yard

Contaminated Sediment Management Decision

Impacted Area / Capacity

Cost Ecological Health

Human Health

Public Acceptance

# of complete ecological exposure pathways

Largest Ecological Hazard Quotient (HQ) calculated for

any one pathway

# of complete human

exposure pathways

Largest Cancer Risk calculated for any one pathway

Estimated Fish COC Concentration / Hazard Level

Decision Criteria: NY/NJ Harbor

Source: Kane Driscoll et al. (2002).

Source: NY/NJ Dredged Material Management Plan and Expert Opinion

#38

NY/NJ Harbor in Criterium DecisionPlus

Goal Criteria Sub-Criteria

#39

NY/NJ Harbor in Criterium DecisionPlus

Goal Criteria Alternatives

Hierarchy Rating Technique: Weights

Alternatives Rating Technique: SMART

with linear value functions

Sub-Criteria

#40

Criteria Levels for Each DM Alternative

  Cost Public Acceptability

Ecological Risk Human Health Risk

  

DM Alternatives

($/CY) Impacted Area/Capacity (acres / MCY)

Ecological Exposure Pathways

Magnitude of Ecological HQ

Human Exposure Pathways

Magnitude of Maximum

Cancer Risk

Estimated Fish

COC / Risk Level

CAD 5-29 4400 23 680 18 2.8 E -5 28

Island CDF 25-35 980 38 2100 24 9.2 E -5 92

Near-shore CDF 15-25 6500 38 900 24 3.8 E -5 38

Upland CDF 20-25 6500 38 900 24 3.8 E -5 38

Landfill 29-70 0 0 0 21 3.2 E –4 0

No Action 0-5 0 41 5200 12 2.2 E –4 220

Cement-Lock 54-75 0 14 0.00002 25 2.0 E -5 0

Manufactured Soil 54-60 750 18 8.7 22 1.0 E –3 0

Blue Text: Most Acceptable ValueRed Text: Least Acceptable Value

#41

USACE/EPA DM Managers Meeting: NY/NJ Harbor Weighting FormAttribute Swung from

Worst to bestConsequence to compare Rank

(1-9)Rate(0-

100)

Benchmark: Worst case on everything

Impacted Area/Capacity of Facility = 6500 (acres/ 106 cubic yards) Magnitude of Ecological Hazard Quotient – Maximum Exposure = 5200 Number of Complete Ecological Exposure Pathways = 41 Number of Complete Human Exposure Pathways = 25 Magnitude of Maximum Cancer Probability (Non-barge worker) = 1* 10-3 Ratio of Estimated Concentration of COCs in Fish to Risk-Based Concentrations = 220 Cost = 54-75 $/CY

9 0

Impacted Area/Capacity of Facility

Change from 6500 (acres/ 106 cubic yards) to 0 (acres/ 106 cubic yards)    

Magnitude of Ecological Hazard Quotient –Maximum Exposure

Change from 5200 to 0    

Number of Complete Ecological Exposure Pathways

Change from 41 to 0    

Number of Complete Human Exposure Pathways

Change from 25 to 12    

Magnitude of Maximum Cancer Probability (Non-barge worker)

Change from 1* 10-3 to 0.028 * 10-3    

Ratio of Estimated Concentration of COCs in Fish to Risk-Based Concentrations

Change from 220 to 0    

Cost Change from (54-75 $/CY) to (0-5 $/CY)    

#42

USACE/EPA Survey Results: Criteria Weights (%)

EPA USACE SRA

Public Acceptability 7.4 12.5 10.77

Ecological Health 35.6 27.1 32.45

Human Health 47.0 40.7 44.10

Cost 10.0 19.7 12.67

#43

Criteria Contributions to Decision ScoreUSACE weighting

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

Cost

Maximum Cancer Probability (Non-Barge Worker)

Ecological Hazard Quotient

Est. COC Conc in Fish / Risk-based Conc

Complete Human Health Exposure Pathways

Complete Ecological Exposure Pathways

Ratio of Impacted Area to Facility Capacity

EPA weighting

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

Cost

Maximum Cancer Probability (Non-Barge Worker)

Ecological Hazard Quotient

Est. COC Conc in Fish / Risk-based Conc

Complete Human Health Exposure Pathways

Complete Ecological Exposure Pathways

Ratio of Impacted Area to Facility Capacity

#44

Solution vs. MCDA Method:Does it Matter?

Software & Method

Alternatives (NY Case Study)

CAD Island CDF

Near-ShCDF

UplandCDF

Landfl Cemnt Lock

ManufSoil

NoAct

ExpertChoice,AHP

5 8 6 7 2 1 3 4

DecisionLab,PROMETHEE

3 8 5 6 2 1 4 7

CritDecPlus,SMART

2 7 4 5 1 3 6 8

#45

People:

Tools:

Process:

Environmental Assessment/Modeling (Risk/Ecological/Environmental Assessment and Simulation Models)

Decision Analysis (Group Decision Making Techniques/Decision Methodologies and Software)

Policy Decision Maker(s)

Scientists and Engineers

Stakeholders (Public, Business, Interest Groups)

Monitor System Response

Data Interpretation, Model Analysis and

Improvement

Implement Management

Strategy

Model Prediction and Management Plan Improvement

Determine Performance of

Alternatives on the Criteria

Identify Criteria to Compare Alternatives

Gather Value Judgments on Relative

Criteria Importance

Define Problem and Generate Alternatives

Risk Assessment, Adaptive Management and MCDAImplementation Framework

#46

Key Take-Away Points

Risk-Based MCDA offers planners:

• Reproducible and defensible management of complex multiple criteria

• A means to define and gauge what is important

• Balancing of expert opinion and stakeholder values

• Better responses, better reporting with opportunities to more clearly get it “out on the table”

#47

• MCDA workshop sites with posted lectures– http://www.risktrace.com/sediments http://www.risktrace.com/nato http://www.risk-trace.com/ports/index.php

• Papers– Yatsalo, B., Kiker, G., Kim, J., Bridges, T., Seager, T., Gardner, K., Satterstrom, K., Linkov, I.

2006. Application of Multi-Criteria Decision Analysis Tools for management of contaminated Sediments. Integrated Environmental Assessment and Management.

– Seager, T., Satterstrom, K., Linkov, I., Tuler, S., Kay, R. 2006. Typological Review of Environmental Performance Metrics (with Illustrative Examples for Oil Spill Response). Integrated Environmental Assessment and Management.

– Linkov, I., Satterstrom, K., Kiker, G., Bridges, T., Benjamin, S., Belluck, D. (2006). From Optimization to Adaptation: Shifting Paradigms in Environmental Management and Their Application to Remedial Decisions. Integrated Environmental Assessment & Management 2:92-98.

– Linkov, I., Satterstrom, K., Seager, T.P., Kiker, G., Bridges, T., D. Belluck, A. Meyer (2006). "Multi-Criteria Decision Analysis: Comprehensive Decision Analysis Tool for Risk Management of Contaminated Sediments". Risk Analysis 26:61-78.

– Linkov, I., Satterstrom, K., Kiker, Batchelor, C., G., Bridges, T.(2006). From Comparative Risk Assessment to Multi-Criteria Decision Analysis and Adaptive Management: Recent Developments and Applications. Environment International 32: 1072-1093.

References