Global Sensitivity Analysis to support model calibration ... · to support model calibration,...

francesca.pianosi@bristol.ac.uk!!

Global Sensitivity Analysis to support model calibration, evaluation, uncertainty propagation and robust decision-making: a toolbox for access to methods and workflows!Francesca Pianosi and Thorsten Wagener Department of Civil EngineeringUniversity of Bristol!! NE/J017450/1

credible.bris.ac.uk

francesca.pianosi@bristol.ac.uk!!

Global Sensitivity Analysis is a set of statistical techniques that provide a structured approach to tackle several types of uncertaintyassociated with the development and application of numerical models!GSA is useful for: :: more efficient model calibration :: better understanding of model response:: prioritizing efforts for uncertainty reduction (e.g. acquisition of new data):: assessing robustness to modeling assumptions :: …!!

francesca.pianosi@bristol.ac.uk!!

NUMERICAL MODEL!

resolution!

parameters!

forcing inputs!

structure(equations)!

FC!LP!BETA!

time!

output!

pdf!

forcing inputs!

parameters!

interactions!

resolution!

Global Sensitivity Analysis investigates how the variation in the output of a numerical model can be attributed to variations of its input factors!!!

francesca.pianosi@bristol.ac.uk!!

Outline!

:: Key concepts underlying GSA techniques !:: Examples of GSA applications :: SAFE: a Matlab/Octave/R toolbox for GSA !

francesca.pianosi@bristol.ac.uk!!

Key concepts underlying GSA techniques!

francesca.pianosi@bristol.ac.uk!!

Steps of sampling-based GSA!

3 POST PROCESSING!

2 MODEL EVALUATION!

1 INPUT SAMPLING!

x1 x2 x3!

sens

itivi

ty!

!!

1 !!

0.5!!

0!

Sensitivity indices!

Latin-hypercubeQuasi-random sampling…!

FC!LP!BETA!output!

pdf!

francesca.pianosi@bristol.ac.uk!!

Steps of sampling-based GSA!

3 POST PROCESSING!

2 MODEL EVALUATION!

1 INPUT SAMPLING!

x1 x2 x3!

sens

itivi

ty!

!!

1 !!

0.5!!

0!

Sensitivity indices!

Sensitivity measured by: :: multiple-start derivatives/differences (e.g. Morris method, DELSA, …)!

:: correlation between inputs and outputs !

:: properties of input distributions after conditioning outputs (Monte Carlo filtering)!!

:: properties of output distribution after conditioning inputs (e.g. Sobol’ method, density-based methods, …)!

francesca.pianosi@bristol.ac.uk!!

Steps of sampling-based GSA!

3 POST PROCESSING!

2 MODEL EVALUATION!

1 INPUT SAMPLING!

x1 x2 x3!

sens

itivi

ty!

!!

1 !!

0.5!!

0!

Sensitivity indices!

!:: Computational cost of post-processingis negligible wrt to model evaluation !:: Required number of model evaluations grows with number of input factors !:: Growth rate differs from method to method!

francesca.pianosi@bristol.ac.uk!!

Choice of GSA method: classification system!

Correlation & Regression Analysis!

Variance- based !&!Density-based!

Multiple-starts derivatives!

Monte-Carlofiltering!

Num

ber o

f mod

el e

valu

atio

ns! >1

0 x

M!

>100

x M

!>1

000

x M!

PAWN!

FAST!

VBSA (Sobol’)!

EET (Morris)!

Regional Sensitivity Analysis!

CART!

Screening! Ranking! Mapping!Specific purpose!

M = number of input factors!

Pianosi et al. 2016 Env.Mod&Soft!

implemented in the SAFEToolbox!

francesca.pianosi@bristol.ac.uk!!

Examples!

francesca.pianosi@bristol.ac.uk!!

Supporting calibration of a land surface model!

!Which parameters mostly affect the model performance? Which parameters have littleinfluence and can be set todefault values?!

latent heat!

soil heat!

surface skin temperature!

soiltemperature!

sensibleheat!

water on canopy!

transpiration!

rainfall!

evaporation!

rainfall!

soilmoisture!

drainage!

evaporation!

runoff!

runoff!

ENERGY!WATER!

The Joint UK Land Environment Simulator (JULES)!

Application tothe Santa Rita creosotesite in the US!

with J. Iwema, R. Rosolem

francesca.pianosi@bristol.ac.uk!!

Supporting calibration of a land surface model!

1-9: parameters: b sathh satcon sm_sat sm_crit sm_wilt hcap hcon albsoil 10-12: initial conditions: tsar_tile sthuf t_soil!

0.2

0.4 1. Sensible heat

sens

itivity

2. Latent heat

1 2 3 4 5 6 7 8 9 10 11 12

0.2

0.4 3. Soil moisture (TDT)

sens

itivity

1 2 3 4 5 6 7 8 9 10 11 12

4. Soil moisture (CRNS)

Sensitivity of RMSE of different simulated variables to uncertain parameters and initial conditions!

with J. Iwema, R. Rosolem

francesca.pianosi@bristol.ac.uk!!

Supporting calibration of a land surface model!

0.2

0.4 1. Sensible heat

sens

itivity

2. Latent heat

1 2 3 4 5 6 7 8 9 10 11 12

0.2

0.4 3. Soil moisture (TDT)

sens

itivity

1 2 3 4 5 6 7 8 9 10 11 12

4. Soil moisture (CRNS)

Sensitivity of RMSE of different simulated variables to uncertain parameters and initial conditions!

with J. Iwema, R. Rosolem

1-9: parameters: b sathh satcon sm_sat sm_crit sm_wilt hcap hcon albsoil 10-12: initial conditions: tsar_tile sthuf t_soil!

francesca.pianosi@bristol.ac.uk!!

Supporting calibration of a land surface model!

samples that improve model performances wrt default set-up default set-up values!

with J. Iwema, R. Rosolem

parameters! initial conditions!

b sathh satcon sm−sat sm−crit sm−wilt hcap hcon albsoil tstar−tile sthuf t−soil

0.3

11.1

0.1

833.3

0

0.1

0.4

0.7

0

0.7

0

0.7

1000019.1

2999900.8

0.1

3

0.1

0.5

260

320

0

1

260

320

francesca.pianosi@bristol.ac.uk!!

Supporting calibration of a land surface model!

parameters! initial conditions!

b sathh satcon sm−sat sm−crit sm−wilt hcap hcon albsoil tstar−tile sthuf t−soil

0.3

11.1

0.1

833.3

0

0.1

0.4

0.7

0

0.7

0

0.7

1000019.1

2999900.8

0.1

3

0.1

0.5

260

320

0

1

260

320

with J. Iwema, R. Rosolem

samples that improve model performances wrt default set-up default set-up values!

francesca.pianosi@bristol.ac.uk!!

Investigating uncertainties in a flood inundation model!

!How important is the choiceof the model’s spatial resolution for flood simulations with respect to other uncertain factors? !

with J. Savage, P. Bates, J. Freer

LISFLOOD-FP model applied to Imera Basin, Italy!

francesca.pianosi@bristol.ac.uk!!

Investigating uncertainties in a flood inundation model!

Flood extent =percentage of wet cells (water depth > 0.10 m)!!!Uncertain input factors:!- spatial Resolution - Channel friction (parameter)- Floodplain friction (parameter)- Forcing Hydrograph (boundary condition)- DEM: Digital Elevation Model!!!

uncertainty in predicted flood extent!

time (hours)!

with J. Savage, P. Bates, J. Freer

francesca.pianosi@bristol.ac.uk!!

most important contributor to uncertainty (Sobol’)!

time (hours)!

Investigating uncertainties in a flood inundation model!

Flood extent =percentage of wet cells (water depth > 0.10 m)!!!Uncertain input factors:!- spatial Resolution - Channel friction (parameter)- Floodplain friction (parameter)- Forcing Hydrograph (boundary condition)- DEM: Digital Elevation Model!!!

uncertainty in predicted flood extent!

time (hours)!

with J. Savage, P. Bates, J. Freer

francesca.pianosi@bristol.ac.uk!!

Investigating uncertainties in a flood inundation model!

!!!!!!!Uncertain input factors:!spatial Resolution ! !Channel friction !Floodplain friction Forcing Hydrograph !DEM!!

with J. Savage, P. Bates, J. Freer

francesca.pianosi@bristol.ac.uk!!

Finding key drivers of slope failure!

What are the dominant drivers of landslides in a slope with properties known at different level of certainty?!!Slope properties (geometry&soil):!!!!!!Design-storm > deeply uncertain:!!!!!!

with S. Almeida and L. Holcombe

evaporation

runoff

rainfall

water table

slip circle

the Combined Hydrology And Slope Stability Model (CHASM)!www.chasm.info!

Slope angle (degrees)! [27,30]!

Thickness of top soil (m)! [2,6]!

etc. (25 in total)!

Rain

fall!

Time!

Duration!

Intensity!

francesca.pianosi@bristol.ac.uk!!

Finding key drivers of slope failure!

Results of CART analysis!

Cohesion/Thickness

top soil

Stable

>2.0 <2.0

>11 <11

>1.5 <1.5

>3 <3

>47 <47

Stable

>18 <18

Depth WT

>80 <80

Rainfall intensity

<5 >5

Stable

<3.2 >3.2

Stable

Thickness top soil

Stable

Fail

Fail

Rainfall duration

Cohesion/Thickness

top soil

Rainfall duration

Rainfall intensity

Stable >7.5 <7.5

Cohesion/Thickness

top soil

Fail Stable

Rainfall intensity

Fail

with S. Almeida and L. Holcombe

francesca.pianosi@bristol.ac.uk!!

Finding key drivers of slope failure!

Results of CART analysis!!:: the dominant drivers of landslides in this slope are:!1 cohesion of the top soil2 thickness of the top soil3 rainfall duration 4 rainfall intensity5 depth of water table!!

Cohesion/Thickness

top soil

Stable

>2.0 <2.0

>11 <11

>1.5 <1.5

>3 <3

>47 <47

Stable

>18 <18

Depth WT

>80 <80

Rainfall intensity

<5 >5

Stable

<3.2 >3.2

Stable

Thickness top soil

Stable

Fail

Fail

Rainfall duration

Cohesion/Thickness

top soil

Rainfall duration

Rainfall intensity

Stable >7.5 <7.5

Cohesion/Thickness

top soil

Fail Stable

Rainfall intensity

Fail

with S. Almeida and L. Holcombe

francesca.pianosi@bristol.ac.uk!!

Finding key drivers of slope failure!

Results of CART analysis!!:: the dominant drivers of landslides in this slope are:!1 cohesion of the top soil2 thickness of the top soil3 rainfall duration 4 rainfall intensity5 depth of water table!!:: thresholds for these drivers that would lead to slope failure!!

Cohesion/Thickness

top soil

Stable

>2.0 <2.0

>11 <11

>1.5 <1.5

>3 <3

>47 <47

Stable

>18 <18

Depth WT

>80 <80

Rainfall intensity

<5 >5

Stable

<3.2 >3.2

Stable

Thickness top soil

Stable

Fail

Fail

Rainfall duration

Cohesion/Thickness

top soil

Rainfall duration

Rainfall intensity

Stable >7.5 <7.5

Cohesion/Thickness

top soil

Fail Stable

Rainfall intensity

Fail

with S. Almeida and L. Holcombe

francesca.pianosi@bristol.ac.uk!!

SAFE: a Matlab/Octave/R toolbox for GSA!

francesca.pianosi@bristol.ac.uk!!

Characteristics of the SAFE Toolbox!

:: It works under Matlab/Octave (an R version is also available)!!:: flexible, modular structure

!> easy to integrate with models running outside matlab!:: tutorial scripts (workflows) to get started … more in our introductory paper on Env. Mod & Soft (2015)!

francesca.pianosi@bristol.ac.uk!!

Uptake in academia!

Freely available for non-commercial use since December, 2014www.bris.ac.uk/cabot/resources/safe-toolbox/!!Introductory paper published on Env. Mod & Soft in May, 2015!!About 300 academic users so far!!!About 300 academic users so far!

francesca.pianosi@bristol.ac.uk!!

Uptake in industry!

Download requests for a closed-code 3-months trial version by: - E.ON Energy (Uncertainty in yield prediction for wind farms)- Pfizer (Physiological Based Pharmacokinetic models)- EDF (Thermochemical Heat Storage)!Ongoing collaboration with:!- Risk Management Solutions

!> support calibration of rainfall-runoff models !> find key drivers of loss models

- Airbus!> uncertainty in aircraft design models

- JBA Trust !> support long-term investment plans for flood risk reduction!

!

francesca.pianosi@bristol.ac.uk!!

Conclusions!

Our group @UoB is very active in GSA research, contributing to both methodological advances and development of GSA tools!!The SAFE Toolbox is freely available for non-commercial use from: www.bris.ac.uk/cabot/resources/safe-toolbox/!