A protocol for model validation

A protocol for model validation

ACCENT/GLOREAM Workshop

Paris, France, 11-13 October 2006

Peter Builtjes, TNO-the Netherlands and FU-Berlin

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation2

I) Introduction

II) A first lay-out of a protocol

III) Testing of the protocol

IV) Discussion

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation3

I) Introduction

• Model validation/evaluation/testing: comparison between

calculated and observed concentrations/depositions

• Observed concentrations should be accurate (the instrument)

and spatial representative (in balance with the model grid)

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation4

Spatial representative

• Ask the expert, rural, Educated guess

• Analysis of concentration patterns in time

• More stations in one grid

• Field study with for example passive samplers

• Determination by modelling-data assimilation

• For regional scale modelling, using rural stations, +/- 20 % ?

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation5

Recent model intercomparison and validation studies: EUROTRAC, EMEP-review, EURO-DELTA

Recommendation:

• No model validation for just one model

• Combine model intercomparison with model validation

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation6

EURODELTA

PM2.5 concentrations over Europe, preliminary/confidential first results

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation7

Carlos Borrego, AIR4EU

Total Model Uncertainty =

Model uncertainty + Input data uncertainty + Variability

Recommended Quality Indicators:

• Correlation Coefficient

• Fractional Bias

• RMSE (RPE)

• NMSE

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation8

Model intercomparison and validation

• City Delta and Euro Delta: JRC-graphical tool

• EMEP model intercomparison: TNO-tool

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation9

Taylor diagram for mean Summer ozone

Fine-scale and coarse-scale models

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation10

Compared to EMEP-review

O3 DAY TIME SUMMER 1999 obs. mean mod. mean residue RMSE corr. σ-ratio*

EMEP_v1.0 83.36 85.39 19.15 24.78 0.45 0.66

MATCH_v1.0 83.36 95.78 20.49 25.61 0.57 0.56

LOTOS_v1.0 83.36 78.29 20.90 26.28 0.47 0.87

LOTOS-EUROS 83.36 80.16 18.91 24.36 0.57 0.98

SO4 Year 2001 obs. mean mod. mean residue RMSE corr. σ-ratio*

EMEP_v1.0 2.30 2.17 1.23 1.87 0.57 1.31

MATCH_v1.0 2.34 2.78 1.29 1.84 0.62 1.26

LOTOS_v1.0 2.34 3.33 1.92 2.96 0.46 1.77

LOTOS-EUROS 2.34 1.87 1.23 1.81 0.50 0.97

NO3 Year 2001 obs. mean mod. mean residue RMSE corr. σ-ratio*

EMEP_v1.0 3.33 3.74 2.03 2.98 0.61 1.42

MATCH_v1.0 3.33 2.88 1.46 2.04 0.61 0.83

LOTOS_v1.0 3.33 3.26 1.94 2.71 0.36 0.89

LOTOS-EUROS 3.33 3.63 1.84 2.52 0.57 1.11

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation11

In TNO:

Quick Scan based on EMEP review tool to test different

model versions of the LOTOS-EUROS model + input:

Required QA/QC

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation12

II) A first lay-out of a protocol

Based on discussions at the ACCENT Workshop on Model

Benchmarking and Quality Assurance

Thessaloniki, 29/30 May 2006

Items of a protocol:

a) Define the purpose of the model and of the validation

Which output should be validated?

Example: Hourly ozone or annual averaged Benzene

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation13

b) Identify the processes required in the model

Is aerosol chemistry required or not?

c) Define the horizontal and vertical resolution, and the time scale

of the output

d) Concerning the input data, decide which data should be fixed

Example: are emissions taken as they are given?

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation14

e) Concerning observations, decide about QA/QC and spatial representativity

d) Quality indicators should be defined, including a threshold below which the model performance will be considered as inadequate

For daily max O3, the correlation coefficient between calculated

and observed should be more than 0.5, based on previous studies

e) Sensitivity runs should be defined for key processes, or key input data

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation15

III) Testing of the protocol

An attempt

a) Purpose of the validation for O3 daily max Summer, over Europe

b) Processes required: Only gasphase chemistry, like EMEP,

RADM, CBM4

(box model validation needed, see Poppe 1996)

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation16

c) Horizontal resolution : 25 x 25 km2, or 50 x 50 km2

(for discussion!!)Vertical resolution: 20 layers upto the lower stratosphere,or 5 layers upto 5 km (for discussion!!)Time resolution: hourly, not for discussion

d) Fixed input data:• Anthropogenic and biogenic emissions• Meteorology, prognostic and diagnostic• Boundary conditions, MOZART/TM5, or Logan• Landuse data base

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation17

e) Observations: EMEP only, rural stationsUncertainty +/- 20 % ???

f) Quality indicators: Based on experience in Euro-Delta/EMEP review

g) Sensitivity runs: for example:• Biogenic emissions• Reactivity of anthropogenic VOC-emissions• Cloud cover• Dry deposition, also over sea

Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation18

IV) Discussion/Statements

Always combine model validation with model intercomparison

Make ensemble approach and data assimilation an integral part of model validation

Work towards toolkit : JRC + TNO-EMEP review

The proposed protocol - seems to work for daymax O3 - should be

tested in several projects and by several groups, and based on experience improved upon

And finally being “accepted”