Institute for Environment and Sustainability1 DELTA Tool Questionnaire - Feedback and main points for discussion FAIRMODE 4 th Plenary and Working Group

Before the Questionnaire…

Feb 2011 - DELTA Tool available to all SG4 members

- Document “Content and User’s Guide”

May 2011 – DELTA Tool downloaded by 23 experts from 13 Member States

May 2011 – A Questionnaire sent to all SG4 members

Feb 2011 - DELTA Tool available to all SG4 members

- Document “Content and User’s Guide”

May 2011 – DELTA Tool downloaded by 23 experts from 13 Member States

May 2011 – A Questionnaire sent to all SG4 members

Institute for Environment and Sustainability 3

Aim of the questionnaire

To collect users opinions mainly on:

• The Template for Reporting Model Performance

• The appropriateness of the statistical indicators and diagram

• The installation and the usage of the DELTA Tool

In order to :

highlight important points needing discussion and agreement

Identify weaknesses inside DELTA and ways for improvement

To collect users opinions mainly on:

• The Template for Reporting Model Performance

• The appropriateness of the statistical indicators and diagram

• The installation and the usage of the DELTA Tool

In order to :

highlight important points needing discussion and agreement

Identify weaknesses inside DELTA and ways for improvement


Feedback received

June 2011 – Feedback by 10 experts from 9countries (AT, BE, IT, NL, PT, UK, SE,

IE, DK)

Models: CAMx, CHIMERE,FRAM, MATCH, AURORA, BELEUROSADMS, OVL, SMOGSTOP, OSPM

June 2011 – Feedback by 10 experts from 9countries (AT, BE, IT, NL, PT, UK, SE,

IE, DK)

Models: CAMx, CHIMERE,FRAM, MATCH, AURORA, BELEUROSADMS, OVL, SMOGSTOP, OSPM

Short presentations (max 10 min) by: Helena Martins (PT) David Carruthers (UK) Stefan Andersson (SE) Helge Olesen (DK) Mihaela Mircea, Guido Pirovano (IT)


Template for Reporting

In general

“The current format is fine, clear, summarizing the essential information about the model’s performance”

“Keep the template/ format as short as possible ( 1 page)”

In general

“The current format is fine, clear, summarizing the essential information about the model’s performance”

“Keep the template/ format as short as possible ( 1 page)”



Points of discussion:

1. What should performance criteria and goals depend upon ?

2. How to select the monitoring stations (representativeness) ?

3. Are the statistical indicators complete ? Are some of them redundant ?

4. Normalisation of the Target indicator


1. What should performance criteria and goals depend upon ?

2. How to select the monitoring stations (representativeness) ?

3. Are the statistical indicators complete ? Are some of them redundant ?

4. Normalisation of the Target indicator




4. Is the 90% concept for the statistical indicators acceptable ?

5. Are we excluding some type of models with the Target template ? Are additional Templates (e.g. annual averages) required and what should they include ?

6. How to make the Template more readable Colors, Titles, Legend ?


4. Is the 90% concept for the statistical indicators acceptable ?

5. Are we excluding some type of models with the Target template ? Are additional Templates (e.g. annual averages) required and what should they include ?

6. How to make the Template more readable Colors, Titles, Legend ?


The Tool in General

Benchmarking – not only the model but the entire system (including input data) ?

Extend the exploration mode options

Keep it restricted to the FAIRMODE community?

Benchmarking – not only the model but the entire system (including input data) ?

Extend the exploration mode options

Keep it restricted to the FAIRMODE community?


Points for discussion - 1a.

Pollutant specific - YES

Scale specific - ?

Should criteria for local scale be less stringent ?

Pollutant specific - YES

Scale specific - ?

Should criteria for local scale be less stringent ?

1. What should performance criteria & goals depend upon ?

Scale Stations

Local All urban (traffic…)

Urban Suburban & urban background

Regional Rural background


Points for discussion - 1b.

Time averaging specific ? - foreseen and linked to limit values

avg.

Geographically dependent ?

We propose Not for criteria, but for goals ?

Seasonally dependent ? We propose Not

Time averaging specific ? - foreseen and linked to limit values

avg.

Geographically dependent ?

We propose Not for criteria, but for goals ?

Seasonally dependent ? We propose Not

PM10 day 1 year

NO2 1h 1 year

O3 8h

Agreement on setting/updating perfromance crietra&goals through joint exercises


Points for discussion - 2.

How to define station representativeness ?

How to select in case of limited number of stations ?

How to select stations in case of data assimilation ?

To be discussed by SG1 & SG4 SG1 uploaded a document, based on replies of “request of information”

How to define station representativeness ?

How to select in case of limited number of stations ?

How to select stations in case of data assimilation ?

To be discussed by SG1 & SG4 SG1 uploaded a document, based on replies of “request of information”

2. How to select the monitoring stations ?



SG1 suggests three quantifiable descriptions of spatial representativeness, all of which will depend on the temporal scale required, e.g. hourly or annual means:

The area (distance) surrounding a monitoring site in which the concentration does not vary by more than a predefined value.

A correlation distance, similar to the ‘range’ used in variograms (suitable for data assimilation methods)

The variability of the concentration in a predefined area surrounding a monitoring site. The required area is likely

to correspond to the model resolution.

SG1 suggests three quantifiable descriptions of spatial representativeness, all of which will depend on the temporal scale required, e.g. hourly or annual means:

The area (distance) surrounding a monitoring site in which the concentration does not vary by more than a predefined value.

A correlation distance, similar to the ‘range’ used in variograms (suitable for data assimilation methods)

The variability of the concentration in a predefined area surrounding a monitoring site. The required area is likely

to correspond to the model resolution.

2. SG1 suggestions (…now discussing)

Bruce Denby Discussion Document SG1http://fairmode.ew.eea.europa.eu/monitoring-modelling-sg1



Cross-validation using ‘leave one out’ or other sampling methods (effective for kriging type applications, time consuming for 4Dvar or Kalman filters)

Splitting the dataset into an assimilation set and a validation set

Complication for urban and local scale applications: sufficient numbers of monitoring stations available to create a validation subset may not exist

Cross-validation using ‘leave one out’ or other sampling methods (effective for kriging type applications, time consuming for 4Dvar or Kalman filters)

Splitting the dataset into an assimilation set and a validation set

Complication for urban and local scale applications: sufficient numbers of monitoring stations available to create a validation subset may not exist

2. How to select the stations in case of data assimilation ? (SG1 – suggestions)

Bruce Denby Discussion Document SG1http://fairmode.ew.eea.europa.eu/monitoring-modelling-sg1


3. Are the statistical indicators complete ?

“Keeping the same statistical indicators and diagrams for all scales and pollutants is good” (most answers).

BUT 1. Some redundancy ?

( e.g. MFE, IOA)

2. Additional indicator – SigM/SigO 3. Use of median, percentiles (5 & 95)4. Adding all stations/statistics in

Exploration mode ?

“Keeping the same statistical indicators and diagrams for all scales and pollutants is good” (most answers).

BUT 1. Some redundancy ?

( e.g. MFE, IOA)

2. Additional indicator – SigM/SigO 3. Use of median, percentiles (5 & 95)4. Adding all stations/statistics in

Exploration mode ?


3. Are the Statistical indicators complete?

5. Should we have indicators showing how far are simulated limit/ thresholds in comparison to the AQD requirements ?

5. Should we have indicators showing how far are simulated limit/ thresholds in comparison to the AQD requirements ?


3. Are the Statistical indicators complete?

Additional statistical indicators suggested in the repliesAdditional statistical indicators suggested in the replies

• MAFE=Mean Absolute Factor Error

• Observed and computed mean• Observed and computed

quartiles• MB=Mean Bias• RMSE=root Mean Square Error• Sigma ratio• PPEA=Pair Peak Estimation

Accuracy• ASPEA=Average Station Peak

Estimation Accuracy• AOT40• SOMO35• Hitrate

• MAFE=Mean Absolute Factor Error

• Observed and computed mean• Observed and computed

quartiles• MB=Mean Bias• RMSE=root Mean Square Error• Sigma ratio• PPEA=Pair Peak Estimation

Accuracy• ASPEA=Average Station Peak

Estimation Accuracy• AOT40• SOMO35• Hitrate


3. Are the statistical indicators complete?

Target Bias/s CRMSE/s M/O SM/SO NMB MNB MFB FAC2 MFE IOA R RDE RPEOBS MOD OBS MOD

Milano_ 0.71 -0.19 0.68 54.32 47.25 0.87 36.63 25.33 0.69 0.13 0.07 0.06 0.84 0.36 0.81 0.73 0.04 0.35Limito_ 0.74 -0.10 0.73 49.93 46.85 0.94 31.38 23.85 0.76 0.06 0.12 0.01 0.86 0.36 0.80 0.69 0.01 0.36Milano_ 0.74 -0.14 0.72 49.87 46.08 0.92 27.70 24.76 0.89 0.08 0.02 0.08 0.87 0.34 0.83 0.71 0.03 0.13Castaga 0.78 -0.14 0.76 33.22 29.48 0.89 26.27 16.03 0.61 0.11 0.11 0.73 0.53 0.74 0.65 0.17 0.41Verona_ 0.78 -0.37 0.69 47.33 37.57 0.79 26.51 20.27 0.77 0.21 0.10 0.21 0.85 0.37 0.80 0.72 0.21 0.44VIMERCA 0.80 -0.21 0.77 44.35 38.83 0.88 25.93 20.10 0.78 0.13 0.01 0.12 0.86 0.37 0.77 0.65 0.07 0.33Treviso 0.80 -0.42 0.68 44.73 32.24 0.72 29.80 18.90 0.63 0.28 0.16 0.26 0.83 0.40 0.77 0.74 0.27 0.34VICENZA 0.81 -0.45 0.67 50.59 35.75 0.71 32.93 19.47 0.59 0.29 0.20 0.29 0.85 0.39 0.75 0.76 0.28 0.45Casteln 0.82 -0.36 0.73 44.84 34.36 0.77 29.08 16.30 0.56 0.23 0.07 0.17 0.86 0.37 0.72 0.69 0.16 0.45Venezia 0.82 -0.40 0.72 40.26 30.13 0.75 25.36 17.22 0.68 0.25 0.17 0.27 0.83 0.40 0.75 0.70 0.22 0.44EMEP_Is 0.83 -0.35 0.75 36.33 26.47 0.73 28.48 19.08 0.67 0.27 0.06 0.25 0.73 0.50 0.75 0.66 0.22 0.30Conegli 0.85 -0.22 0.82 35.28 29.84 0.85 24.62 21.02 0.85 0.15 0.06 0.14 0.76 0.46 0.76 0.62 0.13 0.23Arese 0.86 -0.31 0.80 55.89 43.67 0.78 39.39 23.20 0.59 0.22 0.03 0.17 0.79 0.42 0.69 0.60 0.18 0.53Cantu 0.87 -0.47 0.74 47.98 32.81 0.68 32.49 19.66 0.61 0.32 0.20 0.34 0.75 0.48 0.72 0.68 0.27 0.48Schio 0.88 -0.42 0.77 36.79 26.89 0.73 23.36 16.64 0.71 0.27 0.20 0.31 0.75 0.44 0.73 0.64 0.26 0.52

StdDevMean

Table with more statistics (without benchmarks) in Exploration modeTable with more statistics (without benchmarks) foreseen in Exploration mode


4. The 90% concept

In DELTACriteria for the statistical indicators apply for 90% of the

valid stations

(consistent with RDE/RPE)

From the replies:

“90% is a good choice but DELTA should include the directions regarding the minimum number of stations for model evaluation”

to be discussed in the session SG1&SG4

In DELTACriteria for the statistical indicators apply for 90% of the

valid stations

(consistent with RDE/RPE)

From the replies:

“90% is a good choice but DELTA should include the directions regarding the minimum number of stations for model evaluation”

to be discussed in the session SG1&SG4


5. Additional Templates ?

1. How to adapt the Template for annual averaged limit values (PM10) if :

hourly data are available (now in DELTA)

only one value for the yearly average available ( e.g. OVL model)

2. What to do with Exceedances, AOT40, SOMO35?

1. How to adapt the Template for annual averaged limit values (PM10) if :

hourly data are available (now in DELTA)

only one value for the yearly average available ( e.g. OVL model)

2. What to do with Exceedances, AOT40, SOMO35?


Details on the TARGET diagram – to be agreed

color symbols now note stations, is it more useful to have another statistical indicator, eg. R ?

replace the systematic – unsystematic division along x-axis by another indicator ( e.g. R <0.65 and R >0.65) ?

Make the Title application specific highlighting pollutant, goal of the model evaluation

Modify the Legend with the Target criteria and goal

Details on the TARGET diagram – to be agreed

color symbols now note stations, is it more useful to have another statistical indicator, eg. R ?

replace the systematic – unsystematic division along x-axis by another indicator ( e.g. R <0.65 and R >0.65) ?

Make the Title application specific highlighting pollutant, goal of the model evaluation

Modify the Legend with the Target criteria and goal

6. Make the Template more readable?


Installation :

generally no problems, WinVista, Win7, Linux (Ubuntu)

Instructions should be updated, consistent in the names

Raise the attention to the utilities programs

Other open source programs should be considered for the utilities programs

Installation :

generally no problems, WinVista, Win7, Linux (Ubuntu)

Instructions should be updated, consistent in the names

Raise the attention to the utilities programs

Other open source programs should be considered for the utilities programs

Installing and Using the Tool


Introduce one and the same format for monitored and observed data ?

Make possible the accommodation of wider range of formats

leap year treatment ( now not possible)

Introduce one and the same format for monitored and observed data ?

Make possible the accommodation of wider range of formats

leap year treatment ( now not possible)

Using the Tool – Data formats


make error messages more useful for debugging

display values of statistical indicators in connection with the diagram ( e.g. R and regression line formula in the scatter

improve titles and legends of the graphs, always display the name of the species

Lat min- lat max; lon min – lon max bounds could be added as a station selection criteria; the same could be done for station altitude

allow identification of the stations on the geo-map

add box whisker plot to compare mod vs. obs. Distributions (similar to quantile-quantile plots)

make error messages more useful for debugging

display values of statistical indicators in connection with the diagram ( e.g. R and regression line formula in the scatter

improve titles and legends of the graphs, always display the name of the species

Lat min- lat max; lon min – lon max bounds could be added as a station selection criteria; the same could be done for station altitude

allow identification of the stations on the geo-map

add box whisker plot to compare mod vs. obs. Distributions (similar to quantile-quantile plots)

Running DELTA

Suggested improvements - 1


Box-whisker could be used also to plot the distribution of different indicators (e.g. FB,FE, R etc..) for the same scenario; different scenarios for the same indicator (e.g. PM10 FB for different simulations…)

A cut-off threshold on observed data could introduced (species dependent) to skip pairs having unrealistically low observed values that can alter normalized indicator.

Multi-option explanation requires more clarification

Target Diagram needs an explaining example in the User’s Guide

Box-whisker could be used also to plot the distribution of different indicators (e.g. FB,FE, R etc..) for the same scenario; different scenarios for the same indicator (e.g. PM10 FB for different simulations…)

A cut-off threshold on observed data could introduced (species dependent) to skip pairs having unrealistically low observed values that can alter normalized indicator.

Multi-option explanation requires more clarification

Target Diagram needs an explaining example in the User’s Guide

Running DELTA

Suggested improvements - 2

Documents

Institute for Environment and Sustainability1 DELTA Tool Questionnaire - Feedback and main points for discussion FAIRMODE 4 th Plenary and Working Group