© Crown copyright Met Office Pass the baton: Verification and the NCOF product chain Andy Saulter, Business Support and Waves

© Crown copyright Met Office

Pass the baton: Verification and the NCOF product chainAndy Saulter, Business Support and Waves


Why verify? Why pass the baton?• Two purposes to scientific verification of our

product set..

1. To prove to ourselves and our peers that the research and development that we do is scientifically valid, and an improvement on the status quo

2. To quantify the risk taken by those who use what we create in their work and decision making


Why verify? Why pass the baton?2. To quantify the risk taken by those who use what we

create in their work and decision making

‘To face the sea is, to be sure, no light matter when the sea is in its grandest mood. You must the know the sea, and know that you know it, and not forget that it was made to be sailed over’ – Joshua Slocum (1st solo circumnavigator – 1895-1898)

This talk will concentrate on this second use of our verification

• Developing and communicating the necessary metrics to fulfil this function is not always something we do well

• However, supplying unbiased statements of risk alongside our products should be a key element both within an NCOF product development chain and to the end user – we must clearly communicate risk when we ‘pass the baton’


Risk in decision making – breakdown of an end-user decision

I want to go surfing

Apply cost benefit model

High Benefit vs Cost

Low requirement for risk information

Low Benefit vs Cost

Nil requirement for risk information

Marginal Scenario

High requirement in close analysis of forecast + risk

Apply if at marginal

thresholds for beach / board

Need significant confidence in a

positive outcome based on my

forecast


Risk in decision making – mitigation in the field, ‘alpha factor’• Alpha is employed by Det Norske Veritas, a

major international classification firm – responsible for certifying and assuring offshore operations worldwide

• Alpha factors are applied to weather forecasts for defined working thresholds (often significant wave height) in order to assure that only a low percentage of forecasts would ‘miss’ an event where the threshold is exceeded


Risk in decision making – mitigation in the field, ‘alpha factor’

y = 1.214x

0

0.5

1

1.5

2

2.5

0 0.5 1 1.5 2 2.5

Forecast Hs (m)

Ob

se

rve

d H

s (

m)

MISS HIT

FALSE ALARM

NULL

Alpha = 1


Risk in decision making – mitigation in the field, ‘alpha factor’

y = 1.214x

0

0.5

1

1.5

2

2.5

0 0.5 1 1.5 2 2.5

Forecast Hs (m)

Ob

se

rve

d H

s (

m)

MISS HIT

FALSE ALARM

NULL

Alpha = 0.8


Risk in decision making – mitigation in the field, ‘alpha factor’• Alpha is employed by Det Norske Veritas, a

major international classification firm – responsible for certifying and assuring offshore operations worldwide

• Alpha factors are applied to weather forecasts for defined working thresholds (often significant wave height) in order to assure that only a low percentage of forecasts would ‘miss’ an event where the threshold is exceeded

• The ‘cost’ of a conservative alpha are excessive numbers of false alarms and increased deployment times for construction vessels – often these ships and rigs are hired at rates of £10ks per day


Verification and the NCOF product chain

PROCESS MODEL

CASE STUDIES

OPERATIONAL MODEL (TESTING)

OPERATIONAL MODEL

PRODUCTS

END USER



PROCESS MODEL

CASE STUDIES

OPERATIONAL MODEL (TESTING)

OPERATIONAL MODEL

PRODUCTS

END USER

Potential knock on effects for wider model

Everyday scenarios may not be covered; domain/run time compromises

Performance in new scenarios – how does test period relate to wider climatology?

Performance changes versus existing system; under forecast forcing?

Aligning performance measure(s) to decision risk



Some ground rules for ‘passing the baton’..

• Assume that familiarity with the science will decrease through the product chain – but that an interested end-user will expect a complete ‘audit trail’

• Verification metadata must be clear through all stages

• Know your ‘customer’ (immediate next step in the chain) and provide information on potential negatives as well as positives

• Geo-referencing verification information is vital

• Regions for area based verification must be appropriate


Geo-referencing verification information

• ‘Whole domain metrics’ are not very often the most appropriate for real world applications



Questions to ask ourselves..

• What happens next with my work?

• What are the key issues, parameters, time and space scales that interest the next stage?

• Do my verification metrics assess risk for (any or all of) these?

• Are the metrics and their outcomes explained clearly? Are gaps identified?


Theory into practise – proposed verification for Met Office operational wave forecasting

• Metrics have an identified purpose



• Metrics have anidentifiedpurpose

General Model Performance Statistics

Key performance indicators that define the model’s overall performance for broadscale domains and provide a simple baseline for measuring improvement.

Model Intercomparison Statistics

Agreed indicators from data exchanges that allow headline comparison of different centre’s models

Process Study Statistics Indicators of specific model performance characteristics (e.g. swell arrival time, wind-sea growth)

Model vs Remote Sensing vs In-Situ Statistics

3-way comparison to test a) model and remote sensing errors vs an in-situ benchmark, b) allow estimates or model error with global coverage.These analyses should provide a pre-cursor to the introduction of global post processing or data assimilation methodologies.

Forecaster Guidance Statistics

Regional analyses aimed at providing forecasters with ‘rules of thumb’ in interpreting wave model performance for given locations.

Customer Focused

Verification

Regional or site based analyses providing customers with an easily interpreted guide to forecast performance and assessment of risk associated with taking forecast data.These statistics should also provide a comparison between model and human intervened forecast performance.




• Customer focused verification should be provided covering a number of key strategic areas for wave forecasting (e.g. 4 or 5 regions covering the North Sea)


DNV Alpha metrics – study locations




• Customer focused verification should be provided covering a number of key strategic areas for wave forecasting (e.g. 4 or 5 regions covering the North Sea)

• A regular update cycle and method of publication must exist in order to communicate the results to those needing the information


Theory into practise – referencing skill assessment to customer questions

Output Category Usage Customer Question

QualitativeClimatological

Indicates climatological processdifferences in area, could be used

toplan a monitoring programme

[Planning] Where should I monitor inorder to understand / warn for my areaof interest?

QuantitativeClimatological

Provides broadscale quantificationof climate variations over an area

[Planning] How do regions of my areacompare in terms of operability orresource?

Qualitative Forecast Indicates events in the broadscale;could be used to describe howevents relate to a referenceclimatology

[Response] Should I monitordevelopments in a certain region of myarea more closely in the future?

Quantitative Forecast Warns of broadscale events andallows quantification of risk relativeto an operating threshold

[Response] Can I quantify risks /opportunities for my operations inregions of my area in future?

Area based information


Thank you for listening

• You are here because you are driving NCOF product development

• Have a beer, have a chat, get someone else’s perspective..

• Think about how best to pass the baton…

Documents

© Crown copyright Met Office Pass the baton: Verification and the NCOF product chain Andy Saulter, Business Support and Waves