Cost-benefit analysis of satellite observing systems

WMO

Dr. Lars Peter Riishojgaard, WIGOS Project Manager, WMO Secretariat

On the use of data assimilation to assess

the cost/benefit of meteorological satellites

EUMETSAT Meteorological Satellite Conference,

Side event at WMO, Geneva, September 23 2014

EUMETSAT Meteorological Satellite Conference, Geneva, 22-26 .9.2014

Question: Can we put numbers on the monetary

value of (for instance) satellite observations used for

weather forecasting?

• Economics of meteorological observations

• Money and weather forecasting

• Weather forecasting and observational data

• Impact of observational data

• Cost of satellite data

• Role of data assimilation and NWP

• Caveats

• Final remarks

2

Weather Prediction and the US

Economy; A Macroscopic View

• Department of Commerce: “20% of overall US economy is weather sensitive”: ~$3 trillion/year

– Impact to air and surface transportation, agriculture, construction, energy production and distribution, etc.

• Assume that half of this is “forecast sensitive”: $1.5 trillion/year

• Assume that the potential savings due to weather forecasting amount to 5% of the “forecast sensitive total”: ~$75B/year

(discussed during CBS TECO in Windhoek, 2010)

3

… a Macroscopic View … (II)

• Define “perfect forecast information” as NWP output with useful

skill at two weeks!

• 0 h useful forecast range => $0 in savings

• 336 h useful forecast (two weeks maximum predictability) range

=> $75B in savings

• Assume now that the savings are distributed linearly over the

achieved forecast range for the global NWP system:

– $75B/336h ~ $223B/hr

• This implies that the value to the United States economy of

weather observations, dissemination, forecast products and

services is >$220M per hour of forecast range per year !

(discussed during CBS TECO in Windhoek, 2010)

4


The global picture

• The amount of $75B/year is one estimate of the

magnitude of the total potential socioeconomic benefit

of weather prediction activities to the US economy

• Scaling exercise, using World Bank (2011) numbers:

• Annual GDP of United States: ~$15T

• Annual GDP of all nations combined: ~$70T

– Assuming on average (i) equal sensitivity to weather, and (ii)

equal potential benefits from ability to predict across all nations,

we get an estimated

$75B *($15T/$70T) = $350B as the total global potential

benefit of weather prediction activities (indicating a likely range of

$100B to $1T)

5

What does this have to do with satellite

data? (NWP illustration) There is a need for a global

coverage of observational data,

irrespective of target location of

forecast!

6


What is the cost of acquiring

satellite weather observations?

• Difficult to come up with accurate numbers

• Some agency budgets are publicly known, others are not

• Need to distinguish between systems developed for

research versus operational systems

• Even the latter provide value beyond “just” weather use of

data data, e.g. through impact on climate, air quality, other

environmental monitoring/prediction activities, value of

scientific research, technological advancements, national

prestige, …

• Overall annual expenditure on meteorological satellite data is

likely to be in $3-8B/year range

7

What does NWP and data assimilation

have to do with this?

Beyond a range of 12-24h, NWP output is the

foundation of all weather forecast activities

NWP has objective, quantitative metrics:

Well-defined prediction problem with a “right” answer

(and an infinity of wrong ones)

Well-defined measures for quality of output

Well-established methodologies for assigning merit to

individual observing systems

One can actually somewhat meaningfully define

“perfect forecast information” in an NWP context

8


Jung et al., WMO Impact Workshop in Sedona, May 2012

9

NWP tools for impact assessment

(I)

• OSEs (Observing System

Experiments) are based on

data denial (or addition)

• Impact focuses on the

medium to long range

• Results show the impact of

withdrawing (or adding)

certain data

• OSE results are absolute;

e.g. “observing system X

extends the useful forecast

range by N hours in the

NH”


Gelaro et al, Fifth WMO Impact Workshop, Sedona 2012

10

• FSO (Forecast Sensitivity to Observations) are based on the adjoint of the model/analysis system or an ensemble approach

• Approach focuses exclusively on the short (quasi-linear) range

• Results show the impact of observations in the presence of all other observations

• FSO measures of impact are relative (e.g. often expressed in percentages that add up 100, even for poor forecasts or poor system performance)

NWP tools for impact assessment (II)

Forecast impact experiment from Dec. 2010 to Jan. 2011

Impact Impact / Obs. number

WMO Workshop on the Impact of Various Observing Systems on NWP Sedona – 22-25 May 2012

Could we use this type of FSO information to

rank observing systems by impact per dollar?

19

Of course we can! Simply divide the impact by the

cost of running the system and come up with a third

“impact per dollar” bar chart!

11


Answer: Yes, we can in fact assign some

form of monetary value to (e.g.)

meteorological satellite data

• Armed with the following three pieces of information

– Cost of acquiring the observations (difficult but not impossible to

estimate),

– Overall economic benefits of meteorological products derived from

observations (crudely estimated, or properly analyzed by trained

economists; most difficult part of the exercise),

– Individual contribution of observing systems to NWP skill, e.g. as

measured by FSO diagnostics,

it is (almost frighteningly) easy to devise an NWP-based

cost/benefit ratio for individual elements of the GOS, such as

the satellite components; involves a number of assumptions,

but eminently feasible

12


Caveats

• We may be able to assign value, but should we?

– The respective contributions of the components of the GOS differ

between NWP systems, even at comparable levels of skill

– The contributions vary, depending on which other GOS

components are used in the experiments

– OSE results are expensive to acquire, and often inconclusive

– FSO impacts are relative; even if the overall forecast performance

is poor, some observing systems may stand out due to their large

share in a modest improvement

– The sum of the percentages of the contributions always add

up to 100, even with few observations and poor skill

13


Final remarks

• The economic impact of weather is relatively well understood

• In contrast, the economic impact of weather prediction is generally not well studied and documented

• The cost/benefit of meteorological observations are a subject of intense interest among program managers and decision makers

• The costs are incurred (and known) mostly at the regional levels, the impact is realized and assessed globally, and the benefits accrue locally

• Tendency to focus too much on NWP diagnostics due to their compelling nature

• Impact of radars, geostationary satellites etc. will be underestimated

• Further work on cost/benefit is needed; capability is emerging and decision makers demand this type of information

14

Science

Cost-benefit analysis of satellite observing systems