Acknowledgements to:

ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 1

Data Assimilation Progress and Plans at ECMWF

Lars IsaksenHead of Data Assimilation Section, ECMWF

[email protected]

Acknowledgements to:

Massimo Bonavita, Elias Holm, Patricia de Rosnay, Joaquin Muñoz Sabater, Clément Albergel, Mike Fisher, Yannick Trémolet, Carla Cardinali, Deborah

Salmond, Drasko Vasiljevic, Tomas Kral, and Marta Janiskova


Data assimilation

progress and plans

at ECMWF

4D-VarEnsemble of Data

Assimilations (EDA)

Scalability Future plans

Surface analysesHybrid 4D-Var & EDA


Ensemble

Prediction

System

High-

Resolution

Forecasting

Ensemble of

Data

Assimilations

4D-Var Data

Assimilation

Inter-dependent analysis & forecasting system at ECMWF


ECMWF HPC systemsUntil 2012 IBM Power6 (18400 cores)

Recently upgraded to IBM Power7 (48800 cores)

Operational Forecast and 4D-Var assimilation configuration10-day T1279L91 Forecast (16 km horizontal grid)

12 hour 4D-Var T1279 outer loop T159/T255/T255 inner loop

Upgrade from 91 levels to 137 levels in June 2013

Operational Ensemble of Data Assimilations (EDA) 10 member 4D-Var T399 outer loop and T95/T159 inner loop Upgrade to 25

members in June 2013

50 member Ensemble Prediction System (ENS) at T639L62 15-day forecasts; 50 members; monthly forecasts twice weekly


The increased forecast skill at ECMWF during the last 30 years is primarily due to data assimilation progress

1980 2011

Improved forecasts

are primarily due to improved analyses


“Observation – model” values are computed at the observation time at high resolution: 16 km

4D-Var finds the 12-hour forecast that take account of the observations in a dynamically consistent way.

Based on a tangent linear and adjoint forecast models, used in the minimization process.

80,000,000 model variables (surface pressure, temperature, wind, specific humidity and ozone) are adjusted

A few Characteristics about the ECMWF 4D-Var• All observations within a 12-hour period (~12,000,000) are used

simultaneously in one global (iterative) estimation problem

9Z 12Z 15Z 18Z 21Z


Observations and the forecast model are very important parts of data assimilation!

But the talk today will focus on data assimilation methods, scalability issues, and other challenges


• In the case of physical processes, strong non-linearities or thresholds can occur in the presence of discontinuous/non-differentiable processes

Accuracy of Tangent-Linear and adjoint important: LINEARITY ISSUES

Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity

-12

-8

-4

-2

-1

-0.50.5

1

2

4

8

12Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 45 **u-velocity

-12

-8

-4

-2

-1

-0.50.5

1

2

4

8

12

Non-linear finite difference (FD)

TL integration u-wind increments fc t+12, ~700 hPa


• Regularizations help to remove the most important threshold processes in physical parametrizations which can effect the range of validity of the tangent linear approximation

Accuracy of Tangent-Linear and adjoint important: LINEARITY ISSUES

Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity

-12

-8

-4

-2

-1

-0.50.5

1

2

4

8

12

Non-linear finite difference (FD)

TL integration u-wind increments fc t+12, ~700 hPaThursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity

-12

-8

-4

-2

-1

-0.50.5

1

2

4

8

12


• comparisons of 4D-Var against the version without linearized physics included:

– positive impact on analysis and forecast

– reducing precipitation spin-up problem

-0.04-0.02

00.020.040.060.080.100.12

0 1 2 3 4 5 6 7 8 9 10 11Forecast Day

(a) NHem: 500hPa geopotential - Anomaly correlationN.HEM : 500 hPa geopotential

-0.020

0.020.040.060.080.100.12

0 1 2 3 4 5 6 7 8 9 10 11Forecast Day

(c) NHem: 700hPa rel.humidity - Anomaly correlationN.HEM : 700 hPa rel. humidity

-0.020

0.020.040.060.080.100.120.140.16

0 1 2 3 4 5 6 7 8 9 10 11Forecast Day

(n) Tropics:700hPa rel.humidity - Anomaly correlationTropics : 700 hPa rel.humidity

Anomaly correlation:

grey bars indicate significance

at 95% confidence level

July – September 2011

Impact of the linearized physics processes in 4D-VAR


Background error specification: Ensures that the background model fields are adjusted meteorologically consistently

Increments due to a singleobservation of geopotential height at 1000hPa at 60N with value 10m below the background.


Background error specification: The Balance Operator ensures the height and wind field balance is retained

Increments for a single observation of geopotential height at 1000hPa.

wind increments at 300hPa wind increments 150 metre above surface

Thanks to John Derber for developing this scheme during a 1 year stay at ECMWF in 1994


ECMWF has got an advanced static background error formulation that has been gradually improved over the last 20 years.

But it has only a weakly flow-dependent error specification

Isotropic analysis increments

Analysis increments with omega balance, non-linear balance,

wavelet formulation

Flow-dependency is now being provided by an Ensemble of Data Assimilation


In the ECMWF 4D-Var, the B matrix is defined implicitly in terms of a transformation from the background departure (x-xb) to a control variable χ:

(x-xb) = Lχ

So that the implied B=LLT.

In the current wavelet formulation (Fisher, 2003), the variable transform can be written as:

T is the balance operatorΣb is the gridpoint variance of background errors

Cj(λ,φ) is the vertical covariance matrix for wavelet index j

ψj are the set of radial basis function that define the wavelet transform.

How to introduce flow-dependent background errors

jjj

jbb ,2/12/11 CΣTxx


Cj(λ,φ) are full vertical covariance matrices, function of (λ,φ). Theydetermine both the horizontal and vertical background error correlation structures;

In standard 4D-Var T and Cj are computed off-line using a climatology of EDA perturbations. Σb is computed by random sampling of the static B matrix (randomization procedure, Fisher and Courtier, 1995)

How do we make this error covariance model flow-dependent?

We look for flow-dependent EDA estimates of Σb and Cj(λ,φ)

jjj

jbb ,2/12/11 CΣTxx

How to introduce flow-dependent background errors


Ensemble of data assimilations (EDA)

10 members; 2 inner-loop 4D-Var at T95/159; T399 outer loop; L91

Perturbed observations and SST; Stochastically perturbed physics


The benefit of an Ensemble of Data AssimilationsIn general to estimate analysis uncertaintyTo improve the initial perturbations in the Ensemble Prediction (implemented June 2010)To calculate static and seasonal background error statisticsTo estimate flow-dependent background error variances in 4D-Var - “errors-of-

the-day” (implemented May 2011)To improve QC decisions and improve the use of observations in 4D-Var (implemented

May 2011)To update the static background-error covariance statistics based on the latest EDA

(implemented June 2012) To estimate flow-dependent background error variances for unbalanced variables (June

2013)For online estimation of background error covariances (June 2013)


How is the Ensemble of Data Assimilations (EDA) used to provide flow-dependent background error estimates

Run an ensemble of , say 10, 4D-Var analyses with perturbed observations, Sea Surface Temperature fields and model physics.

Form differences between pairs of analyses (and short-range forecast) fields.These differences will have the statistical characteristics of analysis (and short-range

forecast) error.

L

40°N 40°N

50°N50°N

60°N 60°N

20°W

20°W 0°

0°

Model Level 58 **Temperature - Ensemble member number 1 of 11Thursday 21 September 2006 06UTC ECMWF EPS Perturbed Forecast t+3 VT: Thursday 21 September 2006 09UTC

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

Yellow shading where the short-range forecast is uncertain: give observations more weight in these regions.


Raw Ensemble StDev

Vorticity level 64

Filtered Ensemble StDev

Vorticity level 64

Sampling noise from the 10 member EDA needs to be filtered


• We performs an online calibration (Ensemble Variance Calibration; Kolczynsky et al., 2009, 2011; Bonavita et al., 2011)

• Calibration factors depend on latitude bands and parameter • Calibration factors also depend on the size of the expected error

EDA needs to be calibrated to become statistically consistent

Slide 21

Slide 21

The 4D-VAR&EDA hybrid implementation at ECMWF

Variance post-process

xa+εia

Analysis ForecastSST+εi

SST

y+εio

xb+εib

xf+εif

i=1,2,…,10

EDA Cycle

εif raw

variancesVariance

RecalibrationVariance Filtering

EDA scaled variances

4DVar Cycle

xa

Analysis ForecastEDA scaled Varxb xb


EDA based background error variance for Surface pressure

In May 2011 ECMWF implemented EDA based flow-dependent background error variance in 4D-Var - our first hybrid DA system

The 10-member EDA has been used to estimate the background error variance in the deterministic 4D-Var.

This is the first step towards the implementation of a fully flow-dependent representation of background error covariance.

hPa

Hurricane Fanele, 20 January 2009


Impact on high-resolution

forecast skill

Geopotential height normalized

forecast error differences

experiment-control

11 Jan – 30 Mar 2010

Impact of EDA based variances in hybrid 4D-Var


To update the static background-error covariance statistics based on the latest EDA (implemented June 2012)

Resolution upgrades and more observations since last update resulted in sharper structure functions: reduced correlation length scales both horizontally and vertically


Static background-error covariance statistics (for Jb) updated, based on latest EDA (implemented June 2012)


forecast skill

Vector wind normalized forecast

error differences experiment-

control

8 June – 7 July 2011

Slide 26

Slide 26

Based on two extended 50 member EDA experiments, we can apply a more direct strategy:1. Under the assumption of sampling noise as a random process

2. Time average sampling noise spectrum samples

3. Compute raw filters and time average to smooth out noise

jie SSPnSP 21

raw

eSP

SPn

1

1

Improved statistical noise filtering of EDA variances (implemented June 2012)


Improved statistical noise filtering of EDA variances (implemented June 2012)


forecast skill

Vector wind normalized forecast

error differences experiment-control

10 January to 29 April 2011


EDA-based flow-dependent background errors for unbalanced control vector variables (Tu,Du,LNSPu) - June 2013

90 N 90 Ssurface

topAverage unbalanced temperature (st.dev. in Kelvin)


Impact of EDA-based unbalanced control vectorReduction in Geopotential RMSE (95% confidence, RAOBs)

NH SH

200 hPa

1000 hPa

500 hPa

Based on 90 days in 2012; T511; CY38r1


a) B is computed in a post-processing step of a 25 member EDA

b) EDA perturbations from the past 12 days are used, with an exponential decay factor

c) Updated B is used in HRES 4D-Var

d) EDA still uses static error variances and B: fully interactive (EnKF type) system will be tested next

EDA based flow-dependent online update of background error covariances (B) – also planned for June 2013


Impact of online BReduction in Geopotential RMSE - 95% confidence

NH SH50 hPa

100 hPa

200 hPa

500 hPa

1000 hPa

Period: Feb - June 2012

T511L91, 3 Outer Loops (T159/T255/T255)

Verified against operational analysis


Further 4D-Var related upgrades (planned for Dec 2013)24-hour 4D-Var with over-lapping window

Run twice daily, but using observations for the last 24 hours

Increase of inner loop resolutionMost likely from T159/T255/T255 to T255/T255/T399

Increased and improved use of conventional observationsRetuning of observation errors for both conventional and satellite data


Land Surface Analysis

- Snow depth analysis• New 2D Optimum Interpolation (OI) (operational)

• Ground data (SYNOP and other NRT data)• High resolution NESDIS/IMS snow cover data

- Soil Moisture analysis

• Extended Kalman Filter (EKF) (Operational)

• Uses screen level parameters analysis

- Satellite data related to Soil Moisture

METOP-ASCAT and SMOS Monitoring operational

ASCAT data assimilation (operational late 2013)

- Validation activities (EUMETSAT H-SAF)

ASCAT

NESDIS/IMS snow cover

(16 Jan. 2012)

SMOSECMWF

January 2010

High quality surface and near surface weather products


Use of SYNOP and National Network data

National networks data:

GTS: Sweden (>300), Romania(78), The Netherlands (33), Denmark (43), Finland (183)

FTP: Hungary (61)

SYNOP 06 UTC January 23 2013 National snow data


ASCAT soil moisture productAdvanced Scatterometer on MetOP A/B (launched in 2006/2012)

Active microwave instruments operating at C-band

ASCAT operational EUMETSAT soil moisture product

Soil Moisture Monitoring

Dec 2011-Jan 2012


ESA Earth Explorer mission ; RD developments:SMOS: Soil Moisture and Ocean Salinity

Global assimilation of SMOS brightness temperatures in the ECMWF Simplified

Extended Kalman Filter in research mode

+250

-250

Ave

rage

d so

il m

oist

ure

prod

uct,

June

201

0 ( m

3 m-3)

Sensitivity of TB to soil

moisture (K/0.01 m3m-3)

- Soil moisture from SMOS is expected operational in 2014/2015

- Future SMAP (NASA) in 2015

0

-250


Aircraft temperature bias correction at ECMWF

Based on the variational bias correction scheme developed at ECMWF

Each aircraft is bias corrected individually using three predictors

First predictor: constant temperature correction at cruise level

Second/third predictors: function of the vertical aircraft velocity (dp/dt) to account for ascend/descend bias conditions

It works: The aircraft departures biases and standard deviations are reduced; RAOB biases are reduced too

Implemented in operations November 2011


EUCOS E-SAT

Improved o-b and o-a fit to aircraft temperature data

Aircraft temperature bias correction

Black lines with aircraft bias correction applied, red curves without aircraft bias correction


Bias correction results in reduced temperature biases for RAOB data and for GPS-RO data

Aircraft temperature bias correction

Slide 40 © ECMWF

Observation information content metrics used for wind observation impact evaluation

•Analysis equation:Xb background, Xa analysis, y observations

K is Kalman gain matrix, H observation operator

•OI (Observation Influence) and DFS (Degrees of Freedom for Signal) –The impact of observations to the analysis

•FSO (Forecast Sensitivity to Observations; FEC: Forecast Error Contribution)

–Contribution of observations to the reduction of (24h) forecast error

TTaOI HKy

Hx

ba

feTfe

JJ Hxy

xK

Slide 41 © ECMWF

Wind DFS and FSO per datum as function of altitude

The wind information is most important for the analysis at 50-100 hPa, and for the forecast at 100-200 hPa


Data Assimilation on future computer architectures

Scalability is an important issue

How will the future computer architectures look?

Will we be able to use future parallel computers efficiently for Data Assimilation?

Can we modify our Data Assimilation methods to utilize future computer architectures better?

How scalable is ECMWF’s 4D-Var on todays computer architectures?


ECMWF sustained historic computer performanceAn increase by a factor of 10,000,000 in 30 years

1979 2013

Increase is primarily due to more cores (1 to 50000 in 30 years)

Future increase in performance will almost certainly come from more cores

ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 44Lars Isaksen Annual Seminar, ECMWF, 2011 Slide 44

Scalability of T1279 Forecast and 4D-Var

User Threads on IBM Power6

Speed-up

Operations48 Nodes




Speed-up

Operations48 Nodes

Traj_1 & Traj_2

Traj_0




Speed-up

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2000 3000 4000 5000 6000

10-day Forecast4D-VarIdeal

Min_0

Min_1 & Min_2

Operations48 Nodes

Min_0 has 36000 grid columns

FC model has 2000000 grid columns

Min_1&2 have 89000 grid columns


Continuous Observation Processing Environment (COPE) • Shortens the time critical path by performing observation pre-processing

and screening as data arrive• Improve scalability by removing most observation related tasks from time

critical path• Reduce risk of failures in the analysis during the time critical path • Enables near real-time quality control and monitoring of observations• More modular software• A hub Observation Data Base (ODB) will be central to this approach


Object-Oriented Prediction System – The OOPS project

Data Assimilation algorithms manipulate a limited number of entities (objects):

x (State), y (Observation),

H (Observation operator), M (Model), H*& M*(Adjoints),

B & R (Covariance matrices), etc.

To enable development of new data assimilation algorithms in IFS, these objects should be easily available & re-usable

More Scalable Data Assimilation

Cleaner, more Modular IFS


OOPS More Scalable Data Assimilation

• One execution instead of many will reduce start-up - also I/O between steps will not be necessary

• New more parallel minimisation schemes - Saddle-point formulation

• For long-window, weak-constraint 4D-Var: Minimization steps for different sub-windows can run in parallel as part of same execution with the saddle-point formulation

ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 50Slide

50


Summary of ECMWF’s Data Assimilation strategy • Hybrid DA system: Use EDA information in 4D-Var

Flow-dependent background error variances and covariances, and model error in 4D-Var

Provides improved uncertainty estimation• Long-window weak-constraint 4D-Var based on saddle-point method

• Unified Ensemble of Data Assimilations (EDA) and Ensemble Prediction SystemFor estimation of analysis and short range forecast uncertainty that will benefit the deterministic 4D-VarFor estimation of long range forecast uncertainty (the present role of the EPS)

Note: The EDA is a ‘stochastic EnKF’ with an expensive 4D-Var component. It may be replaced or supplemented by an EnKF system in the future, if beneficial.


Thank you for attending my seminar.Any questions?

Documents

Acknowledgements to: