SPECTRAL AND MORPHING ENSEMBLE KALMAN FILTERS

AND APPLICATIONSJan Mandel, Jonathan D. Beezley, Loren Cobb, Ashok Krishnamurthy,

University of Colorado Denver

Adam K. KochanskiUniversity of Utah

Krystof Eben, Pavel Jurus, and Jaroslav ReslerCzech Academy of Sciences

31st International Symposium on Forecasting, Praha, June 29, 2011

ACKNOWLEDGEMENTS

This work was partially supported by the U.S. National Science Foundation grant AGS-0835579, National Institutes of Health grant 1RC1 LM01641-01, National Institute of Standards and Technology Fire Research grant 60NANB7D6144, and the Academy of Sciences of the Czech Republic project M100300904.

A part of this work was done during visits of Jan Mandel and Jonathan Beezley at the Department of Nonlinear Modeling, Institute for Computer Science, Czech Academy of Sciences. The hospitality of the Institute during those visits is gratefully acknowledged.

Simulation of FireFlux experiment (Clements et al., 2007) with WRF 3.3 and SFIRE 2011. Simulation setup by Adam Kochanski, visualization in VAPOR by Bedrich Sousedik. http://openwfm.org

EXAMPLE 1: WILDFIRE SIMULATION

EXAMPLE 2:AN EPIDEMIC IN

SWEDISTAN

• Pandemic influenza in a population with no immunity.

• Human Development Index set to HDI = 0.30 (similar to Congo).

• No air traffic, dirt roads between cities.

• Initial epidemic has two epicenters.

• Entire country has only one IDP camp for the ill and displaced (homeless).

• No public health efforts except near the IDP camp.

Author: Loren CobbFunding: Nat’l Defence College of Sweden and the US Joint Staff (J8 Directorate)Project: STRATMAS IIDate: 2001

EXAMPLE 3: RAIN FORECASTING

Project Medard, Department of Nonlinear Modeling, Institute for Informatics, Czech Academy of Sciences, http://www.medard-online.cz

WHAT DO THESE THREE PROBLEMS HAVE IN COMMON?

• Strongly nonlinear

• Solutions exhibit coherent, moving features: fire line, epidemic wave, weather front

• Data assimilation: ingesting data while the simulation is running

• standard Bayesian amplitude corrections often give nonphysical solutions

• position corrections needed in addition to amplitude

• Also, in wildfire and epidemic, statistical variability causes secondary fires and epidemics, which keep growing, do not dissipate

DATA ASSIMILATION

• The state-space model: the model state is a probability density p(u)

• Simulation updates the probability distribution by mapping it forward in time.

• Data given with an error estimate, as data likelihood p(u|d) : the probability density of the state u given data d

• Bayesian update of the state: panalysis(u) = const p(u|d)p(u)

• If all is Gaussian, in particular p(u|d) = const exp(-(d-Hu)TR-1(d-Hu)/2), we get new state mean Uanalysis = U + QHT(HQHT+R)-1(d-HU)

• This formula is also used in the non-Gaussian case anyway (also justified as least squares or maximum likelihood)

• Now the game is to estimate the state covariance Q

COVARIANCE ESTIMATION

• The classical Kalman filter evolves the state covariance Q assuming the model is linear.

• Optimal statistical interpolation (OSI) takes a human guess for the covariance Q

• Ensemble Kalman filter (EnKF) is a confluence of two techniques:

• Ensemble forecasting a.k.a. scenarios - the ensemble members are independent simulations: if it rains in 50% of the members we say that the probability of rain is 50%

• Estimation of the state covariance Q from the ensemble (e.g., by sample covariance)

• But this gives the EnKF analysis covariance too small. Fix:

• randomly perturb the data, or

• deterministically move of ensemble members to ensure correct sample covariance (adjustment EnKF, square root EnKF,...)

SOME OF THE CHALLENGES

• Large state vector (easily, in GB), large ensembles needed (often, 100s)

• Covariance estimates

• the state consists of several random fields on a physical domain: the covariance function between locations drops off with their distance (sample covariance with a small sample does not; it needs tapering)

• but the covariance function also changes with the location in the domain; it needs localization

• Non-gaussian probability distributions - the Holy Grail of data assimilation

• here, the locations of coherent features may have approximately gaussian distribution but the value of the field at a fixed location surely does not

SO WHAT IS NEW AND DIFFERENT IN THIS TALK?

• Optimal statistical interpolation by Fast Fourier Transform (FFT) - needs just a laptop instead of a supercomputer. Automatic tapering.

• Morphing - position corrections, not just amplitude, makes distributions close to gaussian.

• FFT EnKF - approximate the covariance by the diagonal of the sample covariance in the frequency space. Then, small ensembles are enough.

• Wavelet EnKF - provides also an automatic localization.

COVARIANCE ESTIMATION BY THE FFTAPPROXIMATION BY THE DIAGONAL IN THE FREQUENCY SPACE

Position correction by morphing EnKF

Data assimilation by FFT and wavelet transforms

Optimal statistical interpolation by FFT

Spectral EnKF by FFT and wavelets

Automatic tapering by FFT diagonal estimation

Given covariance Ensemble of 5 random functions

Sample covariance FFT estimation

From Mandel et al. (2010b)

Jan Mandel, Jonathan D. Beezley, and Loren Cobb Spectral and morphing ensemble Kalman filters

MORPHING

• Moving coherent features: need also position correction

•Replace the state by an extended state: deformation of a reference field + a residual

• by automatic registration: multiscale optimization, also related to advection field found in radar analysis

• run the EnKF on the extended states:

• probability distributions closer to gaussian

• recover ensemble members from the deformation and the residual fields

• basically, replace linear combinations by morphs:

• Intermediate states in the middle created automatically from a linear combination of deformation fields and residual fields

• tricky: the right kind of combination to avoid ghosting

OPTIMAL STATISTICAL INTERPOLATION BY FFT• Use negative power of Laplace operator for the covariance• Green’s function drops off from the diagonal, cheap FFT implementation• Combined with morphing, example: rain fields

Forecast Data

Standard analysis Analysis with morphing

Duplicatestormcenter

OPTIMAL STATISTICAL INTERPOLATIONWITH MORPHING AND MISSING RADAR DATA

Forecast OPERA radar data Analysis

The storm line direction as well as the overall precipitation intensityare interpolated between the forecast and the data

MORPHING ENKF FOR A WILDFIRE SIMULATION

Truth

Analysis - standard EnKF breaks down Analysis - morphing EnKF OK

Forecast

COVARIANCE ESTIMATION BY WAVELETS

Wavelets are basis functions localized both in space. We use orthogonal wavelets; wavelet transform of a function consists of the coefficients of its expansion in a wavelet basis.

Wavelet EnKF uses diagonal approximation of covariance:

• Apply the wavelet transform to ensemble members

• Compute the diagonal of the sample covariance in the wavelet space

• Apply the inverse wavelet transform

Diagonal approximation of covariance in frequency and wavelet space has been justified for weather fields in other data assimilation contexts (Deckmyn and Berre 2005,..)

An example of a wavelet: Coiflet 25

ORTHOGONAL WAVELET BASIS

Wavelet number

Wavelet value

Obtained by shift and argument scaling of a single function, the mother waveletOrganized in octaves (the argument scaling): multiresolution structure

FFT AND WAVELET COVARIANCE ESTIMATIONS, 2 VARIABLES

Position correction by morphing EnKF

Data assimilation by FFT and wavelet transforms

Optimal statistical interpolation by FFT

Spectral EnKF by FFT and wavelets

Covariance estimation, 2 variables

Covariance, sample of 1000 Variable 1, sample of 5 Variable 2, sample of 5

Covariance, sample of 5 FFT estimation, sample of 5 Wavelet estimation, sample of 5

Estimation by FFT results in a distribution that is homogeneous in space,

smearing the distribution across the domain. Wavelet estimation keeps the

spatial structure, while filtering out spurious long-distance correlations.

Jan Mandel, Jonathan D. Beezley, and Loren Cobb Spectral and morphing ensemble Kalman filters

DATA ASSIMILATION FOR EPIDEMIC SPREAD

• The standard S-I-R model (susceptible, infected, recovered) with a spatial diffusion added to model infection spread in space

• Key: the probability distribution of cases in an epidemic distribution is Poisson. Use the Poisson distribution, not gaussian, also to randomize the data in the EnKF, then all falls in place.

• Go back to the EnKF roots: statistical ensemble for the forecast and uncertainty estimation, but use the assumed covariance i.e., optimal statistical estimation

Bayesian Tracking using OSITime 10

Truth Forecast Analysis

Bayesian Tracking using OSITime 20

Truth Forecast Analysis

Bayesian Tracking using OSITime 30

Truth Forecast Analysis

Bayesian Tracking using OSITime 40

Truth Forecast Analysis

Bayesian Tracking using OSITime 50

Truth Forecast Analysis

CONCLUSION

• Morphing and spectral ensemble filters have the potential to improve data assimilation in problems with moving coherent features, such as firelines in a wildfire, epidemic waves, and rain fronts.

• Encouraging preliminary results for optimal statistical interpolation (basically, assumed covariance) and estimation from small ensembles (well under 10), compared to 100s for standard ensemble filters.

• Spectral filters provide automatic tapering (essentially, an estimation of the covariance distance), and localization (estimation how the covariance distance changes from place to place).

• In progress, and future plans:

• estimate the cross-covariances for the fire and the rain applications

• time series data at a point

• morphing for the epidemics

• operational data assimilations systems

THANK YOUFOR YOUR ATTENTION!

REFERENCES

• Clark, T. L., Coen, J. L., Latham, D., 2004: Description of a coupled atmosphere-fire model. International Journal of Wildland Fire, 13, 49-63

• Clements, C. B., Zhong, S., Goodrick, S., Li, J., Potter, B. E., Bian, X., Heilman, W. E., Charney, J. J., Perna, R., Jang, M., Lee, D., Patel, M., Street, S., and Aumann, G.: Observing the dynamics of wildland grass fires – FireFlux – A field validation experiment, Bull. Amer. Meteorol. Soc., 88, 1369–1382

• Deckmyn, A. and L. Berre, 2005: A wavelet approach to representing background error covariances in a limited-area model. Monthly Weather Review, 133, 1279–1294

• Nina Dobrinkova, Georgi Jordanov, and Jan Mandel, WRF-Fire Applied in Bulgaria, Numerical Methods and Applications, Ivan Dimov, Stefka Dimova, and Natalia Kolkovska, eds., vol. 6046 of Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, 2011, pp. 133-140.

• G. Evensen. Data assimilation: The ensemble Kalman filter. Springer, Berlin, 2007

• Georgi Jordanov, Jonathan D. Beezley, Nina Dobrinkova, Adam K. Kochanski, Jan Mandel, and Bedrich Sousedik, Simulation of the 2009 Harmanli fire, 8th International Conference on Large-Scale Scientific Computation, June 2011, Sozopol, Bulgaria. Lecture Notes in Computer Science, to appear. http://arxiv.org/abs/1106.4736

• Ashok Krishnamurthy, Loren Cobb, Jan Mandel, and Jonathan D. Beezley, Bayesian Tracking of Emerging Epidemics Using Optimal Statistical Interpolation, 2010 Joint Statistical Meetings, Vancouver, Canada, July 31- August 5, 2010, http://arxiv.org/abs/1009.4959

• Jonathan D. Beezley, Jan Mandel, and Loren Cobb, Wavelet Ensemble Kalman Filters, IEEE IDAACS Proceedings, 2011, accepted. http://arxiv.org/abs/1102.5554

•Jan Mandel, Jonathan D. Beezley, Loren Cobb, and Ashok Krishnamurthy, Data Driven Computing by the Morphing Fast Fourier Transform Ensemble Kalman Filter in Epidemic Spread Simulations, Procedia Computer Science, vol 1, 1215–23, 2010.

• Jan Mandel, Jonathan D. Beezley, Janice L. Coen, and Minjeong Kim, Data Assimilation for Wildland Fires: Ensemble Kalman filters in coupled atmosphere-surface models, IEEE Control Systems Magazine 29, Issue 3, June 2009, 47-65

• Jan Mandel, Jonathan D. Beezley, and Adam K. Kochanski, Coupled atmosphere-wildland fire modeling with WRF-Fire version 3.3, Geoscientific Model Development Discussions (GMDD) 4, 497-545, 2011

• Edward G. Patton and Janice L. Coen, WRF-Fire: A Coupled Atmosphere-Fire Module for WRF, Preprints of Joint MM5/Weather Research and Forecasting Model Users' Workshop, Boulder, CO, June 22-25, 2004, pp. 221-223, NCAR.