Download pdf - RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

Transcript
Page 1: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

RNN/LSTM Data Assimilation for the Lorenz Chaotic Models

Milt Halem, Harsh Vashistha University of Maryland, Baltimore County

(Tukey 1962): “ An approximate solution to the right problem is worth more than a precise solution to the wrong problem”.”

[email protected] Nov. 28, 2018 GSFC AI Workshop

mailto:[email protected]
Page 2: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

We employ Lorenz ‘63 (‘96) model as DA pilothttps://drive.google.com/open?id=1JaBndNTAmFyHIfvaaZYP1EOgaFceYUeH

https://drive.google.com/open?id=1JaBndNTAmFyHIfvaaZYP1EOgaFceYUeH
Page 3: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

OSSE Performance Evaluation of a Machine Learning

Data Assimilation Scheme

• For generating nature’s true evolving states, we use the Lorenz ‘63 ( ‘96) model with a

Runge-Kutta integration scheme.

• To simulate an imprecise model approximation to nature, we degrade the nature run

with larger time steps and random noise added to the model equations.

• The observing system is simulated by selecting arbitrary portions of data from the true nature model at fixed intervals of time with added noise representing observational

errors.

• Ensemble predictions are made from the approximate model by perturbing initial conditions and adding different errors to represent different parameterizations.

• We train the LSTM to simulate an Ensemble Kalman Filter (EnKF) data assimilation

approach using the DART framework.

• Validation of LSTM data assimilation is compared with nature simulation as well as EnKF

simulations.

Page 4: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

Long Short-Term Memory (LSTM) Networks• LSTM networks are a type of Recurrent Neural Network

- Selectively updates their internal state

- Effectively represents temporal data

- Avoids vanishing or exploding gradient problems

• LSTM consist of multiple gating mechanisms to control its behavior based on the internal Cell State. Intuitively, these mechanism are:

1. Deciding what to forget

2. Deciding what to take from input

3. Making updates

4. Generating outputs

Page 5: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

Results

Lorenz-63 (3 variables) Lorenz-96 (40 Variables)

Weighted RMSE (Training) 0.37073 1.29406

Weighted RMSE (Validation) 0.37318 1.31864

Weighted RMSE (Testing) 0.26642 0.92281

Training and Testing Scores for Lorenz-63 and Lorenz-96 models

Page 6: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

ResultsA snapshot of lorenz-63 model (with errors) forecast and data assimilation cycle using LSTM network

• Figure on the bottom is absolute difference between state and truth for each state

• Figure on the right captures the forecast and DA cycle of the model. The red dot are the start of DA cycle and the end of its tail marks the cycle’s end.Note: Observations does not exist for a section of the run.

Page 7: RNN/LSTM Data Assimilation for the Lorenz Chaotic Models · RNN/LSTM Data Assimilation for the Lorenz Chaotic Models Milt Halem, Harsh Vashistha University of Maryland, Baltimore

carta.umbc.edu

Conclusion• LSTM networks can be used for Data Assimilation in Chaotic models.

• The network can learn to estimate the process of an Ensemble Kalman Filter,

-- Largely possible due to different activation functions on intentionally designated branch for mean and variance estimation.

• Can be used for different configurations of the same model.

• Thank You


Recommended

Advanced Prediction ModelsA Special RNN: LSTM • The gap between the relevant information and the point where it is needed can become unbounded • Empirical observation: Vanilla
Advanced Prediction ModelsA Special RNN: LSTM • The gap between the relevant information and the point where it is needed can become unbounded • Empirical observation: Vanilla Documents
RNN & LSTM
RNN & LSTM Software
Situation Assessment for Planning Lane Changes ...Neural Network (RNN) with Long Short-Term Memory (LSTM) units. We further extend this to a Bidirectional RNN. As Bidirectional RNNs
Situation Assessment for Planning Lane Changes ...Neural Network (RNN) with Long Short-Term Memory (LSTM) units. We further extend this to a Bidirectional RNN. As Bidirectional RNNs Documents
NILC-SWORNEMO at the Surface Realization Shared Task ... · work (RNN) that we used was the Long Short-Term Memory (LSTM). We used a Bidirectional LSTM (Bi-LSTM) in the Encoder because
NILC-SWORNEMO at the Surface Realization Shared Task ... · work (RNN) that we used was the Long Short-Term Memory (LSTM). We used a Bidirectional LSTM (Bi-LSTM) in the Encoder because Documents
Reinforcement Learning: A Brief Introduction · 2012: Distributed deep learning (e.g., Google Brain) 2013: DQN for deep reinforcement learning 1997: LSTM-RNN 2015: Open source tools:
Reinforcement Learning: A Brief Introduction · 2012: Distributed deep learning (e.g., Google Brain) 2013: DQN for deep reinforcement learning 1997: LSTM-RNN 2015: Open source tools: Documents
Deep Learning: Models for Sequence Data (RNN and LSTM) · Recap: Convolutional Neural Network Special type of feedforward neural nets (local connectivity + weight sharing) Each layer
Deep Learning: Models for Sequence Data (RNN and LSTM) · Recap: Convolutional Neural Network Special type of feedforward neural nets (local connectivity + weight sharing) Each layer Documents
Automated Neural Image Caption Generator for Visually Impaired …cs224d.stanford.edu/reports/mcelamri.pdf · 2016-06-20 · then fed into a vanilla RNN or a LSTM. The vanilla RNN
Automated Neural Image Caption Generator for Visually Impaired …cs224d.stanford.edu/reports/mcelamri.pdf · 2016-06-20 · then fed into a vanilla RNN or a LSTM. The vanilla RNN Documents
Intro to Recurrent Neural Networks€¦ · This Lecture: 1. Segmentation Review 2. Intro to Polygon-RNN 3. Modeling Sequences 1. Vanilla RNN 2. Training and Problems 3. LSTM 4. Convolutional
Intro to Recurrent Neural Networks€¦ · This Lecture: 1. Segmentation Review 2. Intro to Polygon-RNN 3. Modeling Sequences 1. Vanilla RNN 2. Training and Problems 3. LSTM 4. Convolutional Documents
The Munich LSTM-RNN Approach to the MediaEval 2014 “Emotion in Music” Task
The Munich LSTM-RNN Approach to the MediaEval 2014 “Emotion in Music” Task Software
Why LSTM outperforms RNN - GitHub Pages...Solution - LSTM With the cell state, it runs straight down the entire chain, with only some minor linear interactions (NOT matrix multiplication
Why LSTM outperforms RNN - GitHub Pages...Solution - LSTM With the cell state, it runs straight down the entire chain, with only some minor linear interactions (NOT matrix multiplication Documents
Recurrent Neural Networks · Recurrent Neural Networks 11-785 / 2020 Spring / Recitation 7 Vedant Sanil, David Park “Drop your RNN and LSTM, they are no good!” The fall of RNN
Recurrent Neural Networks · Recurrent Neural Networks 11-785 / 2020 Spring / Recitation 7 Vedant Sanil, David Park “Drop your RNN and LSTM, they are no good!” The fall of RNN Documents
MULTIPLE-TARGET DEEP LEARNING FOR LSTM-RNN BASED …home.ustc.edu.cn/~sunlei17/pdf/MULTIPLE-TARGET.pdf · LSTM-RNN based speech enhancement. First, a direct map-ping approach using
MULTIPLE-TARGET DEEP LEARNING FOR LSTM-RNN BASED …home.ustc.edu.cn/~sunlei17/pdf/MULTIPLE-TARGET.pdf · LSTM-RNN based speech enhancement. First, a direct map-ping approach using Documents
Some RNN Variantsslazebni.cs.illinois.edu/spring17/lec03_rnn.pdf• A very simplified version of the LSTM! – Merges forget and input gate into a single ‘update’ gate! – Merges
Some RNN Variantsslazebni.cs.illinois.edu/spring17/lec03_rnn.pdf• A very simplified version of the LSTM! – Merges forget and input gate into a single ‘update’ gate! – Merges Documents
Supervised Sequence Labelling with Recurrent Neural Networksgraves/phd.pdfLong Short-Term Memory (LSTM) is an RNN architecture speci cally designed to overcome this limi-tation. In
Supervised Sequence Labelling with Recurrent Neural Networksgraves/phd.pdfLong Short-Term Memory (LSTM) is an RNN architecture speci cally designed to overcome this limi-tation. In Documents
180912 Lec6 RNN architectures(LSTM, GRU 등 최신 RNN 모델, …alinlab.kaist.ac.kr/resource/Lec6_RNN_architectures.pdf• Google’s Neural Machine Translation (GNMT) 3. Overcoming
180912 Lec6 RNN architectures(LSTM, GRU 등 최신 RNN 모델, …alinlab.kaist.ac.kr/resource/Lec6_RNN_architectures.pdf• Google’s Neural Machine Translation (GNMT) 3. Overcoming Documents
RNN & NLP Application - cs.kangwon.ac.krleeck/ML/RNN_NLP.pdf · LSTM RNN 응용 예 • Hand Writing by Machine • Music Composition • Neural Machine Translation • Image Caption
RNN & NLP Application - cs.kangwon.ac.krleeck/ML/RNN_NLP.pdf · LSTM RNN 응용 예 • Hand Writing by Machine • Music Composition • Neural Machine Translation • Image Caption Documents
PowerPoint Presentationbhiksha/courses/deeplearning/Fall.2015/... · Standard RNN • Recurrent ... •Controls how much of the information is to be let through. LSTM: ... possible
PowerPoint Presentationbhiksha/courses/deeplearning/Fall.2015/... · Standard RNN • Recurrent ... •Controls how much of the information is to be let through. LSTM: ... possible Documents
A RNN-LSTM based Predictive Autoscaling Approach on ...ijsrcseit.com/Paper/CSEIT1833498.pdf3Department of Information Technology, PSG College of Technology, Coimbatore, Tamilnadu,
A RNN-LSTM based Predictive Autoscaling Approach on ...ijsrcseit.com/Paper/CSEIT1833498.pdf3Department of Information Technology, PSG College of Technology, Coimbatore, Tamilnadu, Documents