LLNL-PRES-705839

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

AI for Climate Science:Tackling Climate Problem

with Machine Learning

Sookyung Kim

Center for Applied Scientific Computing

Lawrence Livermore National Laboratory

LLNL-PRES-705839

2

Outline

1. Motivation1) Data Challenges in Climate Science 2) Advantage of Deep Learning

2. Spatial Analysis 1) Convolutional Neural Networks2) Super-resolution techniques3) Variational Auto-Encoder4) Tutorial: Basic Tensorflow, CNNs

3. Temporal Analysis1) RNN and LSTM2) Tutorial: Dynamic RNN

4. Spatio-temporal Analysis1) ConvLSTM

Part1

Part2

LLNL-PRES-705839

3

▪ Computationally Expensive to generate dataPhysics based simulation/Modeling

▪ Large Scaled Data

▪ Human Labor of Domain Experts

Motivation(1) Data Challenges in Scientific Research

LLNL-PRES-705839

4

▪ Examples on Climate Science: Extreme Climate Event Analysis

Numerical weather prediction

”Expensive” process!

• 256 million grid points

Tracking extreme event:

”Labor intensive” process!• Hand-picked features, thresholds

• Done by human experts

GCM

RCM

Motivation(1) Data Challenges in Scientific Research

LLNL-PRES-705839

5

Motivation(2) Advantage of Deep Learning for Scientific Research

Deep Neural Networks

Output= sigmoid( woxo+w1x1+w2x2+… +wmxm)

LLNL-PRES-705839

6

Deep Learning enables Science

▪ Deep Neural Network: Pros— Non-linear latent

feature— Can deal with large

scaled data— Minimal Feature

Engineering

Cons— Require large scaled

labeled data— High computing cost for

the training

Classify satellite

images for carbon

monitoring

Analyze

recording

of human brain

Analyze

obituaries for

Cancer-related

disease.

LLNL-PRES-705839

7

Outline

1. Motivation1) Data Challenges in Climate Science 2) Advantage of Deep Learning

2. Spatial Analysis 1) Convolutional Neural Networks2) Super Resolution Techniques3) Variational Auto-Encoder4) Tutorial: Basic Tensorflow, CNNs

3. Temporal Analysis1) RNN and LSTM2) Tutorial: Dynamic RNN

4. Spatio-temporal Analysis1) ConvLSTM

LLNL-PRES-705839

8

Deep convolutional neural networks (CNNs) learn hierarchical feature representation

(1) Convolutional Neural Networks (CNNS)

LLNL-PRES-705839

9

ApplicationsDetection and Localization of Extreme Climate Events

Sookyung Kim et al. "Massive scale deep learning for detecting extreme climate events.”

7th International Workshop on Climate Informatics. 2017.

LLNL-PRES-705839

10

▪ From JWTC historical hurricane report, 1979 ~ 2016▪ JRA-55 reanalysis data▪ 5 channels(cloud fraction, precipitation, surface level pressure, eastward wind, northward wind) ▪ Size of collected grid : 20 longitude by 20 latitude▪ Size of dataset: 109,281

latitude

lon

gitu

de

ApplicationsDetection and Localization of Extreme Climate Events

LLNL-PRES-705839

11

Detection

Regression error (L2 loss) is

~ 4 degrees (about 450 km)

Localization

Almost 100 % test accuracy for detection

Accuracy

ApplicationsDetection and Localization of Extreme Climate Events

LLNL-PRES-705839

12

(2) Super-Resolution Model

Increase localization accuracy using Pixel Recursive Super Resolution

Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. "Pixel recursive super resolution." Proceedings of the

IEEE International Conference on Computer Vision. 2017.

LLNL-PRES-705839

13

ApplicationSuper-Resolution Model for downscaling process

Sookyung Kim et al. "Resolution reconstruction of climate data with pixel recursive model.”

IEEE International Conference on Data Mining Workshops (ICDMW), 2017.

LLNL-PRES-705839

14

(3) Variational Auto Encoder

Learning the compressed embedding of image (data) by encoder and decoder

Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

LLNL-PRES-705839

15

ApplicationSemi-supervised detection of extreme weather events in large climate datasets

Racah, Evan, et al. "Semi-supervised detection of extreme weather events in large climate datasets."

NIPS(2017).

LLNL-PRES-705839

16

(3) Tutorial: Basic Tensorflow

Tensorflow

▪ Open Source Software Library for Machine Intelligence

▪ Generate graph first→ feed data in session

▪ Code: https://github.com/aymericdamien/TensorFlow-Examples/ (Author: Aymeric Damien)

LLNL-PRES-705839

17

(3) Tutorial: CNNs

NMIST classification using CNNs

0

0

0

0

0

0

1

0

0

0

1-hot

encoding

LLNL-PRES-705839

18

(3) Tutorial: CNNs

Several Tensorflow built-in functions

conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)

32 개의 feature 를러닝함Feature(kernel) size 는 5x5

Convolution 한다음에 ReLU activation

LLNL-PRES-705839

19

(3) Tutorial: CNNs

Several Tensorflow built-in functions

conv1 = tf.layers.max_pooling2d(conv1, 2, 2)

Conv1 에서나온아웃풋이미지를2x2 max pooling 해서사이즈를줄임

1 2 0 0

3 1 1 0.1

0 0 0 0

0 1 0 0

3 1

1 02

2

2x2

max

pooling1 2 0 0

3 1 1 0.1

0 0 0 0

0 1 0 0

1.7 0.3

0.25 02

2x2

average

pooling

2

Why pooling?

LLNL-PRES-705839

20

(3) Tutorial: CNNs

Several Tensorflow built-in functions

fc1 = tf.contrib.layers.flatten(conv2)

fc1 = tf.layers.dense(fc1, 1024)

fc1 = tf.layers.dropout(fc1, rate=dropout, training=is_training)

out = tf.layers.dense(fc1, n_classes)

Dropout : 트레이닝할때 random 하게뉴런(stride)을일정 rate끊음으로써Overfitting 을막음

flatten

dense dense

fcCat?

Not cat?

Overfitting:

모델이트레이닝셋에만 너무맞게러닝되어서,

테스트셋에선안좋은성능을갖는현상

= Cat

LLNL-PRES-705839

21

Outline

1. Motivation1) Data Challenges in Climate Science 2) Advantage of Deep Learning

2. Spatial Analysis 1) Convolutional Neural Networks2) Super Resolution Techniques3) Variational Auto-Encoder4) Tutorial: Basic Tensorflow, CNNs

3. Temporal Analysis1) RNN and LSTM2) Tutorial: Dynamic RNNs

4. Spatio-temporal Analysis1) ConvLSTM

LLNL-PRES-705839

22

(1) RNNs : Recurrent Neural Network

LLNL-PRES-705839

23

(1) RNNs

• When dealing with a time series, it tends to forget old information.

When there is a distant relationship of unknown length, we wish to have a “memory” to it.

• Vanishing gradient problem

Problem of RNN

City I love is Seoul

서울 은 내가 좋아하는 도시

LLNL-PRES-705839

24

(1) LSTM: containing memorizing operation

• RNNs: Simple layer

• LSTM: 4 interactive layers (tanh)

LLNL-PRES-705839

25

(1) LSTM: containing memorizing operation

▪ Standard RNN with sigmoid— The sensitivity of the input values

decays over time— The network forgets the previous input

▪ Long-Short Term Memory (LSTM) [2]

— The cell remember the input as long as it wants

— The output can be used anytime it wants

LLNL-PRES-705839

26

Yellow Haze:

• The annual average of PM10 in Korea

peninsula is rapidly and continuously

increasing since 1995

• According to the report in 2009, the

level of PM10 in Seoul is

much higher than those in New York

and Paris.

𝑪𝒐𝒏𝒄𝒆𝒏𝒕𝒓𝒂𝒕𝒊𝒐𝒏 𝒐𝒇 𝒇𝒊𝒏𝒆 𝒅𝒖𝒔𝒕= 𝒇 (𝑐𝑜𝑛𝑐𝑒𝑛𝑡𝑟𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 𝑐ℎ𝑒𝑚𝑖𝑐𝑎𝑙 𝑝𝑜𝑙𝑙𝑢𝑡𝑎𝑛𝑡𝑠,

𝑐𝑙𝑖𝑚𝑎𝑡𝑒 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒𝑠)

Application: Predicting concentrations of fine dust in Seoul using LSTM

LLNL-PRES-705839

27

Dataset

(1) input:- P: Concentration of 6 gaseous pollutants

SO2,CO,NO2,O3, PM10, PM2.5

from all 39 stations in Seoul

- C: 9-measured variables

wind speed(m/s), wind direction,

humidity(%), water vapor pressure(hPa),

dew point temperature(C), surface

pressure(hPa), sunlight(hr), range of

vision(m), surface temperature(C) from

one station in Seoul

(2) output: Future concentration of coarse(PM10)

and fine(PM2.5) dust particles

LLNL-PRES-705839

28

Model

Loss: element-wise MSE

LLNL-PRES-705839

29

Results

Comparison between hourly variation of predicted

PM10 and PM2:5 with ground truth values on sampled day (2017/03/17)

at station6 (Seodaemun district).

LLNL-PRES-705839

30

(2) Tutorial: Dynamical-RNN using LSTM cell

— Code: https://github.com/aymericdamien/TensorFlow-Examples/— Problem: Classify linear sequence vs random sequence

• Class 0 (i.e: [1,0] ): linear sequences (i.e. [0, 1, 2, 3,...]) • Class 1 (I.e: [0,1] ) : random sequences (i.e. [1, 3, 10, 7,...])

LSTM LSTM LSTM LSTM LSTM…..

W

1

0

0

1OR

0 3 6 9 30

1 0 27 7 9

Class 0

Linear

Class 1

Random

LLNL-PRES-705839

31

(2) Tutorial: Dynamical-RNN using LSTM cell

LSTM

LSTM0 LSTM1 LSTM2 LSTMseqlen-1…..

Several Tensorflow built-in functions

• lstm_cell = tf.contrib.rnn.BasicLSTMCell(n_hidden)

n_hidden

• outputs, states = tf.contrib.rnn.static_rnn(lstm_cell, x, dtype=tf.float32, sequence_length=seqlen)

states

x: as list

outputs: as list

x0 x1 x2 xseqlen-1

output0 output1 output2 Outputseqlen-1

LLNL-PRES-705839

32

Outline

1. Motivation1) Data Challenges in Climate Science 2) Advantage of Deep Learning

2. Spatial Analysis 1) Convolutional Neural Networks2) Super Resolution Techniques3) Variational Auto-Encoder4) Tutorial: Basic Tensorflow, CNNs

3. Temporal Analysis1) RNN and LSTM2) Tutorial: Dynamic RNNs

4. Spatio-temporal Analysis1) ConvLSTM

LLNL-PRES-705839

33

How can we deal with spatiotemporal data?

• A pure encoding-prediction structure is not enough.

• We are dealing with spatiotemporal data.

FC-LSTM

LLNL-PRES-705839

34

How to deal with spatiotemporal data?

LLNL-PRES-705839

35

How to deal with spatiotemporal data? : ConvLSTM

• Input and state at a timestep are 3-d tensors.

• Convolution is used for both input-to-state and state-to-state connection.

• Conv-part: capture spatial information

• LSTM-part: capture temporal information

Convolutional LSTM

LLNL-PRES-705839

36

Application(1) Tracking and Predicting Extreme Climate Events:

Spatial Analysis to Spatio-temporal Analysis

Kim, Sookyung, et al. "Deep-Hurricane-Tracker: Tracking and Forecasting Extreme Climate Events."

2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2019.

1. The number of target

event is not known

previously.

2. Event is defined by

spatio-temporal

dynamics and

correlation of multiple

climate variables

Copyright of figure: https://github.com/LBL-EESA/TECA

Challenges of extreme climate event tracking problem: adding Time!

LLNL-PRES-705839

37

Problem 1. How to deal with unknown number of objects?: Density map

𝑿𝒊 ∈ 𝑹𝒎×𝒏× 𝒄

𝒀𝒊 = 𝒚𝒊𝟎

= { 𝒂𝒊𝟎, 𝒃𝒊

𝟎 , 𝒂𝒊𝟏, 𝒃𝒊

𝟏 }𝒚𝒊𝟎

• Design output Yi as density-map with same size of input Xi

• Design problem as pixel-wise regression problem

from climate image to density-map of events

𝒚𝒊𝟏

𝒚𝒊𝟎

𝒚𝒊𝟏

𝒚𝒊𝟐

𝒀𝒊 = 𝒚𝒊𝟎, 𝒚𝒊

𝟏, 𝒚𝒊𝟐

= { 𝒂𝒊𝟎, 𝒃𝒊

𝟎 , 𝒂𝒊𝟏, 𝒃𝒊

𝟏 , (𝒂𝒊𝟐, 𝒃𝒊

𝟐)}

𝒀𝒊 ∈ 𝑹𝒎×𝒏

𝒚𝒊𝟎 𝒚𝒊

𝟏

𝒚𝒊𝟎

𝒚𝒊𝟏

𝒚𝒊𝟐

LLNL-PRES-764004

LLNL-PRES-705839

38

Problem 2: How to deal with spatiotemporal data? : Proposed Tracking Model using Conv-LSTM

• Multi-layered Conv-LSTM

• Loss: pixel-wise MSE between output and GT

LLNL-PRES-764004

LLNL-PRES-705839

39

Experimental Settings

• Dataset• Input: 20-year run (1996 to 2015) 3 hourly climate simulation data

: CAM5 (Community atmospheric model v5)

• Climate variables: zonal wind (U850), Meridional wind (V850), Precipitation

(PRECT)

• Output label: Ground-truth trajectories obtained from TECA (Toolkit got Extreme

Climate Analysis)

• Time-steps: 10 (30 hours)

LLNL-PRES-705839

40

Results

conv-LSTM based tracking model

Ground

Truth

Output

LLNL-PRES-764004

LLNL-PRES-705839

41

Comparison of Average Precision with Baselines

[Reference]

1. Y. Liu et al. "Application of deep convolutional neural networks for detecting extreme weather in climate datasets." arXiv preprint

arXiv:1605.01156 (2016).

2. E. Racah et al. "ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of

extreme weather events." NIPS. 2017.

LLNL-PRES-764004

LLNL-PRES-705839

42

Hurricane Heat-Map Prediction using ConvLSTM

LLNL-PRES-764004

LLNL-PRES-705839

43

Results

LLNL-PRES-764004

LLNL-PRES-705839

44

Application(2) Tracking and Predicting Extreme Climate Events:

How to connect each hurricane?

Visual Object Tracking Challenge

Why don’t we just use VOT method?

https://youtu.be/UgjQDWIGriw • Butterfly effect• “longer term” and “wider

range”

spatio-temporal dynamics

• Multiple climate variables

has scientific interactions

• No Rigid boundary• Events are changing their

shape with no clear boundary

• Climate event is hard to

visually distinguish from each

other.

LLNL-PRES-705839

45

Our Model: Focus learning Module + Tracking Module

: : :

Encoding

Networks

Density

Estimator

Input

Copy

Copy

Focus learning Module

:

Random

noise

Tracking Module

CNNs+FC

� �

Bounding Box

Regressor

Image

w

� �

w

� �

w

� � �

…

…

� � � �

� � ⨀� � + � �� �

� � � � ��

�

� � � � �

� ( � :� )

�

� ( � )

� ( � )

� ( � )

� ( � )

Sookyung Kim et al. ”Learning to focus and track extreme climate events.”

The 30th British Machine Vision Conference (BMVC), 2019.

Application(2) Tracking and Predicting Extreme Climate Events:

How to connect each hurricane?

LLNL-PRES-705839

46

Qualitative Results

Sookyung Kim et al. ”Learning to focus and track extreme climate events.”

The 30th British Machine Vision Conference (BMVC), 2019.

LLNL-PRES-705839

47

Quantitative Results

(b) Comparison with Baselines(a) Comparison with Different Sizes

of Focus Learning Steps (c) Comparison with Augmented Data

LLNL-PRES-705839

48

Data Augmentation

LLNL-PRES-705839

49

Qualitative Results

LLNL-PRES-705839

50

Quantitative Results

(b) Comparison with Baselines(a) Comparison with Different Sizes

of Focus Learning Steps (c) Comparison with Augmented Data

(b) Comparison with Baselines(a) Comparison with Different Sizes

of Focus Learning Steps (c) Comparison with Augmented Data

(b) Comparison with Baselines(a) Comparison with Different Sizes

of Focus Learning Steps (c) Comparison with Augmented Data

LLNL-PRES-705839

51

Result video