Hybrid neural networks for time series learning by Tian Guo, EPFL, Switzerland

Hybrid Neural Networks for

Time Series Learning

Tian Guo

Ph.D.

24 Nov. 2016

Outline

• Introduction

• Hybrid Neural Network (HNN)

• HNN for Real

• Preliminaries

• TreNet: a HNN for learning the local trend of time series

• Experiment results

• Conclusion

2

Introduction

3 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

Introduction

• Time series data: a sequence of data points consisting of

successive measurements made over time.

• Internet of Things

• Sensor networks

• Mobile phones

• And more…


Introduction

• Time series analytics in a variety of applications

• Classification

• Prediction

• Anomaly detection

• Pattern discovery

• And more…

5

Pattern 1 Pattern 2

Introduction Hybrid Neural Network(HNN) TreNet for Local Trend

Introduction

• Time series analytics tools

• Statistics

• Hidden Markov Model (HMM)

• State Space Model

• ARIMA

• Machine learning

• Random Forest

• SVM

• Gaussian Process

•

6

Random Forest


Introduction

• Neural Network and Deep Learning

• Language translation

• E.g., Google’s Multilingual Neural Machine Translation System

• Computer vision

• E.g., Microsoft Research’s PReLU network outperforms Human-Level

performance on ImageNet Classification

• Speech recognition

• E.g., Amazon Echo, Apple Siri

• Time series classification

• E.g. recognize patterns in multivariate

time series of clinical measurements

7

Z. Lipton, et al. “Learning to Diagnose with LSTM Recurrent Neural Networks”. ICLR, 2016.


Introduction

• Fundamental network architectures

• Convolutional neural network: input two-dimensional data, e.g.,

image

• Recurrent neural network: input

8

Unfolded recurrent connections in a RNN


Introduction

• Convolutional Neural Network (CNN)

• Feature learning for images

• To extract high-level features from raw data

• Such high-level features are further used for classification or

regression.

9

K. He, et al. “Delving deep into rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. arxiv.org, 2015


Introduction

• Convolutional Neural Network (CNN)

• CNN in the Human Activity Recognition Problem

• Multichannel time series acquired from a set of body-worn sensors

• To predict human activities by training a CNN over time series

10

J. Yang, et al. “Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition”. IJCAI 2015

Value

Time


Introduction

• Recurrent Neural Network (RNN)

• A powerful tool to model sequence data

• To capture dependency in sequence data

• Long-short term memory network (LSTM)

• A widely used variant of RNNs

• Equipped with memory and gate mechanism

• To overcome gradient vanishing and explosion

11

S. Hochreiter, et al. “Long short-term memory’’. Neural computation, 1997.


Hybrid Neural Networks



• CNN or RNN

• Work well for respective data, i.e. images and sequence data



• A cascade of CNN and RNN

• Classification of EEG data

• Electroencephalogram (EEG) : multiple time series corresponding to

measurements across different spatial locations over the cortex.

• Mental load classification task:

measures the working memory

responsible for transient retention

of information in the brain.

14

P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016





• A key challenge in correctly recognizing mental states

• EEG data often contains translation

and deformation of signal in space,

frequency, and time, due to inter-

and intra-subject differences

15






16





• EEG data classification

17




• Time Series

• Noisy

• Non-stationary

• Hidden information: states, dynamics

• Auto-correlated on the temporal dimension

• Manual feature engineering

• Preprocessing: de-trending, outlier removal, etc.

• Dimension reduction: Fourier Transformation

• Piecewise approximation: PAA, PCA, PLA, etc.

• Application-specific, domain knowledge

18

Why do we need hybrid architectures?



• Hybrid architectures: end-to-end learning framework

• Loss function driven training

• Learning representative features

• Capturing sequential dependency in data

19

Why do we need hybrid architectures?




20



HNN for Real:

TreNet for Learning the Local Trend

21

T. Guo, et al. TreNet: Hybrid Neural Networks for Learning the Local Trend in Temporal Data. In submission to ICLR, 2017


Preliminaries

• Conventional trend analysis in time series

is the seasonal component at time t

is the trend component at t

is the remainder


Preliminaries

• Local trend

• Measure the intermediate local behaviour, i.e. upward or downward

pattern of time series

• For instance, the time series of household power consumption and

the local trends are shown as follows:

• Time series

• Extracted local trend ,

is the duration and is the slope


Preliminaries

• Learning and forecasting the local trend

• Predict ,

• Useful in many applications

• Smart energy

• Stock market

• And more …


Preliminaries

• Learning and forecasting the local trend

• Local raw data

• Global contextual information

in the historical sequence of trend

• To learn a function

is either or


TreNet

• Overview of TreNet

• is derived by training the LSTM over sequence to

capture the dependency in the trend evolving.

• corresponds to local features extracted by CNN from


is derived by training the LSTM over sequence to capture

the dependency in the trend evolving.

corresponds to local features extracted by CNN from

TreNet

• Overview

•


TreNet

• Learning


TreNet

• LSTM

• Feed the sequence of

• Output


TreNet

• CNN

• Output


TreNet

• Feature Fusion and output layers


TreNet

• Learning

• Gradient descent


Experiments

• Datasets

• Daily Household Power Consumption (HousePC)

• Gas Sensor (GasSensor)

• Stock Transaction (Stock)

33

E Keogh, et al. “An online algorithm for segmenting time series”. ICDM, 2001


Experiments

• Baselines

• CNN

• LSTM

• Support Vector Regression (SVR)

• Radial Basis kernel (SVRBF)

• Polynomial kernel (SVPOLY)

• Sigmoid kernel (SVSIG)

• Pattern-based Hidden Markov Model (pHMM)


P. Wang, et al. “Finding Semantics in Time Series”, SIGMOD 2011

Experiments

• Results: overall accuracy


Experiments

• Results: prediction visualization


Conclusion

37

Conclusion

38

• Hybrid neural networks

• TreNet for the Local Trend Learning

• Future work: a generic idea

• Social media streams

• Heterogeneous data

• Influence analysis

• And more…

Thanks! Q & A

This work is supported by EU OpenIoT Project (Open Source Solution for the Internet of Things)

http://www.openiot.eu/

Feel free to contact: [email protected], [email protected]




mailto:[email protected]

mailto:[email protected]

Data & Analytics

Hybrid neural networks for time series learning by Tian Guo, EPFL, Switzerland