Time Series Prediction andSupport Vector Machines

ICONS Presentation

Spring 2006

N. Sapankevych16 April 2006

N. Sapankevych - 4/16/062

Time Series Prediction and SVMs

Research Overview SVMs for Time Series Prediction Applications Q&A

AGENDA

N. Sapankevych - 4/16/063

Time Series Prediction and SVMs

Research focus on Support Vector Machines (SVMs) and their applications for time series prediction

– Extension of previous research work (and qualifiers) w/ MLPs and SVMs as classifiers

Goal to complete SVM time series prediction survey paper by end of Spring 2006 semester

– Work in progress (over 50 papers down-selected from several hundred published in dozens of different journals)

– Build on research work formulated in qualifying exams

Further goal to define thesis topic– Extend SVM time series prediction research– Potential collaboration w/ UCF (more on that later)

RESEARCH OVERVIEW

N. Sapankevych - 4/16/064

Time Series Prediction and SVMs

Quick SVM history and background– Invented by Vapnik early 1990’s

First instantiation actually demonstrated in COLT ‘92

– SVM another “class” of learning machine– Primary application for (mostly binary)

classification problems– Also, SVMs used for regression applications

(function estimation) – More recently, research on use in time series

prediction applications (focus of research) Note here SVR (Support Vector Regression) plays

significant part in this

TIME SERIES PREDICTION USING SVMs

N. Sapankevych - 4/16/065

Time Series Prediction and SVMs

Why use Artificial Neural Networks (ANNs) for time series prediction?

– Traditional methods assume “model” or “data generating process”

ARMA and its derivatives Kalman {other – IMM, MHT, Particle Filter, etc.} Stationary (and sometimes Gaussian) processes assumed

(drawback in some cases) Assumed model itself may be incorrect in some cases

– Neural Networks “let the data speak for itself” Model-free Can work in both linear or non-linear applications Performance tradeoff with computational complexity Found to perform well w/o all the data

TIME SERIES PREDICTION USING SVMs (con’t)

N. Sapankevych - 4/16/066

Time Series Prediction and SVMs

Many SVM advantages over other ANN-based learning machines such as MLPs (based on architecture)

– Guaranteed unique solution to cost function (quadratic programming)

Not necessarily so in MLPs

– Map non-linear functions to linear space using Kernel functions

Would need more layers/neurons as compared to MLPs

– Fewer free parameters for optimization than MLPs Selection of Kernel function and other constants somewhat of

an “art” for both linear and non-linear applications Selection of parameters may be just as hard, however

TIME SERIES PREDICTION USING SVMs (con’t)

N. Sapankevych - 4/16/067

Time Series Prediction and SVMs

SVM basics:– For supervised learning and for a regression application,

define and minimize the “risk” functional

– Define the “loss” function (quadratic in this case)

– x is input vector (input)– y is output vector (response) is free variable– P(x,y) is NOT KNOWN– There are other loss functions

-insensitive loss function most popular (Vapnik)

TIME SERIES PREDICTION USING SVMs (con’t)

),()),(,()( yxdPxfyLR

2)),(()),(,( xfyxfyL

N. Sapankevych - 4/16/068

Time Series Prediction and SVMs

More on loss functions: -insensitive loss function most popular (Vapnik –

linear model shown below)

– Note zero term for values within boundary– These loss functions have been shown (Vapnik,

others) to be robust for function estimation

TIME SERIES PREDICTION USING SVMs (con’t)

otherwisexfy

xfyxfy

),(

),(,0),(

N. Sapankevych - 4/16/069

Time Series Prediction and SVMs

Empirical Risk Minimization (ERM) vs. Structural Risk Minimization (SRM)– ERM means minimize the following by finding :

– SRM means minimize the following by adjusting (C and as well):

– Why the extra term? If you don’t have much data, ERM may be inadequate

method by which to measure “goodness” of fit – add regularization term (also known as capacity control term)

– What does the extra term do? “Flattens” the fit Controls Capacity

TIME SERIES PREDICTION USING SVMs (con’t)

L

iiemp xfy

LR

1

2)),((1

)(

2

1

2

2)),(( wxfy

L

CR

L

iireg

N. Sapankevych - 4/16/0610

Time Series Prediction and SVMs

More on SRM and ERM:– One fundamental difference between SVMs and

MLPs is the regularization term– Regularization term assumes weights (w) are for

linear regression function

What if function to estimate is not linear?– Use Kernels: transform data to higher

dimensional space

TIME SERIES PREDICTION USING SVMs (con’t)

bwxY

TangentHyperbolicXXK

RBFGaussianXXK

PolynomialXXK

iT

i

p

iT

00

2

22

tanh

)(2

1exp

1

N. Sapankevych - 4/16/0611

Time Series Prediction and SVMs

SVM Architecture (from Scholkopf tutorial)

TIME SERIES PREDICTION USING SVMs (con’t)

N. Sapankevych - 4/16/0612

Time Series Prediction and SVMs

How do you solve for weights?– Solution turns out to be quadratic programming

problem– Use Lagrange multipliers to find optimal weights

Non-zero multipliers associated w/ “Support Vectors” {details found in my Qualifier #2 presentation –

classification example below}

TIME SERIES PREDICTION USING SVMs (con’t)

X2

X1

Separating Hyperplane

Support Vectors

N. Sapankevych - 4/16/0613

Time Series Prediction and SVMs

Much more to this effort (work in progress) and more questions to answer specifically for time series prediction:

– How to train SVM for this application? SMO Gradient Ascent Chunking {other} Computational complexity and data quantity issue for training How often do you need to retrain and when?

– Sparse data vs. performance Same “kind” of problem for Kalman Filter (time updates – covariance

growth)– How to pick a Kernel function?– How to pick a regularization function?– How to pick regularization constants?– How to manage error?

Stability?– Any “real time” applications?

Computational complexity may be prohibitive

TIME SERIES PREDICTION USING SVMs (con’t)

N. Sapankevych - 4/16/0614

Time Series Prediction and SVMs

Current research effort focused on exhaustive literature search for SVM Time Series Prediction and its applications– Note K.-R. Muller et. al. “Predicting Time Series

with Support Vector Machines” and Vapnik’s “Statistical Learning Theory” key publications in this area

– Other tutorials and resources available

SVM research relatively new endeavor (about 10+ years)– Published articles growing in number and by

application

APPLICATIONS

N. Sapankevych - 4/16/0615

Time Series Prediction and SVMs

Largest quantity of published articles found to date is from financial market prediction applications

– Journal publications indicate SVMs perform well given non-linear nature of stock price prediction

– Other related financial applications such as credit rating forecasting

Several other applications– Utility forecasting (power load and consumption)– Manufacturing industry (machinery MTBF and reliability

forecasting)– Medical applications (drug effects prediction)– Environmental (rainfall, pollution)

Notably missing (as I have found so far)– Signal Processing– Electromechanical Control Systems– Remote Sensing applications (radar tracking, etc.) – Maybe SVMs not well suited for these real time applications?

APPLICATIONS (con’t)

N. Sapankevych - 4/16/0616

Time Series Prediction and SVMs

Potential collaborative research opportunity and application to my current research

– Dr. W. Linwood Jones at UCF Central Florida Remote Sensing Laboratory (CFRSL) http://www.engr.ucf.edu/centers/cfrsl/Index.htm

– CFRSL focus on satellite-based remote sensing applications

SeaWinds microwave scatterometer measuring ocean surface wind vectors (QuickSCAT satellite)

Several algorithm development efforts (among other things)– Rainfall, windspeed, surface temperature, and other derived

remote sensing products

– Work in progress to establish cooperative research effort and (possibly) generate thesis topic

Possible research: multi-sensor applications using SVMs

APPLICATIONS (con’t)

N. Sapankevych - 4/16/0617

Time Series Prediction and SVMs

Q&A

Q&A

Alexander– 7 months and getting bigger!