Upload
jeffry-jefferson
View
217
Download
0
Embed Size (px)
Time Series Prediction andSupport Vector Machines
ICONS Presentation
Spring 2006
N. Sapankevych16 April 2006
N. Sapankevych - 4/16/062
Time Series Prediction and SVMs
Research Overview SVMs for Time Series Prediction Applications Q&A
AGENDA
N. Sapankevych - 4/16/063
Time Series Prediction and SVMs
Research focus on Support Vector Machines (SVMs) and their applications for time series prediction
– Extension of previous research work (and qualifiers) w/ MLPs and SVMs as classifiers
Goal to complete SVM time series prediction survey paper by end of Spring 2006 semester
– Work in progress (over 50 papers down-selected from several hundred published in dozens of different journals)
– Build on research work formulated in qualifying exams
Further goal to define thesis topic– Extend SVM time series prediction research– Potential collaboration w/ UCF (more on that later)
RESEARCH OVERVIEW
N. Sapankevych - 4/16/064
Time Series Prediction and SVMs
Quick SVM history and background– Invented by Vapnik early 1990’s
First instantiation actually demonstrated in COLT ‘92
– SVM another “class” of learning machine– Primary application for (mostly binary)
classification problems– Also, SVMs used for regression applications
(function estimation) – More recently, research on use in time series
prediction applications (focus of research) Note here SVR (Support Vector Regression) plays
significant part in this
TIME SERIES PREDICTION USING SVMs
N. Sapankevych - 4/16/065
Time Series Prediction and SVMs
Why use Artificial Neural Networks (ANNs) for time series prediction?
– Traditional methods assume “model” or “data generating process”
ARMA and its derivatives Kalman {other – IMM, MHT, Particle Filter, etc.} Stationary (and sometimes Gaussian) processes assumed
(drawback in some cases) Assumed model itself may be incorrect in some cases
– Neural Networks “let the data speak for itself” Model-free Can work in both linear or non-linear applications Performance tradeoff with computational complexity Found to perform well w/o all the data
TIME SERIES PREDICTION USING SVMs (con’t)
N. Sapankevych - 4/16/066
Time Series Prediction and SVMs
Many SVM advantages over other ANN-based learning machines such as MLPs (based on architecture)
– Guaranteed unique solution to cost function (quadratic programming)
Not necessarily so in MLPs
– Map non-linear functions to linear space using Kernel functions
Would need more layers/neurons as compared to MLPs
– Fewer free parameters for optimization than MLPs Selection of Kernel function and other constants somewhat of
an “art” for both linear and non-linear applications Selection of parameters may be just as hard, however
TIME SERIES PREDICTION USING SVMs (con’t)
N. Sapankevych - 4/16/067
Time Series Prediction and SVMs
SVM basics:– For supervised learning and for a regression application,
define and minimize the “risk” functional
– Define the “loss” function (quadratic in this case)
– x is input vector (input)– y is output vector (response) is free variable– P(x,y) is NOT KNOWN– There are other loss functions
-insensitive loss function most popular (Vapnik)
TIME SERIES PREDICTION USING SVMs (con’t)
),()),(,()( yxdPxfyLR
2)),(()),(,( xfyxfyL
N. Sapankevych - 4/16/068
Time Series Prediction and SVMs
More on loss functions: -insensitive loss function most popular (Vapnik –
linear model shown below)
– Note zero term for values within boundary– These loss functions have been shown (Vapnik,
others) to be robust for function estimation
TIME SERIES PREDICTION USING SVMs (con’t)
otherwisexfy
xfyxfy
),(
),(,0),(
N. Sapankevych - 4/16/069
Time Series Prediction and SVMs
Empirical Risk Minimization (ERM) vs. Structural Risk Minimization (SRM)– ERM means minimize the following by finding :
– SRM means minimize the following by adjusting (C and as well):
– Why the extra term? If you don’t have much data, ERM may be inadequate
method by which to measure “goodness” of fit – add regularization term (also known as capacity control term)
– What does the extra term do? “Flattens” the fit Controls Capacity
TIME SERIES PREDICTION USING SVMs (con’t)
L
iiemp xfy
LR
1
2)),((1
)(
2
1
2
2)),(( wxfy
L
CR
L
iireg
N. Sapankevych - 4/16/0610
Time Series Prediction and SVMs
More on SRM and ERM:– One fundamental difference between SVMs and
MLPs is the regularization term– Regularization term assumes weights (w) are for
linear regression function
What if function to estimate is not linear?– Use Kernels: transform data to higher
dimensional space
TIME SERIES PREDICTION USING SVMs (con’t)
bwxY
TangentHyperbolicXXK
RBFGaussianXXK
PolynomialXXK
iT
i
p
iT
00
2
22
tanh
)(2
1exp
1
N. Sapankevych - 4/16/0611
Time Series Prediction and SVMs
SVM Architecture (from Scholkopf tutorial)
TIME SERIES PREDICTION USING SVMs (con’t)
N. Sapankevych - 4/16/0612
Time Series Prediction and SVMs
How do you solve for weights?– Solution turns out to be quadratic programming
problem– Use Lagrange multipliers to find optimal weights
Non-zero multipliers associated w/ “Support Vectors” {details found in my Qualifier #2 presentation –
classification example below}
TIME SERIES PREDICTION USING SVMs (con’t)
X2
X1
Separating Hyperplane
Support Vectors
N. Sapankevych - 4/16/0613
Time Series Prediction and SVMs
Much more to this effort (work in progress) and more questions to answer specifically for time series prediction:
– How to train SVM for this application? SMO Gradient Ascent Chunking {other} Computational complexity and data quantity issue for training How often do you need to retrain and when?
– Sparse data vs. performance Same “kind” of problem for Kalman Filter (time updates – covariance
growth)– How to pick a Kernel function?– How to pick a regularization function?– How to pick regularization constants?– How to manage error?
Stability?– Any “real time” applications?
Computational complexity may be prohibitive
TIME SERIES PREDICTION USING SVMs (con’t)
N. Sapankevych - 4/16/0614
Time Series Prediction and SVMs
Current research effort focused on exhaustive literature search for SVM Time Series Prediction and its applications– Note K.-R. Muller et. al. “Predicting Time Series
with Support Vector Machines” and Vapnik’s “Statistical Learning Theory” key publications in this area
– Other tutorials and resources available
SVM research relatively new endeavor (about 10+ years)– Published articles growing in number and by
application
APPLICATIONS
N. Sapankevych - 4/16/0615
Time Series Prediction and SVMs
Largest quantity of published articles found to date is from financial market prediction applications
– Journal publications indicate SVMs perform well given non-linear nature of stock price prediction
– Other related financial applications such as credit rating forecasting
Several other applications– Utility forecasting (power load and consumption)– Manufacturing industry (machinery MTBF and reliability
forecasting)– Medical applications (drug effects prediction)– Environmental (rainfall, pollution)
Notably missing (as I have found so far)– Signal Processing– Electromechanical Control Systems– Remote Sensing applications (radar tracking, etc.) – Maybe SVMs not well suited for these real time applications?
APPLICATIONS (con’t)
N. Sapankevych - 4/16/0616
Time Series Prediction and SVMs
Potential collaborative research opportunity and application to my current research
– Dr. W. Linwood Jones at UCF Central Florida Remote Sensing Laboratory (CFRSL) http://www.engr.ucf.edu/centers/cfrsl/Index.htm
– CFRSL focus on satellite-based remote sensing applications
SeaWinds microwave scatterometer measuring ocean surface wind vectors (QuickSCAT satellite)
Several algorithm development efforts (among other things)– Rainfall, windspeed, surface temperature, and other derived
remote sensing products
– Work in progress to establish cooperative research effort and (possibly) generate thesis topic
Possible research: multi-sensor applications using SVMs
APPLICATIONS (con’t)
N. Sapankevych - 4/16/0617
Time Series Prediction and SVMs
Q&A
Q&A
Alexander– 7 months and getting bigger!