Upload
julissa-miner
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Universität Hamburg Institut für Wirtschaftsinformatik
Prof. Dr. D.B. Preßmar
Final Results of the NN3 Neural Network Forecasting Competition
Sven F. Crone, Konstantinos Nikolopoulos and Michele Hibon
Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”
2005 SAS & IIF Grant2005 SAS & IIF Grant
RATIONAL
OBJECTIVES
RESULTS
DISCUSSION
FURTHER RESEARCH
2005 SAS & IIF Grant2005 SAS & IIF Grant
RATIONAL
Only 1 evaluation of NN within Forecasting Competitions Distinct fields of research and participation
NN: breakthrough or passing fad?NN: breakthrough or passing fad?
Reid1969
Santa Fe1991
BUSINESS FORECASTING COMPETITIONS
NN COMPETITIONS
Suykens1998
Reid1972
Newbold & Granger1974
Makridakis & Hibon1979
M-Competition 1982
M2-Competition 1988
M3-Competition 2000
H-Competition,Hibon 2006
EUNITE2001
ANNEXG2001
BI Cup2003
CATS2005ISF052005
ISF06 ANNEX 2006
WCCI2006
Only 1 NN entryBalkin & Ord
Most NN competitions = classification (EUNITE’02, WCCI06 etc.) Limited evidence on Regression evaluations
Visit http://www.neural-forecasting.com/competition_data.htm
CI Time Series CompetitionsCI Time Series Competitions
Time Series Data Format Length Submis.
SANTA FE 1991Gershenfeld & Weigend
2 univariate4 multivariate
UV: Laser, UV: Artificial, Sleep, Exchange rate, Astrophysics, Music
1000, 34000, 300000, 100000,
27704, 380830+
Black Box 1998 Suykens & Vandewalle
1 univariate Physics2000
(1000)17
EUNITE 2001 1 multivariate Electrical Load35040(31)
56
ANNEXG 2001Dawson et al.
1 multivariate Hydrology1460 pointsHydrology
12
BI Cup 2003Weber
1 multivariate Sugar sales365 days
(14)10+
CATS 2005, IEEELendasse,
1 univariatein 5 parts
Artificial4905pointas
(95 points, 5*19)25
ISF2005Crone
2 univariate Airline, M3-Competition 144, 85 16
ANNEXG / ISF2006Dawson et al., Crone
3 multivariate Hydrology1460 pointsHydrology
12
WCCI 2006 Predictive Uncertainty, Gawley
1 univariate3 multivariate
UV: Synthetic, Precipitation, Temperature, SO2
380, 10000, 10000, 21000
9
Conduct competition on industry data Evaluate different NN methodologies Can NN forecasting be AUTOMATED on many time series?
Reasons? Modelling DecisionsReasons? Modelling Decisions
Gap between forecasting & NN domains NN evaluations on different data types No positive evidence on M-type data
• Short time series• Noisy time series
Discouraging research findings NN cannot forecast seasonal time series No valid & reliable methodology to model NN No automation of NN modelling possible
Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”
2005 SAS & IIF Grant2005 SAS & IIF Grant
OBJECTIVESa) What is the performance (accuracy, robustness
& resources) of NN in comparison to established forecasting methods?
b) What are the current “best practice” methodologies utilised by researchers to model NN for time series forecasting
Multiple Hypothesis Testing similar to M3-competition
Competition DesignCompetition Design
Multiple empirical Time Series Complete set of 111 time series Reduced set of 11 time series Representative structures monthly industry data
• long & short time series• Seasonal and non-seasonal series
Scaled observations for anonymity No domain knowledge 18 steps ahead forecasts
Simulated ex ante (out of sample) evaluation
Multiple error measures & computational time
Testing of conditions under which NN perform well/bad
NN3 COMPETITION
Competition DesignCompetition Design
46 Submissions for the reduced dataset
9 benchmarks
22 submissions for the complete dataset
8 benchmarks
SubmissionsSubmissions
2005 SAS & IIF Grant2005 SAS & IIF Grant
Automated AI/CI approaches can very well do the job! (batch forecasting)
Balkin’s and Ord approach was not very ‘bad’ after all..
Performance was verified across many metrics (including MASE), parametric + non-parametric
Performance was verified with multiple hypothesis: long/short, seasonal/non seasonal, difficult/easy
So… WHAT do we know NOW that we did not knew before NN3?
2005 SAS & IIF Grant2005 SAS & IIF Grant
Time Series Benchmarks are very hard to beat! Forecast Pro, Theta model and Marc Wildi’s Stat benchmark outperform overall all CI/AI approachesFor the ‘harder’ part of the NN3 dataset – 25 short+non-seasonal series – CI approaches managed to outperform all other approaches!! Full automation seems to be possible in large scale forecasting tasks
+ Side results… New Stat benchmarks that perform outstandinglyImprovement of established forecasting engines in the last 10
years
So… WHAT do we know NOW that we did not knew before NN3?
+
+
+
+
2005 SAS & IIF Grant2005 SAS & IIF Grant
Computational times….
Leaders of the field (Academia + Commercial)
Time series features that would necessitate the use of AI/CI approaches
Replication in a competition of the M3 volume (NN5…111, tourism competition…1000+)
Best practices?
Full automation??
and… WHAT we still DO NOT…
?Sven, Kostas & Michele