Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Background, motivation, and outcomes Methodology Results Conclusion
Evaluation and Ranking of Market Forecasters
Amir Salehipour
ARC DECRA Fellow, University of Technology Sydney
Joint work with David H. Bailey, Jonathan M. Borwein, andMarcos Lopez de Prado
September 29, 2017
Background, motivation, and outcomes Methodology Results Conclusion
Table of Contents
1 Background, motivation, and outcomes
2 MethodologyEvaluation algorithmTraining and testing the algorithm
3 ResultsForecaster accuracyTraders and investors
4 Conclusion
Background, motivation, and outcomes Methodology Results Conclusion
Background, motivation, and outcomes
Motivation
1 Many investors rely on market experts and forecasters when makinginvestment decisions.
2 Ranking and grading market forecasters provides investors with metrics onwhich they may choose forecasters with the best record of accuracy.
Aim of this studyThis study develops a novel ranking methodology to rank the market forecaster;in particular,
1 we distinguish forecasts by their time frame, and specificity, rather thanconsidering all forecasts equally important, and
2 we analyze the impact of the number of forecasts made by a particularforecaster.
Outcomes of this study
1 Across a dataset including 6,627 forecasts made by 68 forecasters, theaccuracy is around 48% implying the majority of forecasters perform atlevels not significantly different than chance.
Background, motivation, and outcomes Methodology Results Conclusion
Forecasts are not reliable enough!
“The forecasts were least useful when they mattered most” [Kaissar].
When analyzing a set of strategists’ predictions from 1999 to 2016, Kaissar foundthat forecasts were surprisingly unreliable during major inflection points [Kaissar]:
1 The strategists overestimated the S&P 500’s year-end price by 26.2% onaverage during the three recession years 2000 to 2002.
2 They underestimated the index’s level by 10.6% for the initial recovery year2003.
3 They overestimated the S&P 500’s year-end level by a whopping 64.3% in2008, but then underestimated the index by 10.9% for the first half of 2009.
Background, motivation, and outcomes Methodology Results Conclusion
Previous ranking
In 2012, the CXO Advisory Group ranked 68 forecasters based on their 6,582forecasts (forecasts made for the S&P 500 index) [CXO1].
That study acknowledged some weaknesses:
All predictions and forecasts were considered equally significant.
The analysis was not adjusted based on the number of forecasts made by aparticular forecaster: some experts made only a handful of predictions, whileothers made many; weighting these the same may lead to distortions whentheir forecasting records are compared.
Background, motivation, and outcomes Methodology Results Conclusion
Our ranking
We extended and advanced the previous ranking (therefore, we used the samedataset).
In particular, the major contributions of our study are:
Investigating in greater detail how market forecasters can be graded andranked;
developing an alternative and comprehensive ranking methodology;
recognizing and prioritizing forecasts by considering different weights forshort- and long-term forecasts, or for important/specific forecasts; and,
investigating and analyzing the most effective and meaningful metrics andmeasures.
Background, motivation, and outcomes Methodology Results Conclusion
Our ranking methodology
1 Part 1:
Every forecast statement is evaluated by calculating the return of the S&P500 index over four periods of one month, three months, nine months, and 12months.
The correctness of the forecast is determined in accordance with the timeframes.
2 Part 2
Each individual forecast is treated according to two factors of time frame andimportance/specificity (not all forecasts are equally important).
Long-term forecasts are treated as more significant than the short-termforecasts (because in the long-term underlying trends, if any, tend toovercome short-term noise; wt ∈ {0.25, 0.50, 0.75, 1.00}).
Specific forecasts are treated more important than non-specific ones(ws ∈ {0.50, 1.00}).
Background, motivation, and outcomes Methodology Results Conclusion
Forecaster’s score
Combined weight for a forecast:
w+i = wt × ws if forecast i is correct
w−i = wt × ws if forecast i is not correct
Where w+i implies when the forecast is true, and w−
i when it is false.
Then, score (accuracy) of a forecaster may be obtained as:
εj =Σ
nji=1w
+i
Σnji=1w
+i + Σ
nji=1w
−i
(1)
where j is the forecaster’s index, and nj is the total number of forecasts made byforecaster j .
Background, motivation, and outcomes Methodology Results Conclusion
Evaluation algorithm
We developed an algorithm, and coded that in the programming languagePython 2.7.
The algorithm reads every record in the dataset, processes the forecaststatements by assigning weights according to the six sets of pre-definedkeywords, and derives the forecasters score.
Keywords: Kt = {{Kt1}, {Kt2}, {Kt3}, {Kt4}}, and Ks = {{Ks1}, {Ks2}},where Kt includes subsets of keywords associated with time frames, and Ks
includes subsets of keywords associated with the importance of the forecasts.
The algorithm analyzes every forecast by applying both sets of keywords tofind any match, and then assigns weights accordingly.
Background, motivation, and outcomes Methodology Results Conclusion
Training and testing
Training the algorithm:
The performance of the algorithm depends on the keywords; we consider aset of 14 forecasters (about 20%) as the training dataset.
We manually analyze and evaluate every forecast in the training set, andcalculate the score of the forecasts.
Then we apply the algorithm to the training dataset, and compare theforecasters’ accuracy obtained by the algorithm against the one obtainedmanually. We update the sets of keywords accordingly.
After several attempts, we obtained an accuracy of 92.16% for thealgorithm’s performance.
Testing the algorithm:
We apply the algorithm to the remaining 54 forecasters, which we call thetesting dataset.
Background, motivation, and outcomes Methodology Results Conclusion
Results – Forecasters accuracy
0
1
2
3
4
5
6
01020304050607080
Abb
y Jo
seph
Coh
enA
den
Sis
ters
Ben
Zac
ksB
erni
e S
chae
ffer
Bill
Car
aB
ill Fl
ecke
nste
inB
ob B
rinke
rB
ob D
oll
Bob
Hoy
eC
abot
Mar
ket L
ette
rC
arl F
utia
Car
l Sw
enlin
Cha
rles
Bid
erm
anC
lif D
roke
Com
stoc
k P
artn
ers
Cur
t Hes
ler
Dan
Sul
livan
Dav
id D
rem
anD
avid
Nas
sar
Den
nis
Slo
thow
erD
on H
ays
Don
Lus
kin
Don
ald
Row
eD
oug
Kas
sG
ary
D. H
albe
rtG
ary
Kal
tbau
mG
ary
Sav
age
Gar
y S
hilli
ngIg
or G
reen
wal
dJa
ck S
chan
nep
Jam
es D
ines
Jam
es O
berw
eis
Jam
es S
tew
art
Jaso
n K
elly
Jere
my
Gra
ntha
mJi
m C
ram
erJi
m J
ubak
Jim
Pup
lava
John
Buc
king
ham
John
Mau
ldin
Jon
Mar
kman
Ken
Fis
her
Lasz
lo B
iriny
iLi
nda
Sch
urm
anLo
uis
Nav
ellie
rM
arc
Fabe
rM
ark
Arb
eter
Mar
tin G
oldb
erg
Mik
e P
aule
noff
Nad
eem
Wal
ayat
Pau
l Tra
cyP
eter
Elia
des
Pric
e H
eadl
eyR
icha
rd B
and
Ric
hard
Mor
oney
Ric
hard
Rho
des
Ric
hard
Rus
sell
Rob
ert D
rach
Rob
ert M
cHug
hR
ober
t Pre
chte
rS
&P
Out
look
Ste
phen
Lee
bS
teve
Sav
illeS
teve
Sju
gger
udS
teve
n Jo
n K
apla
nTi
m W
ood
Tobi
n S
mith
Trad
ing
Wire
%
Total accuracy versus accuracy per forecast and forecast share
Accuracy (This study) Accuracy per forecast Forecast share (no. of forecasts/total forecasts)
Because not every forecaster has made an equal number of forecasts, we analyzedthe accuracy per forecast:
ej =εjnj
(2)
Where, εj is obtained by the algorithm, and nj number of forecasts made byforecaster j .And forecast share of forecaster j :
sj =nj
Σjnj× 100 (3)
Background, motivation, and outcomes Methodology Results Conclusion
Results – Forecasters accuracy vs CXO study
Accuracy gap grasps the changes in the forecasters accuracy between two studies:
-20
-15
-10
-5
0
5
10
15
20
Abby
Jos
eph
Coh
enA
den
Sist
ers
Ben
Zack
sBe
rnie
Sch
aeffe
rB
ill C
ara
Bill
Fle
cken
stei
nB
ob B
rinke
rBo
b D
oll
Bob
Hoy
eC
abot
Mar
ket L
ette
rC
arl F
utia
Car
l Sw
enlin
Cha
rles
Bide
rman
Clif
Dro
keC
omst
ock
Partn
ers
Cur
t Hes
ler
Dan
Sul
livan
Dav
id D
rem
anD
avid
Nas
sar
Den
nis
Slot
how
erD
on H
ays
Don
Lus
kin
Don
ald
Row
eD
oug
Kass
Gar
y D
. Hal
bert
Gar
y Ka
ltbau
mG
ary
Sava
geG
ary
Shi
lling
Igor
Gre
enw
ald
Jack
Sch
anne
pJa
mes
Din
esJa
mes
Obe
rwei
sJa
mes
Ste
war
tJa
son
Kel
lyJe
rem
y G
rant
ham
Jim
Cra
mer
Jim
Jub
akJi
m P
upla
vaJo
hn B
ucki
ngha
mJo
hn M
auld
inJo
n M
arkm
anKe
n Fi
sher
Lasz
lo B
iriny
iLi
nda
Schu
rman
Loui
s N
avel
lier
Mar
c Fa
ber
Mar
k Ar
bete
rM
artin
Gol
dber
gM
ike
Paul
enof
fN
adee
m W
alay
atP
aul T
racy
Pet
er E
liade
sP
rice
Hea
dley
Ric
hard
Ban
dR
icha
rd M
oron
eyR
icha
rd R
hode
sR
icha
rd R
usse
llR
ober
t Dra
chR
ober
t McH
ugh
Rob
ert P
rech
ter
S&P
Out
look
Ste
phen
Lee
bS
teve
Sav
illeSt
eve
Sjug
geru
dSt
even
Jon
Kap
lan
Tim
Woo
dTo
bin
Smith
Trad
ing
Wire
%
Gap = (Accuracy of this study - Accuracy of benchmark)
Positive values reflect improvement in the accuracy over the previous study, andnegative values reflect decreased accuracy.
In particular, 63.24% of forecasters have lower accuracy.
Background, motivation, and outcomes Methodology Results Conclusion
Results – Forecasters time frame and specificity
0%10%20%30%40%50%60%70%80%90%
100%
Abb
y Jo
seph
Coh
enA
den
Sis
ters
Ben
Zac
ksB
erni
e S
chae
ffer
Bill
Car
aB
ill Fl
ecke
nste
inB
ob B
rinke
rB
ob D
oll
Bob
Hoy
eC
abot
Mar
ket L
ette
rC
arl F
utia
Car
l Sw
enlin
Cha
rles
Bid
erm
anC
lif D
roke
Com
stoc
k P
artn
ers
Cur
t Hes
ler
Dan
Sul
livan
Dav
id D
rem
anD
avid
Nas
sar
Den
nis
Slo
thow
erD
on H
ays
Don
Lus
kin
Don
ald
Row
eD
oug
Kas
sG
ary
D. H
albe
rtG
ary
Kal
tbau
mG
ary
Sav
age
Gar
y S
hilli
ngIg
or G
reen
wal
dJa
ck S
chan
nep
Jam
es D
ines
Jam
es O
berw
eis
Jam
es S
tew
art
Jaso
n K
elly
Jere
my
Gra
ntha
mJi
m C
ram
erJi
m J
ubak
Jim
Pup
lava
John
Buc
king
ham
John
Mau
ldin
Jon
Mar
kman
Ken
Fis
her
Lasz
lo B
iriny
iLi
nda
Sch
urm
anLo
uis
Nav
ellie
rM
arc
Fabe
rM
ark
Arb
eter
Mar
tin G
oldb
erg
Mik
e P
aule
noff
Nad
eem
Wal
ayat
Pau
l Tra
cyP
eter
Elia
des
Pric
e H
eadl
eyR
icha
rd B
and
Ric
hard
Mor
oney
Ric
hard
Rho
des
Ric
hard
Rus
sell
Rob
ert D
rach
Rob
ert M
cHug
hR
ober
t Pre
chte
rS
&P
Out
look
Ste
phen
Lee
bS
teve
Sav
ille
Ste
ve S
jugg
erud
Ste
ven
Jon
Kap
lan
Tim
Woo
dTo
bin
Sm
ithTr
adin
g W
ire
Percentage of forecasts time frames
% of forecasts with weight 1.00 % of forecasts with weight 0.75 % of forecasts with weight 0.50 % of forecasts with weight 0.25
0%10%20%30%40%50%60%70%80%90%
100%
Abb
y Jo
seph
Coh
enA
den
Sis
ters
Ben
Zac
ksB
erni
e S
chae
ffer
Bill
Car
aB
ill F
leck
enst
ein
Bob
Brin
ker
Bob
Dol
lB
ob H
oye
Cab
ot M
arke
t Let
ter
Car
l Fut
iaC
arl S
wen
linC
harle
s B
ider
man
Clif
Dro
keC
omst
ock
Par
tner
sC
urt H
esle
rD
an S
ulliv
anD
avid
Dre
man
Dav
id N
assa
rD
enni
s S
loth
ower
Don
Hay
sD
on L
uski
nD
onal
d R
owe
Dou
g K
ass
Gar
y D
. Hal
bert
Gar
y K
altb
aum
Gar
y S
avag
eG
ary
Shilli
ngIg
or G
reen
wal
dJa
ck S
chan
nep
Jam
es D
ines
Jam
es O
berw
eis
Jam
es S
tew
art
Jaso
n K
elly
Jere
my
Gra
ntha
mJi
m C
ram
erJi
m J
ubak
Jim
Pup
lava
John
Buc
king
ham
John
Mau
ldin
Jon
Mar
kman
Ken
Fis
her
Lasz
lo B
iriny
iLi
nda
Sch
urm
anLo
uis
Nav
ellie
rM
arc
Fabe
rM
ark
Arb
eter
Mar
tin G
oldb
erg
Mik
e P
aule
noff
Nad
eem
Wal
ayat
Pau
l Tra
cyP
eter
Elia
des
Pric
e H
eadl
eyR
icha
rd B
and
Ric
hard
Mor
oney
Ric
hard
Rho
des
Ric
hard
Rus
sell
Rob
ert D
rach
Rob
ert M
cHug
hR
ober
t Pre
chte
rS
&P O
utlo
okS
teph
en L
eeb
Ste
ve S
avill
eS
teve
Sju
gger
udS
teve
n Jo
n K
apla
nTi
m W
ood
Tobi
n S
mith
Trad
ing
Wire
Percentage of specific forecasts versus non-specific forecasts
% of specific forecasts % of non-specific forecasts
Background, motivation, and outcomes Methodology Results Conclusion
Results – Traders and investors
Long-term forecasters or “investors”, who have at least 30% of the forecasts withweights 0.75 and 1.00, and short-term forecasters or “traders”, with the majorityof the forecasts with weights 0.25 and 0.50.
We observed no forecaster has 50% or more of his forecasts with weights greaterthan 0.50.
Background, motivation, and outcomes Methodology Results Conclusion
Results – Traders and investors
Investors
0
10
20
30
40
50
60
0
10
20
30
40
50
60
Ken
Fis
her
Jam
es O
berw
eis
Pau
l Tra
cy
Ste
phen
Lee
b
Clif
Dro
ke
Gar
y D
. Ha
lber
t
Jere
my
Gra
ntha
m
Jim
Cra
me
r
Abb
y Jo
seph
Co
hen
Cur
t H
esl
er
Ste
ven
Jon
Kap
lan
Ste
ve S
avi
lle Per
cen
tage
of
fore
cast
s
Acc
urac
y
Accuracy of forecasters in the investor group
Accuracy (This study) Percentage of forecasts with weights > 0.5
Traders
0
20
40
60
80
100
01020304050607080
John
Buc
king
ham
Jack
Sch
anne
p
Dav
id N
assa
r
Dav
id D
rem
an
Cab
ot M
arke
t Let
ter
Loui
s N
avel
lier
Lasz
lo B
iriny
i
Stev
e S
jugg
erud
Rob
ert D
rach
Jaso
n Ke
lly
Bob
Dol
l
Dan
Sul
livan
Ade
n Si
ster
s
Don
Lus
kin
Ben
Zack
s
Gar
y K
altb
aum
Ric
hard
Mor
oney
Igor
Gre
enw
ald
Tobi
n S
mith
Car
l Sw
enlin
Mar
k Ar
bete
r
Ric
hard
Rho
des
Car
l Fut
ia
Don
Hay
s
Jam
es S
tew
art
Trad
ing
Wire
S&P
Out
look
Bob
Brin
ker
Pet
er E
liade
s
Jon
Mar
kman
Mar
tin G
oldb
erg
Jam
es D
ines
Cha
rles
Bid
erm
an
Den
nis
Slot
how
er
Bill
Car
a
Tim
Woo
d
Ber
nie
Scha
effe
r
Lind
a S
chur
man
Ric
hard
Ban
d
Don
ald
Row
e
Pric
e H
eadl
ey
Dou
g Ka
ss
Gar
y S
avag
e
Mar
c Fa
ber
Jim
Jub
ak
Ric
hard
Rus
sell
John
Mau
ldin
Nad
eem
Wal
ayat
Gar
y S
hillin
g
Jim
Pup
lava
Bill
Flec
kens
tein
Com
stoc
k Pa
rtner
s
Bob
Hoy
e
Rob
ert M
cHug
h
Mik
e P
aule
noff
Rob
ert P
rech
ter Pe
rcen
tage
of f
orec
asts
Accu
racy
Accuracy of forecasters in the trader group
Accuracy (This study) Percentage of forecasts with weights <= 0.5
Background, motivation, and outcomes Methodology Results Conclusion
Results – Summary
Major finding: the majority of forecasters perform at levels not significantlydifferent than chance, which makes it very difficult to tell if there is any skillpresent.
Across all forecasts, the accuracy is around 48%.
Two-thirds of forecasts predict as far as only a month.
Only one-third of forecasts predict periods over one month.
Two-thirds of forecasters have an accuracy level below 50%.
Only about 6% of forecasters have their accuracy values between 70%and 79%; the highest accuracy value is still below 80%.
Background, motivation, and outcomes Methodology Results Conclusion
Publication and awards
David H. Bailey, Jonathan M. Borwein, Amir Salehipour, and Marcos L opezde Prado. “Evaluation and ranking of market forecasters”. In: Journal ofInvestment Management. Forthcoming.
Awarded “Silver Bullet” award (20 May 2017).
Among the most viewed papers on the SSRN (www.ssrn.com) with a total2,762 views within six months (28 September 2017).
Background, motivation, and outcomes Methodology Results Conclusion
Bibliography
Terry Burnham, “Ben Bernanke as Easter bunny: Why the Fed can’t prevent the coming crash”,PBS NewsHour, 11 July 2013, available at http://www.pbs.org/newshour/making-sense/ben-bernanke-as-easter-bunny-why-the-fed-cant-prevent-the-coming-crash/.
Terry Burnham, “Why one economist isn’t running with the bulls: Dow 5,000 remains closer thanyou think”, PBS NewsHour, 21 May 2014, available at http://www.pbs.org/newshour/making-sense/one-economist-isnt-running-bulls-dow-5000-remains-closer-think/.
Nir Kaissar, “S&P 500 forecasts: Crystal ball or magic 8?,” Bloomberg News, 23 December 2016,available at https://www.bloomberg.com/gadfly/articles/2016-12-23/s-p-500-forecasts-mostly-hit-mark-until-they-matter-most.
Steve LeCompte, editor, “Guru grades”, CXO Advisory Group, 2013, available athttp://www.cxoadvisory.com/gurus/.
Steve LeCompte, editor, “The most intriguing gurus?”, CXO Advisory Group, 2009, available athttp://www.cxoadvisory.com/4025/investing-expertise/the-most-intriguing-gurus/.
Miles Udland, “Here’s what 13 top Wall Street pros are predicting for stocks in 2015”, BusinessInsider, 3 January 2015, available athttp://www.businessinsider.com/wall-street-2015-sp-500-forecasts-2015-1.
Minitab Inc, “Minitab 17 Statistical Software”, www.minitab.com, 2015, www.minitab.com.