CSEP Overview and Status

Max Werner!

z

T. Jordan, M. Liukis, P. Maechling, D. Schorlemmer, J. Zechar and many more

Collaboratory for the Study of Earthquake Predictability!

•  CSEP goal is an infrastructure for conducting earthquake predictability experiments and research. This entails:

–  Rigorous procedures for registering forecasting and prediction experiments

–  Reproducible evaluations of predictability hypotheses and forecasting models

–  Automated, blind, prospective testing in a standardized, controlled environment (“zero degrees of freedom”)

–  Community-endorsed standards for assessing forecasts & predictions

–  Experiments in a variety of tectonic environments

•  Why? –  Understand earthquake predictability, brick-by-brick

–  Reduce controversies surrounding earthquake prediction

–  Help government agencies in assessing the utility of earthquake forecasts and predictions in the context of risk reduction.

Jordan (2006)

Los Angeles

Zurich

Tokyo

Wellington

GNS Science Testing Center

ERI Testing Center

EU Testing Center SCEC

Testing Center

Testing Center

Upcoming

Beijing

China Testing Center

Collaboratory for the Study of Earthquake Predictability!Infrastructure for automated, prospective assessment of forecasting models in a variety of tectonic environments

CSEP Testing Centers

Los Angeles

Zurich

Tokyo

Wellington

GNS Science Testing Center

Japan 203 models

ERI Testing Center

Italy 48 models

EU Testing Center

California 64 models

SCEC Testing Center

Testing Center

Upcoming

Testing Region

Upcoming

Global 9 models

Beijing

China Testing Center

North-South Seismic Belt

New Zealand 85 models

CSEP Testing Regions & Testing Centers 434 models under test

in September, 2014

Collaboratory for the Study of Earthquake Predictability!

Western Pacific 16 models

Infrastructure for automated, blind, prospective assessment of forecasting models in a variety of tectonic environments

Oceanic Transform Faults 1 model

Models under Evaluation!434

Darfield

El Mayor

Tohoku

RELM Regional Earthquake Likelihood Models PPE Proximity to Past Earthquakes TripleS Simple Smoothed Seismicity EEPAS Every Earthquake a Precursor According

to Scale

STEP Short-Term Earthquake Probabilities ETAS Epidemic-Type Aftershock Sequences DBM Double Branching Model K3 Kernel-based space-time-magnitude

smoothing

Example models:

Status Updates!

•  Multiplicative hybrid RELM models (California) •  3-month model evaluation (California) •  1-day model evaluation (California) •  1-year global model challenges (today) •  A teaser result from the Canterbury experiment (today) •  Prototype external forecast & prediction (EFP) experiment

for M8

Original models Transformation Hybrid model

€

f (λ) = exp[a + b log(1+ λ)( )c ]

order-preserving function, e.g.

D. Rhoades, M. Gerstenberger, A. Christophersen, J. Zechar, D. Schorlemmer, M. Werner & T. Jordan (2014, BSSA, accepted)

Models for SoCal

Target earthquakes: Mainshocks + Aftershocks

Some hybrids show better performance."(penalized for extra parameters)!

Also: best inf gain since 2011/11 (5 eqks. out-of-sample)

Time-series of T- and W-test results, with EEPAS-0F as the reference model.

Schneider M et al. Geophys. J. Int. 2014;198:1307-1318

© The Authors 2014. Published by Oxford University Press on behalf of The Royal Astronomical Society.

3-month model evaluation California!

•  4 new models since 10/2012 •  Conan (adaptive space-time kernels) •  Janus (hybrid model) •  EAS (early-aftershock)

•  6 earthquakes M4.95+

•  No significant differences yet

Testing region: California Forecast model: Conan

(Helmstetter & Werner, 2012)

Testing period (retrospective): 10/2012 – 7/2014 Target events: M ≥ 4.95 (6)

Conan (adaptive space-time smoothing)

Evaluating Conan in Japan (VISES)!

Jan10 Apr10 Jul10 Oct10 Jan11 Apr11 Jul11 Oct11100

101

102

103

Num

ber o

f M3.

95+

ForecastObserved

Jan10 Apr10 Jul10 Oct10 Jan11 Apr11 Jul11 Oct114

5

6

7

8

Mag

nitu

de

130˚ 132˚ 134˚ 136˚ 138˚ 140˚ 142˚ 144˚ 146˚

32˚

34˚

36˚

38˚

40˚

42˚

44˚

130˚ 132˚ 134˚ 136˚ 138˚ 140˚ 142˚ 144˚ 146˚

32˚

34˚

36˚

38˚

40˚

42˚

44˚

−4 −3 −2 −1 0

log10 expected earthquakes M3.95+ per 3 months per 0.1o 2

Retrospective comparison with extent CSEP-Japan models in progress

Werner, Tsuruoka, Yokoi, Helmstetter & Hirata

Pearson residuals for the EEPAS-0R model in the aftershock region of the El Mayor earthquake, computed for the El Mayor period and subsequent eight periods.

Schneider M et al. Geophys. J. Int. 2014;198:1307-1318

Residuals-based Evaluation "

Next-day M3.95+ experiment in CA+

Zechar, 2014

ETAS 1.0 vs. the rest

Kagan-Jackson

STEP Java

ETAS 1.1 (Zhuang)

ETAS (H&W)

(ETAS+K3)/2

ETAS (Rhoades)

K3

ETAS/PPE (Rhoades)

Information gain per earthquake

2012-10-01 to 2014-03-01 62 earthquakes M4+

Zechar, 2014

KJSS (Kagan & Jackson)

Zechar, 2014

Sub-24-hour Earthquake Predictability!

1-day models

Testing region: California Forecast model: ETAS (Werner, Helmstetter, Jackson & Kagan, 2014)

Testing period (retrospective): 1992 – 2012 Target events: M ≥ 3.95 (1396) Reference model: spatially uniform Poisson

Evaluations of Global Models!Testing region: Global M5.95+ <70km depth (high-resolution, 0.1 deg) Forecast model: SHIFT_GSRM Global Strain Rate Map & SHIFT (Bird & Kreemer, 2010) Testing period: 1/1/2014 – 8/7/2014

N-test

•  3 high-resolution (0.1 deg) models, more on the way (GEAR). •  forthcoming binary format required for likelihood tests & information gains. •  new models should agree on common depth and magnitude thresholds.

RETROSPECTIVE EXPERIMENT!2010-12 Canterbury, NZ

Joint SCEC/REAKT/GNS project •  Improve understanding of earthquake triggering •  Improve time-dependent hazard estimates •  Understand influence of “real-time” data on forecasts

Participating Models!

Coulomb Steacy, Gerstenberger Cattania et al.

STEP/ETAS Helmstetter & Werner Hainzl et al.

Smoothing Helmstetter & Werner

Installed Models!

Reference Models Uniform Poisson model

Testing region: Canterbury Forecast model: CRS1

Coulom/Rate-State w/ uncertainties (Cattania et al., 2014) Testing period: From 7.1 Darfield eqk. to 2/28/2012

Coulomb/Rate-State w uncertainties

Information gain

External Forecasts & Predictions!

M8

Kossobokov, Liukis, Rierola, Zechar

Results CSEP

Example of M8 TIPS

This is an example for the TIP alarms. This alarm was released on1.1.1985.

−100 0 100

−100

−50

050

100

M8 TIP 1985a

Pos. TIPNeg. TIPNo Data

Green circles representTIPS where an alarm hasbeen set: ”Yes”

Red circles representTIPS where no alarm hasbeen issued: ”No”

Blue circles show TIPSwhere not enough datawas available. ThoseTIPS have are not beingconsidered in the scoring.

August 19, 2014 1 / 3

Rierola & Zechar, 2014

Delta R Score for fixed odd gambling

The Delta R gambling score approach has been introduced by Zhuangand Zechar in 2010.

The Delta R gambling score considers an earthquake forecast liketaking a bet in a casino. The gambler, i.e. M8 is placing a betagainst the house. For each forecast the gambler (M8) makes, heplaces one reputation point.

The house in this case is a Poisson reference model for which wecalculated historic rates based on the ANSS catalog.

The Delta R gambling score allows for unbounded (∞) winnings whilethe losses per bet cannot be more than 1 by definition.

August 19, 2014 2 / 3

Rierola & Zechar, 2014

Next steps and CSEP integration

Because of the asymmetrical behavior of the Delta R gambling scorewe believe that it is not the best measure for earthquake forecastmodels.We will therefore extend our research and score the forecasting modelusing the pari-mutuel gambling score.Currently everything is coded locally in R. We hope to be able tomove the code into CSEP by the end of this year.

0 2000 4000 6000 8000 10000

0.0

0.2

0.4

0.6

0.8

1.0

ECDF

change in reputation

fract

ion

of s

imul

atio

ns

August 19, 2014 3 / 3

Rierola & Zechar, 2014

Frac

tion

of s

imul

atio

ns

1.0

0

Pessimist M8 Poisson

returns

Summary!

•  Hybrid & ensemble modeling help construct better models.

•  Residuals are an interesting visual aid in evaluating models.

•  1-day model evaluation in California suggests real differences.

•  30-minute forecast group open for business.

•  Global forecasts require resources and consensus.

•  Interesting results from the retrospective Canterbury experiments suggest Coulomb models improving.

•  EFP experiments (M8 & QuakeFinder) pose interesting questions about gambling scores.

CSEP Structure!

Forecast Model 1

Forecast Model N

Testing Procedures

Authoritative Data Source A

Authoritative Data Source B

Authoritative Eqk Catalog

Results

CSEP

Data R

egistry

…

Forecast Registry

External Forecasting Procedure

Special Data Source

Accommodation of External Forecasting

Original models Transformation Hybrid model

€

f (λ)

€

λ

D. Rhoades, M. Gerstenberger, A. Christophersen, J. Zechar, D. Schorlemmer, M. Werner & T. Jordan (2014, BSSA, accepted)