Statistical Assessment of Event Predictors

Björn Schelter

IWSP 4, Kansas City

Statistical Assessment of Event Predictors and

Probabilistic Forecasting

Björn SchelterAndreas Schulze-Bonhage, Hinnerk Feldwisch,

Michael Jachan, Jens Timmer,

Klaus Lehnertz, Ralph Andrzejak, Florian Mormann

IWSP 4, Kansas City

The guidelines

• Use long-term EEG data without pre-selection

• Report results for training and testing data

• Provide both sensitivity as well as specificity (time under false warning?)

• Statistically validate your results

20.04.23 3IWSP 4, Kansas City

Comparison

IWSP 4, Kansas City 420.04.23

Difference vanishes for Poisson distributed seizures.

Results

• Analytical significance level– Tests statistical significance – Poisson process

• Monte-Carlo based technqiues– Can test statistical significance– Can test for various properties of a given

predictor– Powerful if designed correctly and in its

asymptotic

IWSP 4, Kansas City 520.04.23

What is a true prediction?

20.04.23 6IWSP 4, Kansas City

Standard approach

20.04.23 7IWSP 4, Kansas City

Re-raising alarms

20.04.23 8IWSP 4, Kansas City

Pros and cons

• Standard approach can be assessed statistically

• Re-raising alarms can be handled statistically

BUT

Sensitivity of the random predictor is 100%

20.04.23 9IWSP 4, Kansas City

Solution (?) for re-raising alarms

• By Snyder et al.:– Limit the time under warning– Statistics suggested similar to SPC statistics

• But– Time under true warning might be extremely

long

IWSP 4, Kansas City 1020.04.23

General idea

IWSP 4, Kansas City 1120.04.23

Transform features into probability by logistic regression

Probabilistic Features

t-1 t t+1

IWSP 4, Kansas City

Sensitivity is not an appropriate measure for performance here …

Range of Brier score:

Prediction Performance: The Brier Score

[Brier, Monthly Weather Review 78, 1950]

Question: When can an estimated Brier scorebe regarded as significant ?(i.e. prediction performanceabove chance level)

constant-zero predictor

natural predictor

0.25: indecisive 50% predictor

0: perfect predictor

... Indicator of seizure occurrence (0/1)

... Brier score

IWSP 4, Kansas City

• Correction for multiple testing: 5 tests

MPC: mean phase coherence DSI: dynamical similarity index

• Significant results obtained for 3/5 patients

Results: Brier-Scores

[Andrzejak et al., Phys. Rev. E, 67, 2003]

IWSP 4, Kansas City

The Group

Andreas Schulze-Bonhage

Armin Brandt

Caronlin Gierschner

Jens Timmer

Hinnerk Feldwisch

Michael Jachan

Raimar Sandner

20.04.23 15IWSP 4, Kansas City