Prognostic models in the ICU

From development to clinical practice

L. Minne, MSc.Dr. S. Eslami, PharmD

Dr. D.A. Dongelmans, MDProf. Dr. S.E.J.A. de Rooij, MD

Prof. Dr. A. Abu-Hanna

Dept. of Medical InformaticsDept. of Intensive Care

Academic Medical CenterAmsterdam, the Netherlands

Prof. Dr. E. de Jonge, MDDept. of Intensive CareLeiden University Medical CenterLeiden, the Netherlands

Use of prognostic models

1) Benchmarking

2) Decision-making

Expected mortality: 30% 12%

SMR: 0.83 1.25

Hospital 1 Hospital 2

Observed mortality: 25% 15%

Estimates from prognostic model

Use of prognostic models

Your probability to survive is: -7.7631 + (SAPS II score * 0.0737) + (0.9971 * (ln (SAPS II score + 1)))

1) Benchmarking

2) Decision-making

Barriers for use in clinical practice Lack of evidence for:

External validity Clinical credibility Impact on decisions and patient outcomes

Selffulfilling prophecy

Population level vsindividual level

Overview of our research project

1)1) IdentifyIdentify prognostic models, their validity and use in clinical practice

2) Assess prognostic model behaviour over time + effects on benchmarkingbenchmarking

3) Assess clinicians’ predictions, (need for) prognostic models, their validity and impact in decision-makingdecision-making

Red (critical) zone

Yellow (warning) zone

Green (safe) zone

Green (safe) zone

Yellow (warning) zone

Red (critical) zone

Time

Sta

nd

ardi

zed

Mo

rtal

ity R

atio

Mean value

Upper control limit (usually at 3 sigma)

Lower control limit (usually at 3 sigma)

> mean + 4 sigma

mean + 2 sigma : mean + 4 sigma

mean : mean + 2 sigma

mean : mean - 2 sigma

mean - 2 sigma : mean - 4 sigma

< mean - 4 sigma

Benchmarking – Temporal validation

Benchmarking – Temporal validation

SMR > 1 in 15% of the hospitals

Benchmarking – Temporal validation

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Timep=16Data used for recalibration p=19Data used for recalibration

Effect of continuous updating (first level recalibration)

Benchmarking – Temporal validation

SMR > 1 in 35% of the hospitals effect on quality of care assessment!

AgeGender...

Demography Physiology Laboratory ...

Admission

Mortality(Length of Stay)(...)

Outcomes

organ scores day1organ scores day2organ scores day3

…

During Stay

SAPS score

SOFA

Decision-making – Model development

25

Day 3Day 2

0

4

1

1

3

3

Day 4Day 1 Day 5

4

2

0

3

0

3

3

0

0

3

0

3

1

0

0

4

0

4

1

0

0

3

0

4

Renal

Hepatic

Circulatory

Respiratory

Neurological

Coagulation

SAPS

998SOFA score 12 12

Decision-making – Model development

HHM H H

d = 3

LP = a0 + a1SAPS + a2admission_type + a3day +A4number_of_readmissions +

+ b1 Pattern1 + b2 Pattern2 + …

Example at day 3

LP = -9.3 +0.005*SAPS -0.034*3 + 1.23*2 +1.85 SOFA{H,H} + 1.1 SOFA{M,H,H}

Decision-making – Model development

Decision-making – Model performance

Decision-making – Model performance

Decision-making – The end-of-life decision-making process

Observation of multidisciplinary meetings poorly structured no clear guidelines

Factors (implicitly) considered in decision: Degree of organ failure Patient preferences Severity of illness Chance of cognitive limitations

Wish to receive objective information

Conclusions and future work

Decision-making process unstructured

Possible role for mathematical models

But… insufficient evidence on their impact and external validation

Before-after study to measure impact on decision-making

Any questions?

Decision-making – Human predictions

Kappa = 47.3-55.1%

NursesNurses PhysiciansPhysicians

AUCAUC 0.89 0.88

VarVar 6-7% 7-8%