Risk scoring

Allan Wardhaugh

A tale of two units

Holby City, Holby

Sacred Heart

PICU mortality last year

Holby City500 admissions15 deaths

Sacred Heart1200 admissions90 deaths

‘Crude’ mortality

Holby City 3%

Sacred Heart 7.5%

What else do we need to know?

How sick were the patients?Did the patients at Sacred Heart undergo more complex procedures?Does Holby City send sick patients away?Does Holby City discharge patients to die somewhere else?What is the mortality in other units?

Case Mix

Standardised Mortality Ratio

Measured mortalityPredicted mortality – risk adjustment tool

SMR = Measured/ Predicted

∴ SMR > 1 performing poorlySMR < 1 performing well

Predicted mortality

Historical dataNational ‘average’

What about case mix?

Risk adjustment

Predicts mortality for a population based on a statistical model derived from a similar populationExamples

APACHE (Acute Physiology and Chronic Health Evaluation )

CRIB (Clinical Risk Index for Babies)

PRISM (Pediatric RISk of Mortality)

PIM (Paediatric Index of Mortality)

Regression statistics

Target variable – ‘dependent variable’Predictors – ‘independent variables’Regression statistics use association between variables to predict one (DV) from another (IV).Simplest form y = b0 + b1(x)where y = predicted value, b0= regression constant, b1= regression coefficient

Multiple regressiony = b0+b1(x1)+b2(x2)+…bn(xn)

PICU

Mortality is target variable

Predictor variables?

Example

1000 patients 100 died900 survivedProbability of death 0.1

Odds of death 0.11Admitted electively

10 of 100 deaths500 of 900 deathsProbability of death 0.02

Odds of death 0.02

Probability and Odds

ppOdds−

=1

Probabilty = 0 – 1

Odds = 0 - ∞

Regression statistics - logistic

For non-quantitative DV (e.g. dead/alive), logistic regression is usedFor each IV, odds are calculated for likelihood of having DVOdds assymetrical

very small number (0 – 1) if event unlikelyvery large if event likely (>1 - ∞)

Rectified by using natural log of odds – called logit – makes it a linear function

Regression statistics - logit

Log odds = b0+b1(x1)+b2(x2)+…bn(xn)

Probability = odds/(1 + odds)Logit = ln odds

logit

logit

e1ep+

=

PRISM

Pediatric Risk of Mortality Score14 physiological variables

Worst measurement in first 24 hoursNow on PRISM III – relies on scores in first 12 or 24 hours

Probability of PICU death= eR/1 + eR

Where R = 0.207 × PRISM – 0.005 × age(mo) – 0.433 × operative status – 4.782

PRISM III

PRISM – example60 month old non-surgical patient

PRISM Score Mortality Risk (%)

3 1.6

6 2.7

9 5

12 8.9

15 15.3

18 25.2

21 38.6

24 46.1

27 68.5

30 80.2

PRISM - disadvantages

Data collection cumbersome (14 variables over a 24 hour period)May diagnose death rather than predict it (40% deaths occur in first 24 hours)Score may not allow comparison between units – patients poorly managed in first 24 hours will develop high PRISM score, so disease severity will appear to be greater

PIM – Paediatric Index of Mortality –initial cohorts

678 consecutive admissions PICU RCHM 1988814 consecutive admissions RCHM 19901412 consecutive admissions 1994–5 RCHM

PIM – identifying variables

Data collected for admission (for most) and first 24 hours34 Physiological Stability Index measurementsMAP, PIP, PEEP, and othersWorst value in first 24 hours for all

PIM – derivation of model

All PRISM data collected plus additional informationUnivariate analysis carried out on all factors to test for association with mortalityFactors not associated (p>0.1) excluded from further analysisLogistic regression analysis used to derive preliminary model.

PIM – testing the model

Learning and Test cohorts1994 – 96 5695 patients in 8 PICUs (Australia, Birmingham)

Enough patients in each unit to include 20 deaths.Learning sample data analysed to calculate regression coefficientsModel then tested on test sample, and examined for goodness of fit.Regression coefficients re-estimated using all 8 units for final model.

PIM - coefficients

PIM – final equations

elogit/(1+elogit)

Logit = (2.357.pupils)+(1.826.specified diagnosis)+(–1.552.elective admission)+(1.342.mechanical ventilation)+(0.021.(SBP–120))+(0.071.Baseex)+(0.415.(100.FiO2/PaO2))–4.873

PIM - recalibration

PICU outcomes change with timeReferral patterns change with timeAttitudes to withdrawing and limiting care may change with time

Calibration findings

Specific diagnosisResp illness O:E 160:212Non-cardiac post-op O:E 48:82

293 coded diagnostic categories examinedIn-hospital cardiac arrest associated with increaedrisk of deathAsthma. Bronchiolitis, croup, obstructive sleep apnoea, DKA associated with reduced risk

New ‘high risk’ and ‘low risk’ categories introducedPost – op subdivided into with or without CBP.IQ <35 omitted (difficult to code reliably)

SMR

Australia and New Zealand SMR 0.84 (0.76–0.92) UK 0.89 (0.77–1.00).

PRISM vs PIM

PRISM predicted 66% more deaths in this sampleScore altered by treatment in the first 24 hoursMay diagnose rather than predict deathPRISM III data requires 96 measured variablesLicense requiredNote that neither are adequate for individual case prediction – apply to populations only

United KingdomPaediatric Intensive Care Outcome Study

UK PICOS

PIM mortality ratio (observed/expected unit deaths) by unit. Generated using UK PICOS recalibration

.5

1

2M

orta

lity

ratio

200 400 600 800 1000Number of admissions

Your unit Other units Control limits

Mortality ratio calculated using the UK PICOS calibration of PIM in the UK.

Upper and lower control limits represent a 99.9% confidence interval around a mortality ratio of 1 based on the UK PICOS overall mortality of 6.2%..

Outcome

PRISM III 24 hour score re-calibrated for UKPerformance of PIM-2 and PRISM III very similarPIM – 2 recommended as model of choice as data easier to collect

DoH/ WAG fundedRun from Universities of Sheffield, Leicester and LeedsFirst annual report March 2003 – February 2004

2009-11 Unadjusted

2009-11 adjusted (PIM-2R)

League tables

Governments like themJournalists like themLocal politicians like themPatients groups like them

Do any of the above understand them?

9 NICUs over 6 yearsCrude and risk adjusted (CRIB score) mortalityHospitals ranked in league tables each year according to W score

W= 100 × (observed - expected deaths)/No of admissions.Mortality lower than expected if W < 0

Results

Conclusions

Hospitals varied annually in their league positionConfidence intervals for W scores overlapped for all hospital every year except year 3‘Overall, hospital 1 did perform significantly better than expected but it is debatable whether this makes it a model hospital since its performance was inconsistent’.

Summary

PIM/ PIM 2 data easy to collect Useful in comparing unit performanceInterpret with care if number of deaths low (especially <20).Not for use as an individual prediction testImportant to complete as accurately as possiblePICANET randomly check to ensure data qualityLeague tables are unreliable