Key Performance Indicators, Centre Reports, and more

Stephen McDonald

Barbecue talk

20

40

60

80

100

120

per

mill

ion

per

yea

r

1970 1980 1990 2000 2010Year

Rate 95% CIIncident rate

Incident RRT, Australia only

More “good” news

5

10

15

20

30

40

50

60

70

50

100

150

200

0

500

0

200

400

600

0

100

200

300

1980 1990 2000 2010 1980 1990 2000 2010 1980 1990 2000 2010

0-24 25-34 45-54

65-74 75-84 85+

Rate 95% CI

per

mill

ion

per

yea

r

Year

Graphs by age group

AustraliaAge-specific incident RRT rates

Indigenous incidence rates

0

100

200

300

400

500

Per

mill

ion

per y

ear

1985 1990 1995 2000 2005 2010Year

Rate 95% CI

Aboriginal & TSI, Australia

Background

• A number of ongoing work themes exist within ANZDATA for generating output– Stock and flow figures– Annual Report– Contributor requests

• Responses to information needed for various projects

– Research projects (internal and external analyses)

– Outcomes reporting

Outcomes reporting

• Recent years have seen a growth of interest in outcomes reporting

• Centre reports have been part of ANZDATA for many years, with increasing emphasis in recent years– At “parent hospital level”– Limited distribution historically

Why measure outcomes?

.2

.5

1

2

4

Obs

erv

ed /

Exp

ecte

d m

ort

alit

y

0 20 40 60 80units

O/E 98% CI

All Australia & NZ Dialysis Units, 98% confidence intervals

Dialysis outcome

.2

.5

1

2

4

Adj

ust

ed r

ela

tive

risk

0 20 40 60 80

Units, ranked by RR

RR 95% CI

Mortality rate during dialysis treatment in Australia 2006-10, adjustedfor demographics and comorbidities

Variation in transplant outcomes

.25

.5

1

2

5

10

20

50R

R g

raft

failu

re

0 5 10 15 20Units

RR 95% CI

Fully adjusted 1 year graft survival, by unitAll transplant units, Australia and New Zealand, patients transplanted2005-2019, followup to 2010

What is happening to centre reports?

• Greater reporting of demographics and comorbidities

• Adjusted analyses in transplanting centre and dialysis reports– Details of models supplied

• Graphs– Funnel plots– CUSUM plots (transplant)

Centre reports – graph 1

0.00

0.25

0.50

0.75

1.00

Pat

ient

Sur

viva

l

0 1 2 3 4 5Years

CNARAustralia

New Zealand

Survival from 90th Day of Treatment

Centre reports – graph 2

0.00

0.25

0.50

0.75

1.00

Tech

niqu

e S

urvi

val

0 1 2 3 4 5Years

CNARAustralia

New Zealand

Technique Survival - PD at 90 days

But....

0

20

40

60

80

100

Ag

e (o

f pre

vale

nt p

atie

nts)

Everywhere else CNARTS

Adjusted graphs

.7

.8

.9

1

1.1

1.2

1.3

Adj

uste

d S

MR

CNAR Australia New Zealand

Adjusted SMR (95% CI)

Adjusted graphs

CNAR

0

.5

1

1.5

2

2.5

SM

R

0 50 100 150 200Expected Number of Deaths

How are reports derived?

You need a model

• Logistic regression model (transplant), Poisson model (dialysis)

• Adjusted for demographics, comorbidities (donor and XM variables)

• With this model, derive a probability of “expected” failure for each person / graft based on covariate matrix

• Compare this with actual outcomes

www.anzdata.org.au

Which predictors are important?

Recip

ient

age

gen

der &

gra

ft num

ber

+com

orbi

ditie

s

+ HLA

mat

chin

g

+ isc

haem

ic tim

e

+ do

nor a

ge

+ ca

use

dono

r dea

th0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Harrell's CSomer's D

Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009, with sequential addition of covariate groups

www.anzdata.org.au

• Factors within the control of centre– These may be why a particular centre gets

good or bad results• Factors that occur as a result of treatment

decisions• For example, don’t adjust for

– Choice of dialysis modality, HD access– Use of immunosuppressives, rejection, 1

month graft function…

Don’t adjust for…

Other graphical demonstrations of output

• Funnel plots are a static measure and summarise performance (relative to a comparator) over a fixed period of time.– Lack a dynamic element– Weight recent and distant results equally

Adding time – CUSUM

0

100

200

300

400

Num

ber

of tx

-4

-2

0

2

4

Cum

ulat

ive

sum

O-E

01jan2004

01jan2005

01jan2006

01jan2007

01jan2008

01jan2009

Tx date

Twoway CUSUM for a transplant centre

Removing credit for good deeds

0

1

2

3

4

5

Cu

mul

ativ

e s

um

O-E

0 100

200

300

400

Tx number

Oneway CUSUM for for a hospital

Do we need to do more?

Why KPIs?

• Mortality is an insensitive and late indicators of problems– Hopefully rare– Outcome of complex series of events

• Incompletely ascertained

– Important to monitor as best we can• Key Process indicators

– Simpler to understand, easier to address– Need to be valid and correctable (and related

to meaningful outcomes)

KPI Project

• Dialysis KPI project commenced 2011 – At instigation of DNT committee

• 2 markers chosen – Peritonitis and HD access at first treatment– Deliberately limited to existing data collection

• NO additional data collected

– Based on real time ANZDATA data collection

Variation in HD access

8

5 5

40

31

65

18

66

28

10

57

2522

113

17

3668

810

15

7

47

19

12

2952

75434452

29

11

34344144

12

4464

20

1017

3144331923

139

2815

54

6 8

0

.2

.4

.6

.8

1

Pro

por

tion

of li

nes

0 20 40 60Centres

Proportion 95% CI

ANZDATA, access at first HD where first dialysis

Variation in peritonitis rate

27

24

12

6

4

Pat

ient

-mon

ths

per

epi

sod

e

.5

1

1.5

2

2.5

3

Epi

sod

es p

er

pat

ient

-ye

ar

Confidence intervals not shown where upper limit >3Units with <5 person-years PD over 2009 not shown

2009 onlyPeritonitis rates by treating unit

KPI reporting -- access

• Quarterly identified feedback to units

Peritonitis reporting

Where to from here?

• COMMUNICATE• Improve data collection• Improve access to results• Enhance reporting

– Add peritonitis rates– Access subdivided by late referral– Graphs etc etc

• Or is it all just too hard?

How do we view quality?

Centre reports -- SMR