Clinical Prediction Rules

Jen-Hsiang Chuang, MD, MS, PhDCenters for Disease Control Taiwan

jhchuang@cdc.gov.tw

1

Clinical Prediction Rules (CPRs)

• Synonym: clinical decision rules

• Definition: decision-making tools for clinicians including 3 or more variables– Provide the probability of an outcome– Suggest a diagnostic or therapeutic course of

action

Laupacis A, et al. Clinical prediction rules. JAMA 1997;277:488-494. 2

Clinical Prediction Rules Vs. Clinical Practice Guidelines

• Clinical prediction rules– Derived from original research involving many

patients and mathematical analysis

• Clinical practice guidelines– Consensus among experts– GOBSAT (Good Old Boys Sat At Table)

(Miller J, et al. Lancet 2000;355:82-3)– But can include CPRs

3

Functions of CPRs

• CPRs help clinicians cope with uncertainty and improve efficiency– Cope with uncertainty

• Community-acquired pneumonia (Fine MJ, et al. NEJM 1997;336:243-250)

– Improve efficiency• Ottawa Ankle Rules for the use of

radiography (Stiell IG, et al. Ann Emerg Med 1992;21:384-90)

Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 4

Prototype of a CPR for Predicting Death

Predictor variables ScoreAge > 75 yr 6Severe pain 10Emergency 5

Total points 0-21

Interpretation of the scoreHigh risk: > 6 points (30% deaths) -> aggressive TxLow risk: 6 points (3% deaths) -> conservative Tx

Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 5

Three Stages in theEvaluation of a CPR

1. Development of a CPR

2. Prospective validation of a CPR

3. Impact analysis of a CPR

McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 6

So What?

• Q: “Give me the reasons why I need to stay here to listen your presentation?”

• A: a medical informatician may play two roles– Reader role– Developer role

7

Development of a CPR

8

Checklist of Standards for Development of a CPR

1. Definition of outcome

2. Definition of predictor variables

3. Reliability of predictor variables

4. Selection of subjects

5. Sample size

6. Mathematical techniques

7. Sensibility of CPR

8. Accuracy

Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 9

1. Definition of Outcome

• Clearly defined and clinically important– Explicit criteria for diagnosis– Biologic better than behavioral outcome

• Blind assessment of outcome– More important for a “soft” outcome– Less important for a “hard” outcome

10

2. Definition of Predictor Variables

• Clearly defined– Best: collected prospectively, specifically– Less good: collected prospectively as part of

another study– Worst: collected from retrospective review of

records

• Blind assessment of predictor variables

11

3. Reliability of Predictor Variables

• Only reliable variables be included– Intraobserver reliability– Interobserver reliability

• Measurement of reliability– Dichotomous or nominal data: – Ordinal data: weighted – Continuous data: intraclass correlation

coefficient

http://www.dmi.columbia.edu/homepages/chuangj/kappa/12

4. Selection of Subjects

• Patient characteristics stated– Inclusion and exclusion criteria– Method of selection– Clinical and demographic characteristics

• Study site described– Type of institution (primary, secondary, tertiary)– Setting (clinic, ER, hospital ward)– Teaching or non-teaching

13

5. Sample Size

• Overfitting problem– Too few outcome events per predictor variable

• Appropriate sample size– Rule of thumb: at least 10 outcome events per

independent variable– e.g., 3 findings to predict death => at least 30

patients died

14

6. Mathematical Techniques

• Mathematical methods adequately described and justified– Multivariate analysis

• Logistic regression• Discriminant analysis

– Machine learning• Recursive partitioning (including decision tree

learning)• Neural networks

– Survival analysis (survival data only)• Cox's Proportional Hazard Model

15

Multivariate Analysis

• General model

• Logistic regression

– Where P is probability of outcome; G is log odds of outcome

• Discriminant analysis– Compute cutoff (C)– Assign patient to class 1 if G < C; otherwise assign

patient to class 2

nnxbxbxbbG 22110

P

PG

1ln

16

Logistic Regression Vs. Discriminant Analysis

• Logistic regression is much more popular than discriminant analysis (Concato et, al. 1993)– Logistic regression

• Binary outcome• Estimate individual risk and odds ratios

– Discriminant analysis• Categorical outcome• Optimal performance requires many predictor

variables as continuous data• Famous application: diagnosis alcoholism

17

Recursive Partitioning

• Principle– Build an empirical tree diagram by repetitively

splitting patient population into smaller and smaller categories

Yes No

4 5Employed

2

1Yes

3

NoAge>30

18

Recursive Partitioning Vs. Multivariate Analysis

• Recursive partitioning provides a simpler classification rule

• Recursive partitioning may identify nonlinear relationships with outcome event

• Recursive partitioning need greater sample size

• Logistic regression can estimate individual risk and odds ratios

Cook EF, et al. J Chron Dis 1984;37:721-31. 19

Cross SS, et al. Introduction to neural network. Lancet 1995;346:1075-9.

Neural Networks

20

Clinical Applications of Neural Networks

• Diagnosis – AMI, Appendicitis, back pain, dementia, STD

• Imaging– Radiographs, PET, NMR, perfusion scans

• Analysis of wave forms– ECGs, EEGs

• Outcome prediction– Recovery from surgery, cancer, liver transplantation

• Identification of pathological specimens• Genomics

Baxt WG. Application of ANN to clinical medicine. Lancet 1995;346:1135-8.

21

Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.

Advantages of Neural Networks

•Multiple partitioning•Nonlinear partitioning

22

Disadvantages of Neural Networks

• Slow to train

• “Black-boxes”

Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.23

7. Sensibility of CPR

• “Sensibility”– clinically reasonable, easy to use, course of action

described– judgment

• Clinically reasonable– Content validity

• Easy to use– Length of time needed to apply– Simplicity of interpretation

• Course of action described24

8. Accuracy of CPR

• Rationale• Measurement of accuracy

– 2x2 table with sensitivity, specificity, with respective 95% CIs

– Receiver operator characteristic (ROC) curves• Statistical validation

– Cross-validation: Training set vs. test set

Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 25

Classification Performance of a CPR

Predicted Outcome

Actual Outcome

Disease No Disease

Disease 74 244

No Disease 0 247

Sensitivity (95% CI): 1.0 (0.95-1.0)Specificity (95% CI): 0.50 (0.46-0.55)

Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 26

Kuo HS, Chuang JH, Tang GJ, et al. Chin Med J (Taipei) 1999;62:673-681.

27

Example: Development of a CPR

Ottawa Ankle Rules

Stiell IG, et al. Ann Emerg Med 1992;21:384-90

28

Ottawa Ankle Rules

29

The Need for an Ankle Rule

• Blunt ankle trauma– One of the most common injuries in ER– Less than 15% of patients have fractures– Physicians used to order radiography for all

ankle injury patients– 85% negative for fracture– $500 M annually in North America– No widely accepted guideline

Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 30

Study Design• Objective

– Develop CPR with 100% sensitivity• Design

– Prospective survey of ED patients over 5 months• Patient population:

– Setting: Two university hospital EDs in Ottawa– Inclusion: All acute blunt injuries of ankle– Exclusion: < 18 y/o, pregnant, referral, etc

• Data collection– 32 clinical variables collected by 21 trained physicians before

radiography– 100 patients examined by a 2nd physician

31

Study Design (Cont.)

• Measurements of outcomes

– Radiography interpreted by a radiologist blinded to the contents of data collection sheets

• No fracture or insignificant fracture

• Clinically significant fracture

• Data analysis

– Variables found to be both strongly associated with a significant fracture (P < 0.05) and reliable ( > 0.6) were analyzed by logistic regression and recursive partitioning

32

Results

• 70 (10.2%) significant malleolar fractures in 689 ankle injury patients

• Univariate analysis: 17 variables were significantly associated with fractures

• 9 non-reliable variables were further eliminated

• Logistic regression: Sen: 1.0, Spe: .29• Recursive partitioning: Sen: 1.0, Spe: .40

33

689

561128

70#

39#

Yes No31#

67

Yes

12# 494

No19#

21118#

Yes

283

No1#

35 248

0#NoYes

1#441

High Risk 248 Low Risk

A

B

C

D

LEGEND# FractureA Unable to bear weight immediately and in EDB Age 55 or greaterC Bone tenderness B4 or B5D Bone tenderness B8 or B9

Recursive Partitioning of 689 Cases

34

Prospective Validation of a CPR

35

Problems of CPRs With Statistical Validation Only

• Many statistically derived rules fail to perform well when tested in a new population– Overfitting or instability in the original derived

model– Differences in prevalence of disease– Differences in severity of cases– Differences in how the CPR is applied

Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 36

Prospective Validation of a CPR

• Validation– Its repeated application leads to the same

results

• Types of validation– Narrow validation: application of rule in a

similar setting and population– Broad validation: application of rule in multiple

clinical settings with varying prevalence and outcomes of disease

McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 37

Development of a Clinical Prediction Rule

McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 38

Methodological Standards for Validation of a CPR

• Unbiased, wide spectrum patient population

• Blinded assessment of outcomes and predictor variables

• Careful follow-up of predicted normal patients

• Training for correctly applying rules

McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 39

Results of Validation Studies of Ottawa Ankle Rules

Markert RJ, et al. Am J Emerg Med 1998;16:564-7.

Country (Year)

# of Subjects

Sensitivity (%)

Specificity (%)

CA (1993) 1032 100 39

US (1994) 71 100 19

NZ (1994) 350 93 11

US (1994) 631 100 19

US (1995) 422 95 16

40

Impact Analysis of a CPR

41

Reasons for No Impact of an Accurate CPR

• Clinician’s intuition may be as good as the CPR

• Calculations involved may be cumbersome

• Practical barriers to acting on the results of CPR– Medical liability risk– Patient demand factor

McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 42

Methodological Standards for Impact Analysis of a CPR

• Study design– Cluster-based randomized control trial– Before-after study

• Effect on use– e.g., ordering of radiography

• Accuracy of rule

• Acceptability of physicians & patients

Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 43

Impact Analysis of Ottawa Ankle Rules in France

• Randomized 5 EDs to use or not use CPR• 2 in intervention group (906 patients)

– Meeting, pocket cards, posters, and data collection forms

• 3 in usual care group (1005 patients)– data collection forms only

• Results: (unit of analysis was physician)– ordering of radiography: I: 79%; C: 99% (P=.03)– I: 3/112 missed fractures (incomplete data forms: 2,

rule interpretation error by physician: 1)

Auleley GR, et al. JAMA 1997;277:1935-9. 44

Summary

• Development of an effective prediction rule is a long, rigorous, and expensive process

• Properly developed and validated prediction rules can influence clinical practice

45

Performance Evaluation

• Discrimination– Ability of a prediction model to separate those

who experience events from those who do not– Area under a ROC curve (c statistic)

• Calibration– Measures how closely predicted outcomes

agree with actual outcomes– Hosmer-Lemeshow goodness-of-fit test

46

Hosmer-Lemeshow Goodness-of-fit Test

47

48

Thank You!

49