Mark Pletcher 6/9/2011 Prognostic and Genetic Tests

Mark Pletcher6/9/2011

Prognostic and Genetic Tests

An Example

“Mammaprint”

Gene expression profiling for Breast CA Grind up the tumor, extract RNA Incubate with a microarray of DNA

fragments to estimate expression for each gene

70 previously identified genes predict outcomes

Van de Vijver et al. NEJM 2002;347(25):1999-2009

An Example

“Mammaprint”

Pattern of expression correlates with disease-free and overall survival


An Example

“Mammaprint”

10-year probability of:

Survival Free of mets

“Good” pattern 95% 85%“Bad” pattern 55% 51%


Outline

Prognostic vs. Diagnostic Tests Evaluating a Prognostic Test

Accuracy Utility

Genetic Tests (very briefly)

Prognostic vs. Diagnostic Tests

How is a prognostic test different from a diagnostic test?

Diagnostic Test Prognostic Test

Purpose

Chance Event Occurs to Patient

Study Design

Maximum Obtainable

AUROC

Identify Prevalent Disease

Predict Incident Disease/Outcome

Prior to Test After Test

Cross-Sectional Cohort

<1 (not clairvoyant)1 (gold standard)



Classic prognosis:

Prediction of death after diagnosis of a disease


Prognosis, broadly speaking:

Prediction of any future event Death or recurrence of cancer Stroke after presentation for TIA Peri-operative MI in surgical patients First MI in asymptomatic persons


Prognosis vs. Diagnosis: A Spectrum

Grey areas Pre-clinical disease: Coronary calcium “Reversible” disease: Tiny lung CA Irreversible predisposition: Huntington’s

gene


Prognostication ≠ Etiology

Risk factor Causes the disease Reducing it may prevent disease Confounding is crucial issue in observational studies

Risk marker (i.e., prognostic factor) Predicts the disease Need not be concerned about unmeasured

confounders Not all risk markers are risk factors…(e.g., CRP)

Evaluating Prognostic Tests Test Performance

Association Discrimination Calibration Reclassification Pitfalls

Test Utility

Evaluating Prognostic Tests

Association Is the marker associated with

development of the disease? Odds ratio, relative risk, hazard ratio “Independently associated” means after

adjustment for other known predictors


HRadj = 4.6 P<.001



Discrimination Ability to distinguish between people

with higher or lower risk of disease Metrics: just like diagnostic tests!?

Sensitivity/specificity ROC curves


Mammaprint

Sensitivity = 28/30 = 93%Specificity = 41/83 = 49%

Mets <5yr No mets

Coronary artery calcium Predictor of CHD events Adds discrimination AUROC .63.68

FRS = Framingham Risk ScoreCACS = Coronary Artery Calcium Score

Greenland et al. JAMA 2004;291(2):210-215



Discrimination Results are specific to a particular

time point 5-year risk of metastases or death 90-day risk of stroke


Discrimination

Different results at 5 years….


Discrimination

…than at 10 years


Discrimination Often 1 time point is most relevant or

easily communicated, but information is lost…

Can think of a “set” of discrimination statistics/ROC curves

Harell’s C-Statistic Integrated C-statistic for survival data Similar interpretation as AUROC

Harrell et al. Stat Med 1996;15(4):361-87.


Calibration How close is predicted risk to actual

risk?


Prognostic test results are often converted into absolute risk estimates Like post-test probabilities in

diagnosis Required for clinical interpretation Estimated directly in a longitudinal

study


But absolute risk estimates can be “off” When derivation population different

than target population, etc

Framingham example

D’Agostino et al. JAMA 2001;286(2):180-187


Calibration is “orthogonal” to discrimination Awful discrimination but good

calibration Awful calibration but good

discrimination Miscalibration leads to worse

errors, but it’s easier to fix…


Reclassification How often does the test lead to

reclassification across a treatment threshold?

i.e., how often might the test lead to a change in treatment?

CRP reclassification example


Reclassification How often does the test lead to

reclassification across a treatment threshold?

Cook et al. Annals of Int Med 2006;145(1):21-29


Reclassification metrics Net Reclassification Improvement

(NRI) Net % reclassified correctly

Depends on specified treatment thresholds/categories


Pitfalls for prognostic test studies

Loss to follow-up and competing risks Especially problematic if loss is

“differential”



Bias if clinician knows the test result e.g. – persons with coronary calcium+

are: More likely to get revascularization More likely to get referred to ED if they have

chest pain



Overfitting Test will perform best in sample from

which it is derived More variables and “choices” more

danger of overfitting Gene expression arrays, proteomics


Clinical Utility Does it improve health?


Test Result

Better patient understanding of disease/risk

Healthier patient behaviors

Better clinical decisions

1

2

3

Better health

Pletcher et al. Circulation 2011;123;1116-1124


Clinical Utility Cannot be estimated from test

performance metrics alone Need to understand downstream

consequences, including Benefits and harms of interventions based on

test result Harms from test itself Quality and length of life Costs


Clinical Utility Can be estimated directly…

Randomized trial of test-and-treat strategy …or indirectly

Decision analysis/cost-effectiveness modeling

Same issues for diagnostic tests, and especially important when screening apparently healthy people…

Pletcher et al. Circulation 2011;123;1116-1124

Genetic Tests

Potentially useful for mechanistic insight

Prognostic implications across individuals in a family

Otherwise, must meet same standards for prognostic utility as other tests Single gene studies often disappointing

Key concepts For prognostic tests, an element of time

and chance remain (perfect test impossible)

Discrimination vs. Calibration Reclassification indices help us

understand how often a test might change management

Clinical utility depends on accounting for net benefits and harms (and costs)

Documents

Mark Pletcher 6/9/2011 Prognostic and Genetic Tests