Estimating Outcomes Estimating Outcomes in Decision Analysisin Decision AnalysisMiriam Kuppermann, PhD, MPHMiriam Kuppermann, PhD, MPH

Associate ProfessorAssociate Professor

Departments of Obstetrics, Departments of Obstetrics, Gynecology & Reproductive Gynecology & Reproductive Sciences and Epidemiology & Sciences and Epidemiology & BiostatisticsBiostatistics

University of California, San FranciscoUniversity of California, San Francisco

Review—Last LectureReview—Last Lecture

• Formulate an explicit questionFormulate an explicit question• Make a decision treeMake a decision tree• Estimate probabilitiesEstimate probabilities• Estimate utilitiesEstimate utilities• Compute expected utility of each branchCompute expected utility of each branch• Perform sensitivity analysisPerform sensitivity analysis

Preview—Where We Are Preview—Where We Are GoingGoing

Recall Ms. Brooks and her aneurysm.Recall Ms. Brooks and her aneurysm.

We want to: We want to: • Determine her utility: clip, and not clipDetermine her utility: clip, and not clip• Compare incremental utility and costCompare incremental utility and cost• Compare cost-per-unit of utility across Compare cost-per-unit of utility across

private and public uses of funds. private and public uses of funds.

Overview – Today’s Overview – Today’s LectureLecture

• OutcomesOutcomes• Measuring UtilitiesMeasuring Utilities• Quality Adjusted Life Years (QALYs)Quality Adjusted Life Years (QALYs)• Discounting Discounting • Prenatal testing examplePrenatal testing example

Quantifying Health OutcomesQuantifying Health Outcomes• Mortality Mortality • Life yearsLife years

Number of expected years of life Number of expected years of life • Significant morbiditySignificant morbidity

Paralysis, loss of sightParalysis, loss of sight• Quality-Adjusted Life Years (QALYs)Quality-Adjusted Life Years (QALYs)

Expected life years adjusted for the Expected life years adjusted for the valuation of the possible states in each yearvaluation of the possible states in each year

• Financial valuation of outcomesFinancial valuation of outcomesCosts to patient, payor, or societyCosts to patient, payor, or societyWillingness to pay to avoid outcomes, Willingness to pay to avoid outcomes,

obtain treatment obtain treatment

Health Outcomes – MortalityHealth Outcomes – Mortality• MortalityMortality

Death from disease/accident/procedureDeath from disease/accident/proceduree.g. If Ms. Brooks undergoes surgery, one of e.g. If Ms. Brooks undergoes surgery, one of the possible outcomes is mortalitythe possible outcomes is mortality

• Life YearsLife Years Calculate an expected value of life years Calculate an expected value of life years using a probabilistically weighted average of using a probabilistically weighted average of expected life expected life e.g.e.g. If Ms. Brooks does not undergo surgery, If Ms. Brooks does not undergo surgery, her life expectancy is less than if she did not her life expectancy is less than if she did not have aneurysm, these outcomes are have aneurysm, these outcomes are measured in expected life yearsmeasured in expected life years

Health Outcomes – MorbidityHealth Outcomes – Morbidity• MorbidityMorbidity

Some health state that is less than Some health state that is less than perfectperfecte.g. disability from stroke, chronic paine.g. disability from stroke, chronic pain

• Comparison of morbidities Comparison of morbidities Difficult – apples and oranges problem Difficult – apples and oranges problem e.g. which is worse:e.g. which is worse:Blind vs. deafBlind vs. deafDeaf vs. paraplegiaDeaf vs. paraplegiaParaplegia vs. blindParaplegia vs. blind

Outcomes - UtilityOutcomes - UtilityUtility is the currency of outcomesUtility is the currency of outcomes• Scaled from 0 to 1Scaled from 0 to 1• Commonly Death = 0Commonly Death = 0• Perfect Health = 1Perfect Health = 1

Sounds good, but how can this be Sounds good, but how can this be measured?measured?

Utility MeasurementUtility Measurement

Utilities are commonly estimated using Utilities are commonly estimated using comparisons to the 0 and 1 anchorscomparisons to the 0 and 1 anchors• Methods are:Methods are:• Visual Analog ScaleVisual Analog Scale• Standard GambleStandard Gamble• Time Trade-offTime Trade-off

Utility MeasurementUtility MeasurementClinical Scenario:Clinical Scenario:• Patient in the hospital has infection of the leg Patient in the hospital has infection of the leg • Two options: BKA v. Medical ManagementTwo options: BKA v. Medical Management• BKA – below the knee amputation (1% mortality BKA – below the knee amputation (1% mortality risk)risk)• Medical – 20% chance of infection worseningMedical – 20% chance of infection worsening

If worse, AKA – above the knee amputation If worse, AKA – above the knee amputation (10% mortality risk) (10% mortality risk) • How can we value these outcomes?How can we value these outcomes?

Utility MeasurementUtility Measurement

Visual Analog ScaleVisual Analog Scale

Linear Scale from 0 to 1Linear Scale from 0 to 1

(death 0-----------------------------1.0 cure)(death 0-----------------------------1.0 cure)Where is AKA?Where is AKA?

(death 0-----------------------------1.0 cure)(death 0-----------------------------1.0 cure)Where is BKA?Where is BKA?

Utility MeasurementUtility MeasurementVisual Analog ScaleVisual Analog Scale

Advantages: Advantages: quantitative, easy to quantitative, easy to understand, visual understand, visual

Disadvantages: Disadvantages: may bias values to the may bias values to the middle, seems disconnected from medical middle, seems disconnected from medical realityreality

Utility Measurement--Utility Measurement--Standard GambleStandard Gamble

Method relies on patients choosing between:Method relies on patients choosing between: 1) a certain outcome1) a certain outcome 2) a gamble between a better outcome and a 2) a gamble between a better outcome and a worse outcomeworse outcome

How it works:How it works: Choice 1Choice 1: You live with a : You live with a BKABKA Choice 2Choice 2: The gamble – you might have a : The gamble – you might have a curecure; ;

you might you might diedie..

Goal of method is to find the break-even point.Goal of method is to find the break-even point. What probability of death would you accept to avoid What probability of death would you accept to avoid

living with the BKA? living with the BKA?

Utility Measurement – Utility Measurement – Standard GambleStandard Gamble

Choose BKA?

Yes

No

BKA (intermediate outcome)

Perfect health

Death

Live?

p %

(100-p) %

Utility Measurement – Utility Measurement – Standard GambleStandard Gamble

Standard gamble measurement involves questioning patients Standard gamble measurement involves questioning patients to determine the p at which the two outcomes are equivalent to determine the p at which the two outcomes are equivalent

Using expected utilities, the value of p [p = P(cure)] gives the Using expected utilities, the value of p [p = P(cure)] gives the utility:utility: Utility (BKA) x Prob (BKA) = Utility (BKA) x Prob (BKA) = Utility(cure) x (p) + Utility(death) x (1-p)Utility(cure) x (p) + Utility(death) x (1-p) [Note that here, P(BKA) = 1.][Note that here, P(BKA) = 1.]

You can demonstrate that the utility of BKA = p:You can demonstrate that the utility of BKA = p:Utility (BKA) = [Utility(cure) x (p) + Utility(death) x (1-p)] Utility (BKA) = [Utility(cure) x (p) + Utility(death) x (1-p)] = [1.0 * p + 0 * (1-p)] = p= [1.0 * p + 0 * (1-p)] = p

Utility Measurement – Utility Measurement – Standard GambleStandard Gamble

In utility estimation utilizing the standard gamble In utility estimation utilizing the standard gamble measurement, using the simple comparison described, the measurement, using the simple comparison described, the value of p, the probability of perfect health needed to make value of p, the probability of perfect health needed to make the patient indifferent between the two choices is the the patient indifferent between the two choices is the estimated utility of the outcome.estimated utility of the outcome.

Advantages: Advantages: the incorporation of risk into the model, the incorporation of risk into the model, comparison or choice between different outcomes.comparison or choice between different outcomes.

Disadvantages: Disadvantages: possible nonrealistic choices patients may possible nonrealistic choices patients may be asked to make, the difficulty of understanding the be asked to make, the difficulty of understanding the question (especially for non-gamblers), and the difficulty for question (especially for non-gamblers), and the difficulty for some in understanding a math equation.some in understanding a math equation.

Utility Measurement – Utility Measurement – Time Trade-offTime Trade-off

Time Trade-off involves patients choosing between: Time Trade-off involves patients choosing between: quality of life vs. length of time alivequality of life vs. length of time alive

We want to determine when patients are equivocal between We want to determine when patients are equivocal between choice:choice: Time A * Utility A = Time B * Utility BTime A * Utility A = Time B * Utility B e.g. -- If you have a life expectancy of 30 years with a BKA, e.g. -- If you have a life expectancy of 30 years with a BKA, how much time would you give up to live in your current state?how much time would you give up to live in your current state?

Would you give up 5 years? 3 years? 1 year?Would you give up 5 years? 3 years? 1 year? 30 years * Utility (BKA) = (30-x) years * 1.0 If you’re willing 30 years * Utility (BKA) = (30-x) years * 1.0 If you’re willing to give up 3 years, that means the utility of BKA is: to give up 3 years, that means the utility of BKA is: = [(30-3)*1/ 30] = 27/30 = 0.9= [(30-3)*1/ 30] = 27/30 = 0.9

Utility Measurement – Utility Measurement – Time Trade-offTime Trade-off

Time trade off can be used to measure the utility of Time trade off can be used to measure the utility of an outcome in the fashion described. Like an outcome in the fashion described. Like standard gamble, patients need to understand standard gamble, patients need to understand subtlety of questions being asked.subtlety of questions being asked.

Advantages: Advantages: the simplicity of the choice between the simplicity of the choice between different outcomes, consideration of long-term different outcomes, consideration of long-term outcomes.outcomes. Disadvantages: Disadvantages: fails to incorporate risk, lack of fails to incorporate risk, lack of clarity of when time traded occurs, different clarity of when time traded occurs, different valuation of time during life, and the theoretical valuation of time during life, and the theoretical lack of realism of the choice.lack of realism of the choice.

Utility Measurement – Utility Measurement – Health State Excluding Health State Excluding

Clinical ProblemClinical Problem 0 to 1 utility scaling is simple and useful for determining the 0 to 1 utility scaling is simple and useful for determining the utility of different outcomes for a single patient.utility of different outcomes for a single patient.

However, comparisons across patients or across programs However, comparisons across patients or across programs raises the issue of ‘Is a utility of 1 really available?’raises the issue of ‘Is a utility of 1 really available?’

Consider: (a) polio vaccinations for children, and (b) hip Consider: (a) polio vaccinations for children, and (b) hip replacement surgery for 85-year old patients.replacement surgery for 85-year old patients.

Typically (on average) the health state of a polio-free 5-year old Typically (on average) the health state of a polio-free 5-year old will be very close to 1, whereas the health state of an 85-year old will be very close to 1, whereas the health state of an 85-year old even with a perfectly functioning hip will typically be less than even with a perfectly functioning hip will typically be less than one.one.

Utility Measurement – Utility Measurement – ChainingChaining

• The utility of the health state of the 85-year old can The utility of the health state of the 85-year old can be determined by “chaining”.be determined by “chaining”.

• Consider first the clinical situation under review: what Consider first the clinical situation under review: what is the utility, on a 0 to 1 scale, of a hip with reduced is the utility, on a 0 to 1 scale, of a hip with reduced function and some pain? Let us say utility = .8.function and some pain? Let us say utility = .8.

• Consider next the utility of an 85-year old with no hip Consider next the utility of an 85-year old with no hip problems, but some other reduced function or chronic problems, but some other reduced function or chronic pain. Here, perhaps overall utility = .9.pain. Here, perhaps overall utility = .9.

• Total utility for the bad-hip case = .8 * .9 = .72Total utility for the bad-hip case = .8 * .9 = .72 Here, the perfect-hip case utility = 1 * .9 = .9Here, the perfect-hip case utility = 1 * .9 = .9

Utility Measurement – Utility Measurement – Additional InformationAdditional Information

• Multi-Attribute Health Status Classification Multi-Attribute Health Status Classification SystemSystem

• Developed by Health Utilities, Inc.Developed by Health Utilities, Inc.

•Available at: Available at: http://www.healthutilities.com/overview.htmhttp://www.healthutilities.com/overview.htm

Utilities in Decision Utilities in Decision AnalysisAnalysis

• Now that we have methods to estimate Now that we have methods to estimate utilities, these can be used in the DAutilities, these can be used in the DA• However, our outcomes often include both However, our outcomes often include both mortality and morbiditymortality and morbidity• Want a way to add in life expectancyWant a way to add in life expectancy• Quality Adjusted Life-Years (QALYs)Quality Adjusted Life-Years (QALYs)

QALYsQALYs• QALYs are generally considered the QALYs are generally considered the standard unit of comparison for outcomes standard unit of comparison for outcomes

• QALYs = time (years) x quality (utils)QALYs = time (years) x quality (utils)

• e.g. 30 years after BKA, util(BKA) = 0.9e.g. 30 years after BKA, util(BKA) = 0.9

= 30 x 0.9 = 27 QALYs= 30 x 0.9 = 27 QALYs

QALYsNo aneurysm rupture0.9825

No surgery34.86 Die

Aneurysm rupture 0.450.0175 Survive

0.55

No aneurysm ruptureDifference 1

_ QALYs -2.85 Survive surgery0.902 Die

Aneurysm rupture 0.45Clipping 0 Survive

32.01 0.55Key Inputs Surgery-induced disabilityRupture risk/yr 0.0005 0.075Expected life span 35RR rupture w/ surgery 0 Surgical deathSurgical mortality 0.023 0.023Surg morb (disability) 0.075

0.0

Ms. Brooks

17.5

35.0Normal survival

Disability, shorter survival

5.8

Immediate death

Normal survival 35.0

Normal survival

Normal survival

Early death

Early death

35.0

17.5

35.0

QALYsQALYs

QALYsQALYsAneurysm ExampleAneurysm Example

• We said life expectancy is reduced by 2/3, We said life expectancy is reduced by 2/3, so instead of 35, it is = 35 * .333 = 11.67so instead of 35, it is = 35 * .333 = 11.67

• Here, we have assigned a utility of .5 to Here, we have assigned a utility of .5 to surgery-induced disability, so QALYs = surgery-induced disability, so QALYs =

years * utils = 11.67 * .5 = 5.8years * utils = 11.67 * .5 = 5.8

Outcomes - DiscountingOutcomes - Discounting• However, are all years considered However, are all years considered equal?equal?

• Consider:Consider: Favorite MealFavorite Meal

Extreme PainExtreme Pain

Lifetime IncomeLifetime Income

Outcomes - DiscountingOutcomes - Discounting• Generally, present > futureGenerally, present > future• One common way to value the different times One common way to value the different times is discounting is discounting • Essentially this year is worth Essentially this year is worth δδ more than more than next yearnext year• δδ is commonly set at 0.03 or 3% is commonly set at 0.03 or 3%• In order to compare values of all future times, In order to compare values of all future times, a calculation, net present value, is often useda calculation, net present value, is often used• NPV = 1 / (1 + NPV = 1 / (1 + δδ))t t Where t is number of years Where t is number of years in the futurein the future

Outcomes - DiscountingOutcomes - Discounting• Aneurysm ExampleAneurysm Example• If utility is 0.5 and life expectancy is 3 If utility is 0.5 and life expectancy is 3 yearsyears• NPV would be: NPV would be: Utility / (1 + Utility / (1 + δδ))t t

• However, when is year 1? Often, since However, when is year 1? Often, since events in year one occur on average half events in year one occur on average half way through, we use 0.5 for year 1way through, we use 0.5 for year 1• NPV = 0.5 / (1.03)NPV = 0.5 / (1.03)0.50.5 + 0.5 / (1.03) + 0.5 / (1.03)1.51.5 + 0.5 / + 0.5 / (1.03)(1.03)2.52.5

• NPV = 0.5 * {(1.03)NPV = 0.5 * {(1.03)-0.5 -0.5 + (1.03) + (1.03) -1.5-1.5 + (1.03) + (1.03) --

2.52.5}}

QALYsdisc

No aneurysm rupture0.9825

No surgery21.37 Die

Aneurysm rupture 0.45

0.0175 Survive0.55

No aneurysm ruptureDifference 1

Δ QALYs -1.63 Survive surgery0.902 Die

Aneurysm rupture 0.45

Clipping 0 Survive19.74 0.55

Key Inputs Surgery-induced disabilityRupture risk/yr 0.0005 0.075

Expected life span 35RR rupture w/ surgery 0 Surgical deathSurgical mortality 0.023 0.023

Surg morb (disability) 0.075

0.0

Ms. Brooks

13.3

21.4Normal survival,

worry

Disability, shorter survival

4.8

Immediate death

Normal survival,worry

21.4

Normal survival

Normal survival

Early death,worry

Early death

21.5

13.4

21.5

Outcomes - DiscountingOutcomes - Discounting

Outcomes - DiscountingOutcomes - Discounting• Another issue is partial yearsAnother issue is partial years• Can use similar adjustment:Can use similar adjustment:• Exponent is half way through time periodExponent is half way through time period• Numerator is multiplied by fraction of yearNumerator is multiplied by fraction of year• e.g. disability for 1.6 years, utility of 0.5e.g. disability for 1.6 years, utility of 0.5• NPV = NPV = Utility / (1 + Utility / (1 + δδ))t t

• NPV = 0.5 / (1.03)NPV = 0.5 / (1.03)0.50.5 + 0.5*0.6 / (1.03) + 0.5*0.6 / (1.03)1.31.3

Discounting – Special TopicDiscounting – Special Topic

• Issues with exponential discounting:Issues with exponential discounting:• Relatively easy to manipulateRelatively easy to manipulate• However, assumes same difference However, assumes same difference between any two time periods is the same between any two time periods is the same value difference value difference • ConsiderConsider

today today : : vs. tomorrow and vs. tomorrow and 10 yrs vs. 10 yrs + 1 day 10 yrs vs. 10 yrs + 1 day

• Essentially, “today” versus all other time Essentially, “today” versus all other time periods is valued higher for many outcomesperiods is valued higher for many outcomes

Discounting – Special TopicDiscounting – Special Topic• Importance of discount rate chosen:Importance of discount rate chosen:• Imagine a benefit of $1,000,000 to be Imagine a benefit of $1,000,000 to be received in 30 yearsreceived in 30 years• Present value of this benefit at different Present value of this benefit at different discount rates:discount rates:

1% -- $741,9231% -- $741,923

3% -- $411,9873% -- $411,987

8% -- $99,3778% -- $99,377

17% -- $9,00417% -- $9,004

Overall ReviewOverall Review• OutcomesOutcomes

MortalityMortalityMorbidityMorbidity

• Measuring UtilitiesMeasuring UtilitiesVisual AnalogVisual AnalogStandard GambleStandard GambleTime Trade-offTime Trade-off

• Quality Adjusted Life Years (QALYs)Quality Adjusted Life Years (QALYs) QALYs = time (years) x quality (utils)QALYs = time (years) x quality (utils)• Discounting Discounting

NPV = NPV = Utility / (1 + Utility / (1 + δδ))t t

Informed Decision-Making in Prenatal Testing

Miriam Kuppermann, PhD, MPHAssociate Professor

Decision and Cost-Effectiveness Analysis EPI 213

January 20, 2005Department of Obstetrics, Gynecology, & Reproductive Sciences

Prenatal Tests for Chromosomal Disorders

Screening Tests (non-invasive)Expanded AFP Nuchal translucency

Diagnostic Tests (invasive)Amniocentesis Chorionic villus sampling (CVS)

Guidelines For Prenatal Testing Are Dichotomized By Maternal Age

Women < 35 Screening first Invasive testing only

if “positive” results

Women > 35 Invasive testing

offered Screening as an

option

Rationale for GuidelinesNeed to limit access to invasive testing

Inherent risk of procedure Limited availability of providers,

laboratoriesAge 35 selected as the threshold

Increasing risk with advancing maternal age

Threshold set where risks equal Cost/benefit considerations

Kuppermann, Nease, Goldberg, Washington. Who should be offered prenatal diagnosis? The 35-year-old question. Am J Public Health 1999; 89:160-3

Equal Risk Threshold

Risk of Miscarriage = Risk of Down Syndrome

Implicit assumption: equally burdensome outcomes

Procedure-related miscarriage

Down-syndromeaffected infant

First Challenge to Guidelines

Do women find procedure-related miscarriage and Down-syndrome-affected birth to be equally burdensome?

How Do Women Value Testing Outcomes?

1082 socioeconomically and age-diverse women

English-, Spanish- or Chinese-speaking

Interviewed <20 weeks pregnant Preferences for testing outcomes Demographic/attitudinal questions Testing behavior

Simplified Decision Tree for Prenatal Testing

Preference Score

“Utility” Ranges from 0 to 1 1 = ideal outcome, 0=death Measured using “time tradeoff””

Time Tradeoff Preference Elicitation

Choice A Choice B

Which do you prefer?

40 years of life remaining with affected child

40 years of life remaining with unaffected child (give up 0 years of life)

Time Tradeoff Preference Elicitation

40 years of life remaining with affected child

30 years of life remaining with unaffected child

(give up 10 years of life)

Which do you prefer?

Choice A Choice B

Calculation of Time Tradeoff Scores

reduced life expectancy with unaffected child (30 years)UTTO = __________________________________________

full life expectancy with affected child (40 years)

= 0.75

 

Median utility for procedure-related

miscarriage= 0.86

Median utility for Down-syndromeaffected infant

= 0.73

On average, women do not equally weight the outcomes of procedure-related

miscarriage and Down syndrome-affected birth

P<0.001 by Wilcox sign rank test

Kuppermann, Nease, Learman, Gates, Blumberg, Washington. Procedure-related miscarriages and Down syndrome-affected births: implications for prenatal testing based on women’s preferences. Obstet Gynecol 2000; 96:511-6.

Preference Difference Score

One way to look at the relative value women assign to procedure-related miscarriage and DS-affected birth

Pref score misc – Pref score DS

Higher score = greater preference for miscarriage over DS

0

25

50

75

100

125

150

175

200

Num

ber

-1 -.75-.5 -.25 0 .25 .5 .75 1

Preferences Vary Substantially

Utility misc - Utility DS

First Conclusion

Current guidelines do not adequately reflect the preferences of pregnant women.

Second Challenge to GuidelineSecond Challenge to Guideline

Old paradigm: COST BENEFITBenefits (in $$ terms) of program should exceed costs. Costs of offering testing should be offset by savings accrued by averting the birth of Down-syndrome-affected infants

New paradigm: COST EFFECTIVENESSNo $$ value assigned to outcomes. Cost of offering testing should be “worth” the gain in quantity and quality of life.

How is Cost Effectiveness Measured?

Δ CC

Δ EE

Δ Costs

Δ QALY==

QALY = Quality–Adjusted Life Year

Cost Effectiveness of Prenatal Diagnosis QALYs Lifetime cost Cost-utility ratio

Age 20

Amniocentesis 24·16 $54,080 $14,200

No testing 24·08 $52,940

Age 35

Amniocentesis 20·39 $61,490 $12,600

No testing 20·30 $60,360

Age 44

Amniocentesis 17·08 $59,020 $11,300

No testing 16·98 $57,890

Harris, Washington, Nease, Kuppermann. Cost utility of prenatal diagnosis and the risk-based threshold. Lancet 2004; 363:276-82.

Second Conclusion

Offering invasive testing to women of all ages and risk levels can be cost effective.

Implications

Guidelines should be changed to enable all women to make informed choices about which prenatal tests, if any, to undergo.

Tools that help women and partners make informed choices should be implemented.