Nils-Olov Stålhammar

Health Economics II – 2010Health Economic Evaluations Part V

Lecture 3

Evaluations based on patient level data

Nils-Olov Stålhammar

NICE on Estimating Effectiveness of Interventions • The most reliable evidence come from experimental studies with high

internal and external validity that have inclusion and exclusion criteria that have been defined a priori.

• A hierarchy that describes relative validity for estimating relative treatment effect:

1. Randomised Controlled Trials Randomised = patients are randomised to one of two or more treatment arms Controlled = a comparison is made to placebo or other active treatment The RCT is often Double blind = neither the physician nor the patient knows

which treatment arm the patient has been randomised to

2. Controlled observational studies Can be prospective and randomised – the main differences vs. RCT are then that

the study is open (not blinded) and treatment as in clinical practice Can also be done as a retrospective database study

3. Observational studies without control4. Expert opinion

How can clinical trials be used?

• Retrospective analysis based on a single clinical trial• Modelling analysis using data from clinical trials and

other sources• Prospective analysis alongside a clinical trial (trial

based evaluation)– Good data quality (at least high internal validity)– Good use of resources– Good timing

– Probably the type of analysis with the highest credibility

Health economics in drug development

Preclinicalphase

Phase I-II Phase III Phase IV

Studies in clinical practice to defend price

Cost-of-illnessMethodologicalproblems identifiedEarly models Baseline info

expandedModels continuouslyupdated and refinedClinical trials may be used to test questionnaires

Comprehensiveevaluations -alongside clinical trialswhen appropriate

Registrationand launch

First time into man

Conflicting characteristics of a typical clinical trial and a health economic study

Clinical trial Health economic study- Efficacy - Cost and effectiveness

- Double blind - Open?

- Selected patient population - All patients likely to be treated

- Compare to placebo - Compare to treatment likely to be replaced

- Investigations not routinely - Investigations as in clinical done in clinical practice practice

- Conducted in specialist clinics - The setting should be representative for clinical

practice

Conflicting characteristics of a typical clinical trial and a health economic study

Clinical trial Health economic study- Rigid treatment protocols - Allow for variance in

High compliance practice

- No follow-up after discon- - Data is needed for all patients tinuation of treatment during the planned study period

- Intermediate outcomes - Final outcomes are needed

- Short study period - A rather long period must often be considered

Factors influencing the decision to use or not to use a clinical trial for a prospective economic evaluation

• To which extent are the desired criteria fulfilled?• How much can the clinical trial be modified without

jeopardizing other objectives?• What are the alternatives?

– modelling? (may be needed anyway)– later trials?

• How strong is the need for HE data?

Important aspects when designing a health economic evaluation

• Alternatives to be compared• Patient population• Perspective; Costs to be included• Appropriate measure of outcome/effectiveness• How to measure and value costs and effectiveness• Study time period• How to address uncertainty

Additional important aspects when designing a trial based health economic evaluation

• Length of study period should be ‘sufficient’– Extrapolation beyond study period often necessary (If a life

threatening disease is studied then the HE-evaluation will typically cover the remaining life time)

– Ideally, all patients in the RCT should be followed for the same time period

• Sample size and power for the economic evaluation– Economic evaluation often require bigger sample size– Could lead to ethically challenging questions– Usually sample size is given – calculate the power

Calculating the sample size

= The probability of rejecting H0 when it is true = The probability of not rejecting H0 when it is falseThe power of a study = 1-Assuming that two variables have the same std. And are normally distributed, the

required sample size N can be expressed as (1 = mean of variable 1):

Key

H0 is false H0 is true

Our decision H0 is rejected Correct Type I error

H0 is not rejected Type II error Correct

21

2

2/ )(

ZZ

N

H0 = no difference (for instance)


• Record all costs or focus on selection of costs (those supposed to be affected by the treatments studied)?– ‘Noise’ vs. risk of excluding costs that are affected– Challenging to identify truly related costs

• How to collect information about resource use– If directly from patients, consider ‘recalling bias’

• If data is collected from subset of patients/centres, do not allow ‘self-selection’

• Collect information on units of resource use, multiply with unit price/cost in the analysis

• How to extrapolate beyond trial period – if needed

Economic evaluation from the 4S study

• Objective: To investigate whether simvastatin therapy in post-MI and angina patients with serum total cholesterol levels of 5.5 to 8.0 mmol/l is cost-effective

• Costs: Costs of simvastatin and costs of CVD hospitalization• Effectiveness: Number of life years gained• Study period: Median 5.4 years

• During study gain in life years:– Undiscounted 0.065– Discounted 0.054

0 5 10 15 20

% Survival

1

• Assumes no additional benefit after the end of the trial due to simvastatin = all patients face the same risk of dying

• Both treatment groups have the same average life expectancy of 10 additional years (epidemiological data for 65 year old post MI patients);

• Assuming constant rate of death, half of the survivors at end of study dead at 10 years and all patients are dead by 20 years

• Note that other assumptions could be made re. effect after stop of treatment • Rebound effect – higher rate of

death in previously treated pts• Pro-longed effect

End of study mortality benefit

End of clinical study period

Years

Placebo

Simvastatin

Life Years Saved (LYS)

UndiscountedDiscounted

During study gain in life years 0.065 0.054

End of study mortality difference * 0.312 0.186

Total 0.377 0.240

*) Assumes patients alive at end of study have life expectancy of 10 additional years

Cost per Life Year Saved (LYS)

= Net costs / life years saved= (Drug cost - hospital reductions) / life years saved= (21,100 SEK - 7,560 SEK) / 0.240 LYS= 13,540 SEK / 0.240 LYS= 56,400 SEK / life years saved (£5,500 /life year saved)[= 250,000 SEK / LYS during the clinical study]

Note: • Treatment is here supposed to stop at end of clinical trial even though this type of treatment

typically is intended to be life long• Analysing continued treatment requires additional assumptions• Restriction of analysis may also be partly justified by low compliance in the long run and

effect of discounting


• If a multinational trial, consider what can be pooled

Country Country specific Country specific Country specific cost* and mortality cost*, trial mortality prices only

1 $11,450 $5,921 $46,8182 $60,358 $91,906 $57,6363 $244,133 $90,487 $53,8914 $181,259 $93,326 $69,1455 ** ** $65,800All $45,892 $45,892 $45,892

** = dominantR J Willke, Nov 1997

* = prices and resource use

Tirilazad mesylate for treatment of aneurysmal subarachnoid hemorrhage

Cost per death averted (trial involved Europe and Australia)


• How to deal with missing data– To which extent can data be extrapolated from what has been

observed?• How to address uncertainty

– Methodological• Reference case, sensitivity analysis (one- and two-way analyses)

– Sampling variation• Confidence intervals (Stochastic analysis)

– Extrapolation beyond trial and to other settings• Modelling; Sensitivity analysis

• Specify all above in a plan before the study starts

Quantifying uncertainty – sampling variation

• Challenging to calculate Confidence Interval for a ratio (ICER = C/E)– Fieller’s method; assumes that numerator and denominator follow

a joint normal distribution– Non-parametric bootstrapping

ΔQALYs

ΔC

Re-sampling from the original data to build an empirical estimate of the sampling distribution of the ICER

1. Draw samples and calculate mean C and E for treatment group and control group

2. Calculate ICER3. Repeat 1-2 many times4. Calculate CIs

Upper limit of CILower limit of CI

CEAC – Cost-effectiveness acceptability curve

ΔQALYs

ΔC100’/QALY

75’/QALY

Threshold WTP

P that intervention is cost-effective CEAC

75’ 100’

1

Alternative: Net Monetary Benefit (NMB) The variance of NMB can be easily defined

TREC /0)()( BABAT CCEER

NMBICER for A

NMB of A vs. Existing care

Neg. value of increm. cost of A

RT or WTP

ΔE=RT

0 CERT

Upper 95% CI

Lower 95% CI

Net Monetary Benefit (NMB)

TREC /0)()( BABAT CCEER

0 CERT

BBTAAT CERCER

When multiple mutually exclusive alternatives: The alternative with the highest (average) NMB is most cost-effectiveAverage NMB makes sense, average C-E ratio does not

Acceptability curve based on Net Monetary Benefit (NMB)

• Bootstrapping from primary data (RCT) will show that option X is cost-effective in Px% of all ’trials’ for a given WTP

• Changing WTP will change Px

• P for multiple alternatives must sum to 1

P

WTP

X

Y

Z

Sensitivity Analyses



Intermittent treatment of GERD

The clinical studyRandomized, double-blind, 3 different drug therapies, 1 year follow-up, 6 countries

The economic evaluationCost-effectiveness analysis, direct and indirect costs, one analysis for each country, days without medication and symptom free days

1. AIMCompare cost and effect during one year in the three treatment arms.Both direct and indirect costs.Number of days without medication and number of symptom free days.One analysis for each country.

2. METHOD2.1 GENERAL CONSIDERATIONSData on effect and quantities of used resources will be collected within the trial. Quantities will with two exceptions be pooled. The exceptions are sick-leave and mode and time of transport.Country specific prices from other sources.ITT analysis. When observations are missing, values must be imputed.Both stochastic analysis and sensitivity analysis will be done.

Intermittent treatment of GERDMain contents of the economic protocol

Cont.

2.2 THE EFFECT2.2.1 Days without medication2.2.2 Days without symptomsAll days between two consecutive visits if the patient is symptom free at the first of these two visits plus half the number of days between two consecutive visits if the patient is symptom free at the second visit only.2.3 THE COSTS2.3.1 The quantitiesMedication, visits, endoscopies. To avoid protocol driven costs, visits in the non-treatment phase are included only if symptoms suggestive of a relapse have reoccurred at such a visit.Mode of transport and travelling time are collected at two visits for the calculation of the typical cost of transport. These quantities will not be pooled.Number of hours absent from work due to sick-leave and visits. Will not be pooled.

Cont.

2.3.2 The pricesCountry specific prices/costs will be specified in the report. 2.4 IMPUTING VALUESThree groups of patients:I) completing according to protocolII) withdrawn but followedIII) lost to follow up / complete discontinuationFor group III) values will be imputed based on observations in group II) after withdrawal. The monthly imputed cost and effect will be assumed to be the same in all three treatment arms.2.5 STATISTICAL ANALYSISStatistical analysis with CI for incremental c-e ratios.2.6 SENSITIVITY ANALYSIS2.6.1 Uncertainty with regard to costs2.6.2 Uncertainty with regard to incomplete data3. COMMUNICATION OF RESULTS

Main lessons (remaining questions)• Do not underestimate the workload!• There will always be patients LTFU and it is

not obvious how values should be imputed when observations are missing

• Two measures of effectiveness?• Trial-wide or country specific data?

Intermittent treatment of GERD

A suggested checklist for assessing economic evaluations

1. Was a well-defined question posed in answerable form?

2. Was a comprehensive description of the competing alternatives given?

3. Was there evidence that the programs´ effectiveness had been established?

4. Were all the important and relevant costs and consequences for each alternative identified?

5. Were costs and consequences measured accurately in appropriate physical units?

6. Were costs and consequences valued credibly?

7. Were costs and consequences adjusted for differential timing?

8. Was an incremental analysis of costs and consequences of alternatives performed?

9. Was a sensitivity analysis performed?

10. Did the presentation and discussion of study results

include all issues of concern to users?

Drummond et al. 1997

Checklist for conducting a health economic analysis alongside a clinical trial, version 1.2

“Stalhammar’s 13 points”

1. Is the planning period sufficiently long?- will there be time to set up a model to identify cost drivers?- will there be time to develop and test CRF-questions?- will there be time to make the economic study an integrated part of the clinical study?

2. Are relevant alternatives being compared?- are the alternatives relevant for clinical practice?- is current practice among the alternatives?- is ‘no treatment’ a relevant alternative?

Cont.

3. Does patient management reflect clinical practice?- ‘clinical practice treatment protocol’ or treatment at

doctor’s discretion - a question of sample size?

4. Is the study period sufficiently long?- modelling beyond end of study period?- is the study period unnecessarily long (=reduced power)? - is the study period of the same length for all patients?

Cont.

5. What kind of resource consumption should be assessed?- perspective of the analysis?- focus on disease related cost drivers (sometimes difficult to define but a broader focus requires a larger sample size)

6. How should costs be assessed?- billing data (possible in US)- resource consumption multiplied with prices

- how should prices be gathered?- use prices from the year of the study- resource consumption from CRF is preferable,

can indirectly come from patient or other sources (non-study ‘sites’)

Cont.

7. Are there any protocol driven costs?- will some arm(s) be favoured if these costs are disregarded?- will the related (disregarded) procedure affect subsequent costs and outcomes?

8. How should effectiveness be measured?- if one-dimensional, which dimensions of outcomes will be disregarded? are they important?- if disease specific, to which extent will comparisons with other disease areas be possible?- how should data on effectiveness be captured? - if QALYs, patient or population weights?

Cont.

9. How to adjust for baseline differences?- if different employment rates; total cost per employed patient? - if different travelling cost per visit; use the same average travelling cost for all visits irrespective of arm?

10. Cost and effectiveness for patients lost to follow-up?- can costs and effectiveness be based on patient characteristics at time of discontinuation?- pre-specify a basic rule, use others in a sensitivity analysis- minimise discontinuations; can patients be followed ‘outside’ of the clinical study?

Cont.

11. If international study, which data can be pooled across countries?- test for homogeneity?- the assumption that efficacy data can be pooled is usually the rational for the multinational study

12. How to assess uncertainty?- statistical analysis of stochastic data; confidence interval for CE-ratio- sensitivity analysis of deterministic data; new CIs- present as acceptability curve and scatter diagram

13. How much of the analysis should be specified in advance?- as much as possible!- increases credibility

Old exam questionRandomised Controlled Trials (RCT) are generally regarded as the best source for evidence on a drug’s (pharmaceutical’s) efficacy. Well performed RCTs are generally a necessary condition for the successful licensing of a drug, i.e. approval to market and sell the drug. Quite often the RCTs are also used to collect economic data. However, there are several problems that researchers face when conducting an economic evaluation as part of a trial that has been designed primarily for achieving licensing (i.e. to study efficacy and safety). Describe at least five such problems. It should be clear from your answer why each aspect you mention is a problem.

Old exam questiona) Rearrange the cost-effectiveness decision rule (i.e. that the cost for an extra unit of effectiveness should be lower than a certain threshold value) so that it illustrates the decision rule for i) the Net Monetary Benefit (NMB) of the programme, and ii) the Net Health Benefit (NHB)of the programme. b) What is the main advantage of the NMB approach compared to working with incremental cost-effectiveness ratios? c) Draw a figure where the NMB and its upper and lower limits of the confidence intervals (CI) are presented as a function of the threshold value. Illustrate the situation where the lower limit for the CI for the incremental cost-effectiveness ratio is defined, but where the upper limit is not defined.

Documents

Nils-Olov Stålhammar