HIV Vaccine Trials

1. Does vaccine prevent HIV infection?

Vaccine Efficacy (VE) = 1 −P(infected|vaccine)

P(infected|placebo)

2. What is vaccine’s impact on post-infection outcomes?Viral load, CD4 count, genetic distance, time from infection diagnosis

to AIDS, time from infection diagnosis to secondary transmission, ....

Vaccines being designed to ameliorate post-infection variables.Merck/HVTN trial 502

Recent HIV Vaccine Trial

RV144: ALVAC + AIDSVAX

16395 subjects randomized to vaccine or placebo.51/8197 infected in vaccine arm.74/8198 infected in placebo arm.

V̂E = 1 − 0.00620.0090 = 0.31.

“This vaccine did not lower the viral loads of people who werevaccinated but caught the virus anyway.” -New York Times (For

First Time, AIDS Vaccine Shows Some Success, Sept 25, 2009)

The Problem

Placebo Arm Vaccine Arm

Infected Infected

The Problem (continued)

Example:

Let Y =log10-transformed viral load.

Immune System

Strong Weak

Vaccine – Y = 5 Y = 5

Placebo Y = 3 Y = 5 Y = 4

Observational Study:

Treatment

Other Variables

Outcome

Our Situation:

Treatment

Infection

Outcome

Other Variables

The Problem

Placebo Arm Vaccine Arm

Infected Infected

Observed Data

Zi =

{0 if subject i = 1, · · · ,N randomized to placebo

1 if randomized to vaccine

Si =

{0 if not infected

1 if infected

Yi = outcome

Potential Outcomes / CounterfactualsSi (z) = infection status indicator if subject i is assigned treatment z .Yi (z) = outcome if assigned treatment z .

Average Causal EffectACE = E [Y (1) − Y (0)|(S(0),S(1)) = (1, 1)]Does the vaccine affect post-infection outcomes for those who would

have been infected regardless of treatment assignment?

Assumptions

1. Stable Unit Treatment Value Assumption (SUTVA)The potential outcomes for each trial participant are

unrelated to the treatment assignment of other subjects.

2. Ignorable Treatment Assignment

(S (0) ,S (1) ,Y (0) ,Y (1)) ⊥⊥ Z

3. Monotonicity: Si(1) ≤ Si (0).Everyone infected in the vaccine arm would have been infected

if randomized to placebo.

Vaccine viral loads:

f aiv (y) = fv (y)

Placebo viral loads:

f aip (y) is not identifiable.

fp(y) = (1 − VE )f aip (y) + VEf prot

p (y)

Bounds (Hudgens, Hoering, Self 2003):

Vaccine viral loads:

Placebo viral loads:

Selection bias model (Gilbert, Bosch, Hudgens 2003):Define w(y) ≡ P(S(1) = 1|S(0) = 1,Y (0) = y).

f aip (y) =

w(y)

1 − VEfp(y)

where

∫w(y)fp(y)dy = 1 − VE .

Perform sensitivity analyses:

Let w(y) =exp(α+ βy)

1 + exp(α+ βy)

where β is unidentifiable sensitivity parameter.

Odds Ratio (OR):Given infection in the placebo arm, for a 1-unit increase in viral load,

the odds of being infected if randomized to the vaccine armmultiplicatively increase by OR = exp(β).

OR=0 OR= 0.001 OR= 0.01 OR= 0.02 OR= 0.05

OR= 0.1 OR= 0.2 OR= 0.333 OR= 0.5 OR= 0.667

OR= 0.8 OR= 0.909 OR= 1 OR= 1.1 OR= 1.25

OR= 1.5 OR= 2 OR= 3 OR= 5 OR= 10

OR= 20 OR= 50 OR= 100 OR= 1000 OR = ∞

Sensitivity Analysis of ACE (mean vaccine viral load minus mean placeboviral load given infected regardless of treatment assignment)

β

AC

E

VaxGen’s trial of AIDSVAX B/B

• Conducted between 1998 and 2003

• 5403 subjects

• 61 sites throughout North America and the Netherlands

• 2:1 vaccine:placebo assignment

• Vaccine was not found to protect against infection(V̂E = 0.048)

• Among non-whites V̂E = 0.47• 30/604 (5.0%) infected in vaccine arm• 29/310 (9.4%) infected in placebo arm

Sensitivity Analysis of ACE (mean vaccine viral load minus mean placebo viralload given infected regardless of treatment assignment) for non-white cohort inVaxGen’s Phase III Trial of AIDSVAX B/B.

0.05 0.10 0.20 0.50 1.00 2.00 5.00 10.00 20.00

−1.5

−1.0

−0.5

0.0

0.5

1.0

Odds Ratio

AC

E

Our Elicitation ExperienceElicited sensitivity parameters from 10 HIV vaccine experts forSTEP trial.

Consider the Merck/HVTN trial. Given two participants assigned placebo whobecome infected during the course of the trial: who do you believe would bemore likely infected if, contrary to fact, they were assigned vaccine?

the person with the higher set-point viral loadthe person with the lower set-point viral load

We would like you to translate your belief into a range of possible odds ratios.Consider two people infected in the placebo arm with set-point viral loads of4.0 and 5.0 log10 copies/ml (approximately corresponding to the 25th and 75thpercentiles of the MACS cohort). Suppose these two people had instead beenassigned vaccine. Then the odds of infection in the vaccine arm for theindividual with the set-point viral load of 5.0 is times the odds ofinfection for the individual with the set-point viral load of 4.0. Please fill in thisblank, giving us both a plausible lower and upper limit for this odds ratio.

lower limit for the odds ratioupper limit for the odds ratio

Non-white cohort of VaxGen Trial

0.05 0.10 0.20 0.50 1.00 2.00 5.00 10.00 20.00

−1.5

−1.0

−0.5

0.0

0.5

1.0

Odds Ratio

AC

E

|

3.00

Eliciting a Counterfactual Sensitivity Parameter

Mentally Challenging

“Friendly revenge”email: “After heavily concentrating [for a coupleof hours] on your abstract questions I am getting back at you, whonavigate in the abstract for a living, with the attached puzzle [aSudoku].”

Biases

What if they’re all wrong?

“At one level direct elicitation of experts’ opinions seems a bad idealikely to perpetuate the errors of the past. On the other handelicitation of experts’ knowledge and analytical processes iscrucial.” -Cox (1998)

Appendages to the Method

Conditioning on Baseline Covariates

Time-to-event outcomes

Relaxing monotonicity

Monotonicity Assumption?

Monotonicity

S(1)0 1

S(0)0 p00 p01 1 − p0

1 p10 p11 p0

1 − p1 p1

p01 = 0 ⇒p11 = p1

log(

p00p11

p10p01

)= ∞

Pr(S(0) = 1|S(1) = 1) = 1.

Placebo Vaccine

Relaxing Monotonicity

S(1)0 1

S(0)0 p00 p01 1 − p0

1 p10 p11 p0

1 − p1 p1

p01 > 0 ⇒p11 < p1

log(

p00p11

p10p01

)<∞

Pr(S(0) = 1|S(1) = 1) < 1.

Placebo Vaccine

Relaxing Monotonicity (Jemiai and Rotnitzky, 2005)

S(1)0 1

S(0)0 p00 p01 1 − p0

1 p10 p11 p0

1 − p1 p1

ψ ≡log“

p00p11p10p01

”

p11

φ ≡Pr(S(0) = 1|S(1) = 1).Specify ψ, β0, and β1; perform analysis; repeat over plausible range.

β0

Placebo

β1

Vaccine

BoundsZhang and Rubin, 2003Jemiai and Rotnitzky, 2005

S(1)0 1

S(0)0 p00 p01 1 − p0

1 p10 p11 p0

1 − p1 p1

If p0 + p1 < 1 then bounds are uninformative.

p11 very small

Placebo Vaccine

Sensitivity Analysis of SCE(t) (probability of starting antiretrovirals by t=2years in placebo minus probability of starting in vaccine given infectedregardless of treatment assignment) for non-white cohort in VaxGen’s Phase IIITrial of AIDSVAX B/B.

OR = exp(β)

Est

ima

ted

SC

E(t

=2

ye

ars

)

−0

.8−

0.4

0.0

0.4

0.8

0.05 0.1 0.2 0.5 1 2 5 10 20

||

Sensitivity analysis of SCE(t=2 years) for VaxGen’s non-white cohort.

φ=Pr(infected in placebo | infected in vaccine)

−0.

6 −0.

5

−0.

4 −

0.3

−0.

2

−0.

1

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.99

.05 .2 .5 2 5 20

.05

.2.5

25

20

−0.

6 −0.

5

−0.

4 −

0.3

−0.

2 −

0.1

0

OR0 = exp(β0)O

R1

=ex

p(β 1

)

φ = 0.95

.05 .2 .5 2 5 20.0

5.2

.52

520

−0.

6 −0.

5

−0.

4 −

0.3

−0.

2 −

0.1

0

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.9

.05 .2 .5 2 5 20

.05

.2.5

25

20

−0.6

−0.

5

−0.

4

−0.

3 −

0.2

−0.

1

0

0.1

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.8

.05 .2 .5 2 5 20

.05

.2.5

25

20

−0.9

−0.8 −0.7

−0.

6

−0.5 −0.4

−0.3

−0.

2

−0.

1

0

0.1

0.2

0.3

0

.4

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.5

.05 .2 .5 2 5 20

.05

.2.5

25

20

−0.9

−0.

8

−0.7

−0.

6

−0.

5

−0.4 −0.3

−0.2

−0.

1

0

0.1 0.2

0.3

0

.4

0.5

0.7

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.094

.05 .2 .5 2 5 20.0

5.2

.52

520

Prostate Cancer Prevention Trial

18,882 randomized to daily finasteride or placebo for 7 years

• 17% on finasteride had detectable cancer

• 23% on placebo had detectable cancer

• 36% of cancers on finasteride were severe

• 22% of cancers on placebo were severe

Z randomized treatment assignment (1=finasteride)S(z) indicator of cancer detected if assigned to treatment z

Y (z) cancer severity if assigned to treatment z

Estimand: E (Y (1) − Y (0)|S(0) = S(1) = 1)

Elicitation Experience

Three parameters to elicit:

1. Consider those in the finasteride arm who got prostate cancer. Whatpercentage of them do you believe would have gotten cancer if, contrary tofact, they were randomized to the placebo arm? What are plausiblelower and upper values for this percentage? lower upper

Optimist: 100% (100–100%)Pessimist: 90% (80–95%)

2. Given two men assigned placebo who got cancer during the course of thetrial: who do you believe would be more likely to have gotten cancer if,contrary to fact, they were assigned finasteride?

the person with the higher Gleason scorethe person with the lower Gleason scorethe two are equally likely

We would like you to translate your belief into a range of possible odds ratios.Consider two people who got cancer in the placebo arm, one with a Gleasonscore of >=7 and the other <7. Suppose these two people had instead beenassigned finasteride. Then the odds of cancer in the finasteride arm for theman with the high Gleason score is times the odds of cancer for theman with the low Gleason score. Please fill in this blank, giving us both aplausible lower and upper limit for this odds ratio.

lower limit for the odds ratioupper limit for the odds ratio

Optimist: 1.2 (1.05–1.35)Pessimist: 3.0 (2.0–4.0)

3. Given two men assigned finasteride who got cancer during the course of thetrial: who do you believe would be more likely to have gotten cancer if,contrary to fact, they were assigned placebo?

the person with the higher Gleason scorethe person with the lower Gleason scorethe two are equally likely

Again, we would like you to translate your belief into a range of possible oddsratios. Consider two people who got cancer in the finasteride arm with Gleasonscores of >=7 and <7. Suppose these two people had instead been assignedplacebo. Then the odds of cancer in the placebo arm for the man with the highGleason score is times the odds of cancer for the man with the lowGleason score. Please fill in this blank, giving us both a plausible lower andupper limit for this odds ratio.

lower limit for the odds ratioupper limit for the odds ratio

Optimist: 1.2 (1.05–1.35)Pessimist: 0.33 (0.25 – 0.50)

Optimist PessimistPr(S(0) = 1|S(1) = 1) 1.0 (1.0–1.0) 0.90 (0.80–0.95)OR0 1.2 (1.05–1.35) 3.0 (2.0-4.0)OR1 1.2 (1.05–1.35) 0.33 (0.25-0.50)

Sensitivity Analysis of ACE (Proportion with severe cancer in finasteride minusproportion with severe cancer in placebo given cancer regardless of treatmentassignment) for Prostate Cancer Prevention Trial.

−5 −4 −3 −2 −1 0 1 2 3 4 5

0.0

0.1

0.2

0.3

0.4

AC

E

0.01 0.02 0.05 0.14 0.37 1 2.7 7.4 20 55 148

β0

OR

Estimated ACE95% C.I.ACE at β0 = + − ∞95% C.I. at β0 = + − ∞

0.1

0.1

5

0.2

0.2

5

φ = 0.99

OR0 = exp(β0)

OR

1=

exp(

β 1)

0.14 0.37 1 2.7 7.4

0.14

0.37

12.

77.

4

0.0

5

0.1

0.1

5

0.2

0.2

5

OR0 = exp(β0)O

R1

=ex

p(β 1

)

φ = 0.95

0.14 0.37 1 2.7 7.4

0.14

0.37

12.

77.

4

0.05

0.1

0.1

5

0.2

0.2

5 0

.3

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.9

0.14 0.37 1 2.7 7.4

0.14

0.37

12.

77.

4

−0.05

0

0.05

0.1

0.1

5

0.2

0.2

5 0

.3

0.3

5

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.8

0.14 0.37 1 2.7 7.4

0.14

0.37

12.

77.

4

−0.

3

−0.

25

−0.2 −0.15 −0.1 −0.05

0

0.05

0.1

0.15

0.2 0.25 0.3

0.35 0.4

0.45

0.5

OR0 = exp(β0)

OR

1=

exp(

β 1)

φ = 0.5

0.14 0.37 1 2.7 7.4

0.14

0.37

12.

77.

4

−0.

5 −

0.4

−0.

35

−0.

3

−0.

25

−0.

2

−0.

15

−0.

1

−0.

05

0

0.05

0.1

0.15 0.2

0.25 0.3 0.35

0.4 0.45

0.5 0.55

0.6

0.65 0.7

OR0 = exp(β0)

OR

1=

exp(

β 1)

S(0),S(1) Independent, φ = 0.24

0.14 0.37 1 2.7 7.40.

140.

371

2.7

7.4

Discussion

1. Showing over wide range2. Is there a need to elicit?3. A priori versus post-hoc4. Interpretation of sensitivity parameters5. Range of selected parameter incompatible with data

References

Hudgens, Hoering, Self (2003), Statistics in Medicine, 22: 2291–2298.Gilbert, Bosch, Hudgens (2003), Biometrics, 59: 531–541.Jemiai (2005), PhD Dissertation w/ Rotnitzky at Harvard Biostatistics.Shepherd, Gilbert, Jemiai, Rotnitzky (2006), Biometrics, 62: 332-342.Jemiai, Rotnitzky, Shepherd, Gilbert (2007), JRSS-B, 69: 879-901.Shepherd, Gilbert, Mehrotra (2007), TAS, 61: 56-63.Shepherd, Gilbert, Lumley (2007), JASA, 102: 573-582.Shepherd, Redman, Ankerst (2008), JASA, 484: 1392-1404.