1

Sensitivity Analysis for Residual Confounding

Sebastian Schneeweiss MD, ScD

Division of Pharmacoepidemiology and Pharmacoeconomics

Department of Medicine, Harvard Medical School,

2

Outline

1. Residual Confounding and what we can do about it

2. Simple sensitivity analysis: Array Approach3. Study-specific analysis: Rule Out

Approach4. Using additional information: External

Adjustment

3

Unmeasured (residual) Confounding:

[smoking,healthy lifestyle, etc.]

Drug exposure

Outcome

RREO

OREC RRCO

CU

CM

4

Unmeasured Confounding in Claims Data

Database studies are criticized for their inability to measure clinical and life-style parameters that are potential confounders in many pharmacoepi studies OTC drug use BMI Clinical parameters: Lab values, blood pressure, X-

ray Physical functioning, ADL (activities of daily living) Cognitive status

5

Strategies to Minimize Residual Confounding

Choice of comparison group Alternative drug use that have the same perceived

effectiveness and safety Multiple comparison groups

Crossover designs (CCO, CTCO) Instrumental Variable estimationHigh dimensional proxy adjustment

6

Strategies to Discuss Residual Confounding

Qualitative discussions of potential biasesSensitivity analysis

SA is often seen as the ‘last line of defense’ A) SA to explore the strength of an association as a

function of the strength of the unmeasured confounder B) Answers the question “How strong must a

confounder be to fully explain the observed association”

Several examples in Occupational Epi but also for claims data

Greenland S et al: Int Arch Occup Env Health 1994

Wang PS et al: J Am Geriatr Soc 2001

7

Dealing with confounding

Schneeweiss, PDS 2006

Confounding

Unmeasured Confounders

Measured Confounders

Design

•Restriction

•Matching

Analysis

•Standardization

•Stratification

•Regression

Unmeasured, but measurable in

substudy

•2-stage sampl.

•Ext. adjustment

•Imputation

Unmeasurable

Design Analysis

•Cross-over

•Active comparator (restriction)

•Instrumental variable

•Proxy analysis

•Sensitivity analysis

Propensity scores

•Marginal Structural Models

8

A simple sensitivity analysis

The apparent RR is a function of the adjusted RR times ‘the imbalance of the unobserved confounder’

After solving for RR we can plug in values for the prevalence and strength of the confounder:

1)1(

1)1(

0

1

CDC

CDC

RRP

RRPRRARR

1)1(

1)1(

0

1

CDC

CDC

RRP

RRP

ARRRR

9

A made-up example

Association between TNF-a blocking agents and NH lymphoma in RA patients Let’s assume an observed RR of 2.0 Let’s assume 50% of RA patients have a more

progressive immunologic disease … and that more progressive disease is more likely

to lead to NH lymphoma Let’s now vary the imbalance of the hypothetical

unobserved confounder

10

Bias by residual confounding

4.5

2.5

0.8

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

RRadjusted

RRCD

PC1

Fixed:ARR = 2.0

PC0 = 0.5

11

2. Array approach

fix X Y fix Z2 Z1

ARR RRCD PC1 PC0 RRadjusted % Bias % Bias = [(ARR-RRadj.)/(RRadj.-1)]*100

2.0 4.5 0.0 0.5 5.5 -77.782.0 4.0 0.0 0.5 5.0 -75.002.0 3.5 0.0 0.5 4.5 -71.432.0 3.0 0.0 0.5 4.0 -66.672.0 2.5 0.0 0.5 3.5 -60.002.0 2.0 0.0 0.5 3.0 -50.002.0 1.5 0.0 0.5 2.5 -33.332.0 1.0 0.0 0.5 2.0 0.002.0 0.8 0.0 0.5 1.8 33.332.0 0.5 0.0 0.5 1.5 100.002.0 4.5 0.1 0.5 4.1 -67.472.0 4.0 0.1 0.5 3.8 -64.862.0 3.5 0.1 0.5 3.6 -61.542.0 3.0 0.1 0.5 3.3 -57.142.0 2.5 0.1 0.5 3.0 -51.062.0 2.0 0.1 0.5 2.7 -42.112.0 1.5 0.1 0.5 2.4 -27.592.0 1.0 0.1 0.5 2.0 0.002.0 0.8 0.1 0.5 1.8 25.812.0 0.5 0.1 0.5 1.6 72.732.0 4.5 0.2 0.5 3.2 -55.262.0 4.0 0.2 0.5 3.1 -52.942.0 3.5 0.2 0.5 3.0 -50.002.0 3.0 0.2 0.5 2.9 -46.152.0 2.5 0.2 0.5 2.7 -40.912.0 2.0 0.2 0.5 2.5 -33.332.0 1.5 0.2 0.5 2.3 -21.432.0 1.0 0.2 0.5 2.0 0.002.0 0.8 0.2 0.5 1.8 18.752.0 0.5 0.2 0.5 1.7 50.002.0 4.5 0.3 0.5 2.7 -40.582.0 4.0 0.3 0.5 2.6 -38.712.0 3.5 0.3 0.5 2.6 -36.362.0 3.0 0.3 0.5 2.5 -33.332.0 2.5 0.3 0.5 2.4 -29.272.0 2.0 0.3 0.5 2.3 -23.532.0 1.5 0.3 0.5 2.2 -14.812.0 1.0 0.3 0.5 2.0 0.002.0 0.8 0.3 0.5 1.9 12.122.0 0.5 0.3 0.5 1.8 30.772.0 4.5 0.4 0.5 2.3 -22.582.0 4.0 0.4 0.5 2.3 -21.432.0 3.5 0.4 0.5 2.3 -20.002.0 3.0 0.4 0.5 2.2 -18.182.0 2.5 0.4 0.5 2.2 -15.792.0 2.0 0.4 0.5 2.1 -12.502.0 1.5 0.4 0.5 2.1 -7.692.0 1.0 0.4 0.5 2.0 0.002.0 0.8 0.4 0.5 1.9 5.882.0 0.5 0.4 0.5 1.9 14.292.0 4.5 0.5 0.5 2.0 0.002.0 4.0 0.5 0.5 2.0 0.002.0 3.5 0.5 0.5 2.0 0.002.0 3.0 0.5 0.5 2.0 0.00

4.5

3.5

2.5

1.5

0.8

0.0 0.

2 0.4 0.

6 0.8 1.

0-100

-50

0

50

100

150

200

250

300

350

% Bias

RRCD

PC1

Fixed:ARR = 2.0

PC0 = 0.5

4.5

2.5

0.8

0.0 0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

RRadjusted

RRCD

PC1

Fixed:ARR = 2.0

PC0 = 0.5

1)1(

1)1(

0

1

.

CDC

CDC

adj

RRP

RRP

ARRRR

drugepi.o

rg

12

Pros and cons of “Array approach”

Very easy to perform using ExcelVery informative to explore confounding with

little prior knowledge Problems: It usually does not really provide an answer

to a specific research question4 parameters can vary -> in a 3-D plot 2

parameter have to be kept constantThe optical impression can be manipulated

by choosing different ranges for the axes

13

Same example, different parameter ranges

3.0

1.7

0.8

0.2

0.3

0.3

0.4

0.4

0.5

0.5

0.6

0.6

0.7

0.7

0.0

0.5

1.0

1.5

2.0

2.5

3.0

RRadjusted

RRCD

PC1

Fixed:ARR = 2.0

PC0 = 0.5

14

Conclusion of “Array Approach”

Great tool but you need to be honest to yourself

For all but one tool that I present today: Assuming conditional independence of CU and CM

given the exposure status If violated than residual bias may be overestimated

Drug exposure

Outcome

RREO

OREC RRCO

CU

CM

Hernan, Robins: Biometrics 1999

?

15

More advanced techniques

Wouldn’t it be more interesting to know How strong and imbalanced does a confounder

have to be in order to fully explain the observed findings?

RRCO OREC

16

Example:

Psaty et al: JAGS 1999;47:749

CCB use and acute MI.

The issue:

Are there any unmeasured factors that may lead to a preferred prescribing of CCB to people at higher risk for AMI?

OREC

RRCO

ARR = 1.57

ARR = 1.30

17

3. Rule Out Residual ConfoundingHow strong does an unmeasured confounder have to be to fully explain the observed findings?

The relationship between OREC and RRCD for a given ARR, RRC, PC, PE.

Data from Psaty et al.: Assessment and control for confounding by indication in observational studies. J Am Geriatri Soc 1999;47:749-54.a(prim)

RRCD PC PE ARR=1.57 OREC ARR=1.3 OREC

1.2 0.2 0.01 1.57 1.3 0.0311771.5 0.2 0.01 1.57 1.3 23.90 0.0143442 0.2 0.01 1.57 28.79 1.3 5.11 0.008733

2.5 0.2 0.01 1.57 9.03 1.3 3.42 0.0068633 0.2 0.01 1.57 5.97 1.3 2.78 0.005928

3.5 0.2 0.01 1.57 4.73 1.3 2.45 0.0053674 0.2 0.01 1.57 4.06 1.3 2.25 0.004993

4.5 0.2 0.01 1.57 3.65 1.3 2.12 0.0047255 0.2 0.01 1.57 3.36 1.3 2.02 0.004525

5.5 0.2 0.01 1.57 3.15 1.3 1.94 0.0043696 0.2 0.01 1.57 2.99 1.3 1.88 0.004244

6.5 0.2 0.01 1.57 2.87 1.3 1.84 0.0041427 0.2 0.01 1.57 2.77 1.3 1.80 0.004057

7.5 0.2 0.01 1.57 2.68 1.3 1.77 0.0039858 0.2 0.01 1.57 2.61 1.3 1.74 0.003924

8.5 0.2 0.01 1.57 2.56 1.3 1.71 0.003879 0.2 0.01 1.57 2.51 1.3 1.69 0.003824

9.5 0.2 0.01 1.57 2.46 1.3 1.68 0.00378210 0.2 0.01 1.57 2.42 1.3 1.66 0.003746

0.00

2.00

4.00

6.00

8.00

10.00

0 2 4 6 8 10

RRCD

OR

EC

ARR=1.57

ARR=1.3

drugepi.o

rg

18

Caution!

Psaty et al. concluded that it is unlikely that an unmeasured confounder of that magnitude exists

However, the randomized trial ALLHAT showed no association between CCB use and AMI

Alternative explanations: Joint residual confounding may be larger than

anticipated from individual unmeasured confounders Not an issue of residual confounding but other biases,

e.g. control selection?

19

Pros and cons of “Rule Out Approach”

Very easy to perform using Excel Meaningful and easy to communicate

interpretationStudy-specific interpretationProblems:Still assuming conditional independence of CU

and CM “Rule Out” lacks any quantitative assessment

of potential confounders that are unmeasured

20

External Adjustment

One step beyond sensitivity analysesUsing additional information not available in

the main studyOften survey information

21

Strategies to Adjust residual con-founding using external information

Survey information in a representative sample can be used to quantify the imbalance of risk factors that are not measured in claims among exposure groups

The association of such risk factors with the outcome can be assess from the medical literature (RCTs, observational studies)

Velentgas et al: PDS, 2007

Schneeweiss et al: Epidemiology, 2004

22

In our example:

Rofecoxib Acute MI

RREO

From Survey data in a

subsampleFrom medical

literature

OREC RRCO

[smoking,aspirin, BMI, etc.]

CU

CM

23

More contrasts

% Bias††

COX-2 (872) vs.

non-selective NSAIDs (1,302)

COX-2 (872) vs.

non-users (6,611)

COX-2 (872) vs.

naproxen (238)

Rofecoxib (244) vs.

naproxen (238) Potential confounder:*

Obesity (BMI30 vs. BMI<30)

-0.11 4.31 2.42 0.01

Aspirin use (use vs. non-use)

0.29 -0.08 -0.34 -1.28

Smoking (current vs. never)

-1.97 -2.41 -2.36 -0.61

Educational Attainment ( high school vs. >high school)

-2.36 -1.13 -3.67 -5.61

Income status ($20,000 vs. >$20,000)

-1.44 -1.08 -1.47 -1.65

Net confounding:

Sum of all negative biases: -5.88 -4.69 -5.08 -9.15

Weighted average: -1.56 -0.54 -1.86 -3.15

Sum of all positive biases: 0.29 4.31 -0.34 0.01

24

Sensitivity of Bias as a Function of a Misspecified RRCD :

-20

-15

-10

-5

0

5

10

15

20

1 1.5 2 2.5 3 3.5 4 4.5RRCD

Bia

s o

f R

RE

D i

n %

COX-2 vs. non-selective NSAIDsCOX-2 vs. non-usersCOX-2 vs. naproxenRofecoxib vs. naproxen

Literature estimate

RRCD = 1.7

Obesity (BMI >=30 vs. BMI<30)

25

Sensitivity towards a misspecified RRCO from the literature:OTC aspirin use (y/n)

-20

-15

-10

-5

0

5

10

15

20

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

RRCD

Bia

s o

f R

RE

D i

n %

COX-2 vs. non-selective NSAIDsCOX-2 vs. non-usersCOX-2 vs. naproxenRofecoxib vs. naproxen

Literature estimate

RRCD = 0.7

26

4. External AdjustmentGiven external information for selected factors on OREC from survey data and RRCD from the literature,

how much confounding is caused by not controling for these factors?

Data from Schneeweiss et al.: Assessment of bias by unmeasured confoundersin pharmacoepidemiologic claims data studies using external information. Epidemiology 2004, in press.

Unmeasured covariate: Aspirin (use vs. non-use) Bias as a function of misspecification of the RRCD from the literature:Data source: Lit MCBS MCBS assumed MCBSParameter: RRCD p(C) OREC true RRED p(E) app RRED CRR % biasSensitivity: varry const const const const calc calc

COX vs. 0.1 0.1 0.9 1 0.4 1.0093282 1.009 0.933

NSAID 0.2 0.1 0.9 1 0.4 1.0081979 1.008 0.820

0.3 0.1 0.9 1 0.4 1.0070929 1.007 0.709

0.4 0.1 0.9 1 0.4 1.0060124 1.006 0.601

0.5 0.1 0.9 1 0.4 1.0049555 1.005 0.496

0.6 0.1 0.9 1 0.4 1.0039215 1.004 0.392

0.7 0.1 0.9 1 0.4 1.0029096 1.003 0.291

0.8 0.1 0.9 1 0.4 1.0019192 1.002 0.192

0.9 0.1 0.9 1 0.4 1.0009495 1.001 0.095

1 0.1 0.9 1 0.4 1 1.000 0.000COX vs. 0.1 0.09 1.03 1 0.12 0.997608 0.998 -0.239

non-user 0.2 0.09 1.03 1 0.12 0.9978943 0.998 -0.211

0.3 0.09 1.03 1 0.12 0.9981752 0.998 -0.182

0.4 0.09 1.03 1 0.12 0.9984507 0.998 -0.155

0.5 0.09 1.03 1 0.12 0.998721 0.999 -0.128

0.6 0.09 1.03 1 0.12 0.9989863 0.999 -0.101

0.7 0.09 1.03 1 0.12 0.9992468 0.999 -0.075

0.8 0.09 1.03 1 0.12 0.9995024 1.000 -0.050

0.9 0.09 1.03 1 0.12 0.9997535 1.000 -0.025

1 0.09 1.03 1 0.12 1 1.000 0.000COX vs. 0.1 0.09 1.15 1 0.79 0.9892571 0.989 -1.074

naproxen 0.2 0.09 1.15 1 0.79 0.9905337 0.991 -0.947

0.3 0.09 1.15 1 0.79 0.9917884 0.992 -0.821

0.4 0.09 1.15 1 0.79 0.9930216 0.993 -0.698

0.5 0.09 1.15 1 0.79 0.9942339 0.994 -0.577

0.6 0.09 1.15 1 0.79 0.9954259 0.995 -0.457

0.7 0.09 1.15 1 0.79 0.996598 0.997 -0.340

0.8 0.09 1.15 1 0.79 0.9977507 0.998 -0.225

0.9 0.09 1.15 1 0.79 0.9988846 0.999 -0.112

1 0.09 1.15 1 0.79 1 1.000 0.000Rofecox vs. 0.1 0.1 1.6 1 0.51 0.9597175 0.960 -4.028

naproxen 0.2 0.1 1.6 1 0.51 0.9644945 0.964 -3.551

0.3 0.1 1.6 1 0.51 0.9691917 0.969 -3.081

0.4 0.1 1.6 1 0.51 0.9738113 0.974 -2.619

0.5 0.1 1.6 1 0.51 0.9783551 0.978 -2.164

0.6 0.1 1.6 1 0.51 0.9828249 0.983 -1.718

0.7 0.1 1.6 1 0.51 0.9872227 0.987 -1.278

0.8 0.1 1.6 1 0.51 0.99155 0.992 -0.845

0.9 0.1 1.6 1 0.51 0.9958085 0.996 -0.419

1 0.1 1.6 1 0.51 1 1.000 0.000

Unmeasured covariate: BMI (obese vs. non-obese)Data source: Lit MCBS MCBS assumed MCBSParameter: RRCD p(C) OREC true RRED p(E) app RRED CRR % bias

-20

-10

0

10

20

1 2 3 4 5 6 7 8

R RCD

-20-15-10-505101520

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

R RCD

-20

-10

0

10

20

1 1.5 2 2.5 3 3.5 4 4.5

R RCD

-20

-10

0

10

20

1 1.5 2 2.5 3 3.5 4 4.5

R RCD

-20

-10

0

10

20

1 1.5 2 2.5 3 3.5 4 4.5

R RCD

-20

-10

0

10

20

0 1 2 3 4 5 6 7 8

R RCD

COX-2 vs. non-selective NSAIDs

COX-2 vs. non-users

COX-2 vs. naproxen

Rofecoxib vs. naproxen

-20

-15

-10

-5

0

5

10

15

20

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

RRCD

Bia

s o

f R

R ED i

n %

COX-2 vs. non-selective NSAIDsCOX-2 vs. non-users

COX-2 vs. naproxenRofecoxib vs. naproxen Literature estimate

RRCD = 0.7

drugepi.o

rg

27

Limitations Example is limited to 5 potential confounders

No lab values, physical activity, blood pressure etc. What about the ‘unknow unknowns’?

To assess the bias we assume an exposure–disease association of 1 (null hypothesis) The more the truth is away from the null the more bias

in our bias estimate However the less relevant unmeasured confounders

become Validity depends on representativenes of sampling

with regard to the unmeasured confounders We could not consider the joint distribution of

confounders Limited to a binary world

28

Solving the Main LimitationsNeed a method

That addresses the joint distribution of several unmeasured confounders

That can handle binary, ordinal or normally distributed unmeasured confounders

Lin et al. (Biometrics 1998): Can handle a single unmeasured covariate of any

distribution But can handle only 1 covariate

Sturmer, Schneeweiss et al (Am J Epidemiol 2004): Propensity score calibration Can handle multiple unmeasured covariates of any

distribution

29

Summary

Sensitivity analyses for residual confounding are underutilized although they are technically easy to perform

Excel program for download (drugepi.org)The real challenge is the interpretation of

your findingsThis is all summarized in Schneeweiss PDS

2007