Analysis of Observational Health Care Data Using SASlocalcontrolstatistics.org/other/LCdraft.pdfAnalysis of Observational Health Care Data Using SAS Douglas Faries, Robert L. Obenchain,

Local Control Page 1 of 19

Analysis of Observational Health Care Data Using SAS Douglas Faries, Robert L. Obenchain, Josep Maria Haro and Andrew C. Leon

Carey, NC: SAS Press Anticipated publication date: January 2010

Outline of Contents of Chapter 7

The Local Control Approach using JMP Robert L. Obenchain, PhD, FASA

Principal Consultant, Risk Benefit Statistics LLC, Carmel, Indiana Adjunct Professor, Biostatistics, IU Medical School, Indianapolis

Abstract

The local control approach to adjustment for treatment selection bias and confounding in observational studies is illustrated here using JMP because local control is best implemented and applied in highly visual ways. The local control approach is also unique because it hierarchically clusters patients in baseline covariate x-space; applies simple nested analysis of variance (ANOVA) models (treatment within cluster); and ends up being highly flexible, non-parametric, and robust. Although the local control approach is classical rather than Bayesian, its primary output is a full distribution of local treatment differences (LTDs) that contains all potential information relevant to patient differential responses to treatment. All concepts are illustrated using freely distributable data on 10,000 patients that were generated to be like those from a published cardiovascular registry containing 996 patients.

7.1 Introduction

7.1.1 Fundamental Problems with Randomization in Human Studies 7.1.2 Fundamental Local Control Concepts 7.1.3 Statistical Methods most useful in Local Control 7.1.4 Contents of the remaining Sections of Chapter 7


7.2 Some Traditional Analyses of Hypothetical Patient Registry Data

7.2.1 Introduction to the LSIM10K Data Set 7.2.2 Analyses of Mortality Rates and Costs using Covariate Adjustment 7.2.3 Analyses of Mortality Rates Using Estimated Propensity Score Deciles 7.2.4 Analyses of Mortality Rates and Costs using Inverse Probability Weighting

7.3 The Four Phases of a Local Control Analysis

7.3.1 Introduction to the Four Tactical Phases of a Local Control Analysis 7.3.2 Phase One: Revealing Bias in Global Estimates by Making Local Comparisons 7.3.3 Phase Two: Determining Whether an LTD Distribution is Salient 7.3.4 Phase Three: Performing Systematic Sensitivity Analyses

7.3.4.1 Alternative Clusterings that Readers Can Try on Their Own 7.3.4.2 Review of Clustering Concepts Useful in Sensitivity Analysis

7.3.5 Phase Four: Identifying Baseline Patient Characteristics Predictive of Differential Treatment Response

7.4 Conclusions

Acknowledgments

References Appendix: Propensity Scores and Blocking/Balancing Scores

A.1 Review of the Fundamental Theorem of Propensity Scoring A.2 Practical Problems with Estimated Propensity Scores A.3 Range of Blocking/Balancing Scores: From Most Fine to Most Coarse A.4 Cluster Membership is an Asymptotic Blocking/Balancing Score


Selected Chapter 7 Figures in Color (In the SAS Press publication, all figures are in Gray Scale.)

Figure 7.1 Marginal and Within Propensity-Score-Decile Distributions of ejfract by treatment (0 or 1)

20

30

40

50

60

70

ejfr

act

0 1

trtm

20

30

40

50

60

70

ejfr

act

10 11 20 21 30 31 40 41 50 51 60 61 70 71 80 81 90 91 100

101

PSdec By Tr t


Figure 7.2 Estimated within Decile propensity score Means vs. Observed Fractions of Patients Treated

Local Fraction Treated = 0.0101 + 1.0223 * Mean Estimated PS within Decile

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Trt

Fra

ctio

n

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

MeanPS


Figure 7.3. Numbers of Treated and Untreated Patients by Propensity Score Decile

Figure 7.4 Detailed Histograms of Estimated Propensity Scores

0

1 0 0

2 0 0

3 0 0

4 0 0

5 0 0

6 0 0

7 0 0

8 0 0

9 0 0

1 0 0 0

Cou

nt

1 2 3 4 5 6 7 8 9 1 0

PS d e c ile

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Untreated

Treated

= 0.397

= 0.521


Figure 7.5 LTDs for 6-month Mortality (Treated minus Untreated) by PScore Decile

LTD Main Effect

= .03389

Figure 7.6 IPWs for Treated or Untreated Patients derived from PS Estimates

-0.1200

-0.1000

-0.0800

-0.0600

-0.0400

-0.0200

0.0000

0.0200

LTD

mor

t

0 1 2 3 4 5 6 7 8 9 10 11

PSdec ile

2.0

4.0

6.0

8.0

10.0

IPW

0 1

trtm


Figure 7.7 IPW versus Unweighted Predictions of 6-Month Risk of Mortality

Treated or Untreated

RiskCA = 0.00259 + 0.8421*RiskIPW

0.00

0.10

0.20

0.30

Ris

kCA

0.0 0.1 0.2 0.3

RiskIPW


Figure 7.8 Customized JMP Analyze Menu

Figure 7.9 Open Data File Dialog Box with the LSIM10K Data Set highlighted


Figure 7.10 JMP Select Columns Dialog Box for Local Control

Figure 7.4 Vertical Cuts of the JMP Clustering Dendrogram Produce a Requested Number of Clusters from the Hierarchy

100Clusters

50Clusters

2Clusters

25Clusters


Figure 7.6 LC Unbiasing TRACE for the LTD Main Effect in the First Outcome

First Outcome: Y1 = Mortality within Six Months Y1 = Across Cluster Average LTD Outcome

Y1 plus Two Sigma = Upper Limit for Average LTD Y1 minus Two Sigma = Lower Limit for Average LTD

NCreq = Number of Clusters Requested

Table showing the across cluster summary statistics displayed in Figure 7.6

NCreq NCinfo Y1 LTD MAIN

Y1 Local Std Err

Y1 Lower Limit

Y1 Upper Limit

1 1 0.0250 0.00313 0.0313 0.0188 5 5 -0.0270 0.00319 -0.0333 -0.0206 10 10 -0.0306 0.00326 -0.0371 -0.0241 20 20 -0.0315 0.00338 -0.0383 -0.0248 50 50 -0.0351 0.00340 -0.0419 -0.0283 100 100 -0.0372 0.00346 -0.0441 -0.0302 200 199 -0.0393 0.00351 -0.0463 -0.0322 500 492 -0.0398 0.00360 -0.0470 -0.0326 700 660 -0.0383 0.00363 -0.0456 -0.0311 900 816 -0.0365 0.00363 -0.0438 -0.0293

NCreg = Number of Clusters Requested

NCinfo = Number of Informative Clusters Found

-0.05

-0.045

-0.04

-0.035

-0.03

-0.025

-0.02

-0.015

Y

1 10

532

100503020

100050

0

300

200

NCreq


Figure 7.7 LC Unbiasing TRACE for the LTD Main Effect in the Second Outcome

Second Outcome: Six Month Cumulative Cardiac Related Cost Y2 = Across Cluster Average LTD Outcome

Y2 plus Two Sigma = Upper Limit for Average LTD Y2 minus Two Sigma = Lower Limit for Average LTD

NCreq = Number of Clusters Requested

Table showing the across cluster summary statistics displayed in Figure 7.7

NCreq

NCinfo Y2 LTD

MAIN Y2 Loc Std Err

Y2 Low Limit

Y2 Upr Limit

1 1 255.08 206.21 -157.33 667.50 5 5 664.94 208.99 246.96 1082.92 10 10 698.80 214.19 270.42 1127.19 20 20 341.77 220.48 -99.19 782.74 50 50 -71.92 214.84 -501.59 357.75 100 100 -73.59 212.71 -499.02 351.83 200 199 -24.28 209.32 -442.93 394.37 500 492 127.11 184.59 -242.08 496.30 700 660 248.93 183.97 -119.01 616.87 900 816 312.96 181.35 -49.74 675.66

All results expressed in 1998 US Dollars ($)

-600

-400

-200

0

200

400

600

800

Y

1 10

532

100503020

100050

0

300

200

NCreq


Figure 7.8 LTD Graphics and Summary Stats for 816 Informative Clusters

LTD Distribution for Y1 = mort6mo LTD Distribution for Y2 = cardcost Mean LTD 0.0385 Mean LTD 168.74 Std Dev 0.14882 Std Dev 4962.92 Std Err Mean 0.00150 Std Err Mean 50.09 Upper 95% Mean 0.0356 Upper 95% Mean 266.93 Lower 95% Mean 0.0415 Lower 95% Mean 70.55 Inform Clust Pats 9816 Inform Clust Pats 9816 Nonparametric Density Nonparametric Density Kernel Std 0.02131 Kernel Std 710.55

Note also that neither of the above LTD Distributions is very much like the “best fitting” Normal distributions displayed in RED on the above histograms and probability plots.

-4

-3

-2

-1

0

1

2

3

4

Nor

mal

Qua

ntile

Plo

t

-1

-0.8

-0.6

-0.4

-0.2 0

0.2

0.4

0.6

0.8 1

-4

-3

-2

-1

0

1

2

3

4

Nor

mal

Qua

ntile

Plo

t

-400

00

-300

00

-200

00

-100

00 0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0


Figure 7.9 Bubble Plot of LTD Scatter for 816 Informative Clusters

Bubble Area = Total Number of Patients within Cluster


Figure 7.10 Artificial LTD Distribution Dialog Box


Figure 7.11 JMP Side-by-Side Comparison using the Spread Option of the Artificial and Observed LTD Distributions

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

LTD

Artificial Observed

LTDtype


Figure 7.14 JMP Multivariable Model for Predicting mort6mo LTDs from Seven Baseline Patient x-Characteristics

Summary of Fit RSquare 0.144396RSquare Adj 0.141948Root Mean Square Error 0.188451

Analysis of Variance

Source DF Sum of Squares

Mean Square

F Ratio

Model 28 58.65786 2.09492 58.9892

Error 9787 347.57256 0.03551 Prob > F

C. Total 9815 406.23042 <.0001

-1.0

-0.7

-0.4-0.2

0.1

0.4

0.7

1.0

Y1L

TD

Act

ual

-1.0 -0.7 -0.4 -0.1 0.2 0.4 0.6 0.8 1.0

Y1LTD Predicted P<.0001

RSq=0.14 RMSE=0.1885


Figure A.1. The x-space geometry of linear functionals from logistic regression

Figure A.2 The x-space geometry of small, compact, and numerous clusters that intersect a thin slab of rounded propensity score estimates from logistic regression

Constant PS Estimate Calipersfrom Logistic Regression

x2

x3

x1

LinearLinearFunctionalFunctional

3-D Slab

UnsupervisedNo PS Estimates Needed

x2

x3

x1

3-D Clusters (Informative orUninformative)


Table 7.1. Nested ANOVA table with effects for treatment within cluster

Source

Degrees of Freedom

Interpretation

Clusters

(Subgroups)

K = Number of Clusters

Cluster Means are Local Average

Treatment Effects (LATEs) when Xs are Instrumental Variables (IVs)

Treatment within Cluster

I = Number of Informative Clusters K

Local Treatment Differences (LTDs)

are of interest when X Variables either are or are not IVs.

Error

Number of

Patients K I

Outcome Uncertainty

and/or Model Lack of Fit

Table 7.3. Five general approaches to analysis of observational data

Covariate Adjustment (CA) using Multivariable Models

History: Generalization of ANOVA and regression models. Advantages: Ubiquitous; widely taught and well accepted; implemented in all statistical analysis packages. Disadvantages: Essentially ignores imbalance (for example, always uses all available data); global, parametric models are difficult to visualize and thus may be unrealistic; results are frustratingly sensitive to model specification details; p-values can be small simply due to large sample sizes.

Inverse Probability Weighting (IPW)

History: Heuristic modification of CA somewhat similar to Horvitz-Thompson adjustment in sample surveys. Advantages: As easy to perform as CA; does attempt to adjust for local variation in treatment selection fraction (imbalance); requires software for (diagonally) weighted regression. Disadvantages: Basically the same as CA; IPW focuses on up-weighting rarely observed outcomes (never really ignores observations from uninformative clusters); basic variance assumptions somewhat counterintuitive (least frequently observed outcomes are treated as being more precise)


Propensity Score (PS) Matching and Subgrouping

History: Fundamental PS theory has attracted more and more attention over the last 25 some years; motivates use of traditional matching and subclassifying approaches. Advantages: Intuitive, weak assumptions; widely applicable; many results easily displayed using histograms. Disadvantages: Results may be less precise than they appear (are reported) to be; no built-in sensitivity analyses; not a standard method implemented in current commercial statistical software.

Instrumental Variable (IV) Methods

History: Adding IV variable(s) to structural equation models can identify causal effects when the given x-covariates are correlated with model error terms due to endogenous effects, omitted covariates, or errors in variables. Newest IV approaches use patient clustering and nonparametric, local PS estimates. Advantages: Near the top of the theoretical pecking order. Disadvantages: IV assumption is very strong and not testable; implemented only in some commercial statistical packages.

Local Control Methods using Patient x-clusterings (Unsupervised Learning)

History: Generalization of nested ANOVA (treatment within cluster) and hierarchical models. Advantages: Intuitive, weak assumptions; widely applicable; guaranteed asymptotic balancing scores finer than propensity scores; inferences based upon bootstrap confidence or tolerance intervals; built-in sensitivity analyses. Disadvantages: Quite new; completely different focus from that of traditional parametric models; not a standard method implemented in current statistical software packages.

Documents

Analysis of Observational Health Care Data Using SASlocalcontrolstatistics.org/other/LCdraft.pdfAnalysis of Observational Health Care Data Using SAS Douglas Faries, Robert L. Obenchain,