Pros and Cons of Clinical Trials vs. Observational Studies

Beth Devine PCORP Summer Institute

July 14, 2015

• Randomized Trials • Definitions: Pragmatic-explanatory continuum

indicator summary (PRECIS) • Examples

• Observational Studies • Definitions and types of observational data • Advantages and disadvantages of observational

data research • Good practices in observational data research • Examples

Outline

• Define the RCT continuum • Describe appropriate use of pragmatic vs.

explanatory trials

• List and define the major types of observational studies

• Describe uses, advantages and disadvantages of the major types of observational studies

• Locate good research practice tools for use when conducting observational studies

Learning Objectives

Part I: RCTs

Randomization is the ONLY way to guarantee unbiasedness, particularly as it relates to

unknown or unrecorded prognostic factors!

Benefit of RCTs

• Pragmatic – describes trials that help users choose between options for care

• Explanatory – describes trials that test causal research hypotheses

• Both are randomized • Represents a spectrum

• Impossible to perform a purely pragmatic or purely explanatory trial

• Reflects judgments made by trialists in study design phase

Pragmatic-Explanatory Continuum

Thorpe. CMAJ. 2009;188(10):E47-E57

PRECIS Tool – 10 ‘extreme’ domains Explanatory Pragmatic

Participants Restricted Take all ‘comers’ Interventions Strict instructions Flexible instructions

Seasoned practitioners/settings

Full range of practitioners/settings

Comparator Restricted (placebo) ‘Usual practice’ Standardized ‘Ordinary’ attention

Follow-up Frequent/extensive No formal F/U; registries Direct/immediate/ surrogate

Objectively measured; Assessed under usual conditions

Compliance (participant)

Closely monitored/followed

Unobtrusive

Adherence (provider) Closely monitored Unobtrusive Analysis Intent to treat All patients

Thorpe. CMAJ. 2009;188(10):E47-E57

PRECIS Tool

Thorpe. CMAJ. 2009;188(10):E47-E57

PRECIS Tool - Examples

Thorpe. CMAJ. 2009;188(10):E47-E57

Part II: Observational Studies

• Subject not randomized • Treatments/exposure delivered in natural settings • Cohort studies – identify exposure; then outcome(s)

• Retrospective – cross-sectional or longitudinal • All data collected before commencement of study

• Prospective – typically longitudinal • Consequential outcomes data collected after

commencement of study

• Case-control studies – identify outcome; then exposure(s) • Always retrospective - longitudinal

Observational Studies

• Useful for characterizing a population • Useful in CER/PCOR • Avoids voluntary participation

• Generalizable to target population; often not the case in RCTs

• Faster and cheaper • Data collected as part of larger surveillance goals • Usually does not require expensive protocol

Advantages of observational studies

• Study design • Treatments, exclusion/inclusion criteria/follow-up

period etc. determined by the data at hand

• Outcomes • All relevant outcomes may not be available

• Causal Inference – eliminating bias • Must deal with confounders • Requires use of more advanced statistical

techniques

Disadvantages of observational studies

• RCTs often produce internally valid estimates, but may not be externally valid (generalizable)

• Internal validity is NECESSARY but NOT

SUFFICIENT for external validity. • Observational studies cannot provide externally

valid estimates if they are not internally valid.

A word about validity

• CER (with observational data) only relevant when there is clinical equipoise • In presence of strong treatment

preferences it is difficult to control for confounding or bias

• Retrospective data are most useful here • Specify hypothesis, up front • Specify population, comparators,

outcomes of interest

Good Practice Recommendations

• Specify study design

• Strongest design always includes a control group

• Cohort – pre/post • Good to assess one or more outcomes

• Can assess one or more exposures

• Case-control • Good to assess rare outcome (usually one)

• Can assess many exposures

• Case-Crossover Designs • Individuals serve as their own controls

• Case-Time-Control Designs • Case-crossover design with external control group to control for

temporal trends

Good Practice Recommendations

• Two types of bias • Observed – can address by including covariates or

stratification • Unobserved – requires advanced techniques

• Address presence of treatment effect heterogeneity

• Issues in estimation – sample size etc. • Issues in interpretability of results – level of aggregation • Issues in generalizability– what does the mean effect

tell us • What are the moderators of treatment effect – mostly

use baseline characteristics

Good Practice Recommendations

• Observational data invaluable source of data for CER

• Clear descriptions of hypothesis, study design and methods are important for a good observational CER study

• Confounding is main issue • Often advanced statistical methods are needed

to address confounding

Good Practice Recommendations

• Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. http://www.effectivehealthcare.ahrq.gov/registries-guide-3.cfm.

• Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guideline. http://www.strobe-statement.org/

• REporting of studies Conducted using Observational Routinely-collected Data (RECORD). http://record-statement.org/

• ISPOR, Good Pharmacoepidemiology Practices, 2008.

• ISPOR Good Research Practices for Observational Data (many) http://www.ispor.org/workpaper/practices_index.asp

• European Network of Centers for Pharmacoepidemiology and Pharmacovigilance (ENCePP). http://www.encepp.eu/

• Luce BR et al. Principles for planning and conducting comparative effectiveness research. Journal of Comparative Effectiveness Research 2012; 1(5): 431-440.

• ISPE Guidelines for Good Pharmacoepidemiology Practices (GPP). https://www.pharmacoepi.org/resources/guidelines_08027.cfm

References

Case Study I: Prospective Cohort CER Study of Intermittent Claudication (IC): Impact of Intervention Type on Patient Function and Health-Related Quality of Life

Devine Alfonso-Cristancho Yanez, Edwards Patrick, Armstrong Devlin, Symons, Thomason, Meissner, Clowes, Lavallee, Kessler, Flum, and CERTAIN Collaborative Funded by AHRQ R01HS020025 (PI: Flum)

Evidence Generation

Clinical Practice Partners

Dissemination & Implementation

Patient Voices

Clinician Offices

Long-term Care

Facilities

Hospitals

Study Design: Multisite, longitudinal, prospective, observational cohort study conducted from 2011-2013 Aims: Compare baseline, 6 & 12 month functional, health-related quality of life and symptoms among subjects receiving medical management vs. surgical or endovascular procedures for treatment of intermittent claudication Hypothesis: At 12-months, surgical and endovascular procedures are associated with greater improvements in function, health-related quality of life, and symptoms than the medical management cohort

IC Study Methods

Lessons Learned from the IC Study

• Patient reported outcome (PRO) measures can be used as primary and secondary outcomes

• Recruitment efforts are often Intense • Always adjust analyses for baseline characteristics • Data collected from both electronic health records and

directly from patients provides the opportunity to compare patient reported outcomes to clinically reported outcomes

• Engage a biostatistician to assist with study design, power calculations and analyses

• Infrastructure is expensive to build initially; once built, additional studies can be conducted for modest incremental investment

Case Study II: Retrospective Cohort Estimating the costs of atrial fibrillation and associated adverse events Forrester SH, Li M, Roth G, Devine EB

Study Design: • Matched (1:4), retrospective cohort study of patients ≥18

years old with incident AF(ICD-9 427.31) between 2008 and 2010

• Aim: • Estimate the incremental costs of ‘events’ (ischemic

stroke, myocardial infarction, systemic embolism, intracranial hemorrhage, or GI bleed) in patients with AF

Atrial fibrillation (AF) study methods

Lessons Learned from the A fib Study

• Time invested in developing study design and protocol, a priori, is a must

• When using administrative claims data, clearly define run-in period; date of index diagnosis; date of intervention; define adequate follow-up period • Illustrations are helpful in refining these design

characteristics – draw it out!

• Control for baseline characteristics and potential confounders

• Engage a biostatistician to assist with study design, power calculations and analyses

Thank You! Questions? bdevine@uw.edu