Clinical Translation of Omics Predictors:  Lessons Learned

David L DeMets, PhDUniversity of Wisconsin-Madison

Overview

Considerable interest exists in using genomic data to develop predictors for risk and for treatment response

A rigorous process needs to be followed to validate genomic biomarkers before being used to guide clinical treatment

Despite the large infrastructure in place at Duke, inadequaciesin the scientific process yielded very misleading results.

IOM received a request to provide guidance

IOM Report1. What happened? Precipitating Events at Duke University

2. Overview of the IOM Omics Committee Charge

3. Why did it happen? Lessons learned from the Duke Saga

IOM Recommendation Role of FDA / Center for Device & Radiological Health

Institute of MedicineThe National Academies Press, 2012

2003 The ‘Institute for Genome Sciences & Policy’ created at Duke

2004 Junior investigator Anil Potti joins laboratory of Joseph Nevins

‘06-’07 ▪ NEJM: (Potti et al) genomic biomarker for lung cancer ▪ Nature Medicine: ( Potti et al) genomic signatures

to guide use of chemotherapeutics▪ JCO: (Dressman et al) genomic biomarker for ovarian ca.▪ JCO: (Hsu et al) genomic biomarker for cisplatin-resistant

pts

‘06-’07 Keith Baggerly et al (MD Anderson statisticians) correspond with Duke authors, unable to reproduce their results

‘07-’08 ‘Clinical Genomics Studies Unit’ established at DukeClinical trials initiated using marker driven treatment

strategies ▪ BOP0801 in early stage Breast Cancer▪ TOP0602 in stage IIIB/IV Non Small Cell Lung Cancer▪ TOP0703 in early stage Non Small Cell Lung Cancer

Precipitating Events at Duke(IOM Report Table B-2)

ClinicalTrials.gov ID NCT00636441 BOP0801

NCT00509366 TOP0602

NCT00545948 TOP0703

Disease Breast cancer Lung cancer Lung cancer

Start date April 2008 February 2007 October 2007

Date listed in clinicaltrials.gov

March 2008 July 2007 October 2007

Patient accrual Intended Actual

270 56

8047

117 24

Sponsor DOD Eli Lilly / Duke / NCI Eli Lilly / Duke

PI Markom, MD (Duke) Vhlahovic, MD, MHS (Duke)

Ready, MD, PhD (Duke; Comp. Cancer Center)

Chemosensitivity predictor

Doxorubicin and docetaxel

Cisplatin Pemetrexed and Vinorelbine

Termination date 11/4/2010 11/4/2010 2/3/2011

Citations in clinicaltrials.gov

Potti et al., 2006a Bild et al., 2006; Potti et al., 2006a

Potti et al., 2006a,b; Potti et al., 2007b

Three Trials at Duke University

Problems Identified by Keith Baggerly & Kevin Coombes (1)

• MD Anderson biostatisticians asked to examine methodology in published papers in order to apply locally

• Could not reproduce results of publications by Duke investigators

• Problems first identified in letters to the authors and the journals

• After several direct communications with Duke investigators and after initial exchanges with journal editors, Baggerly et al were rebuffed.

• Rejected by lead authors, committees, and journal editors as a “squabble among statisticians”; later acknowledged to be “numerous missed signals”.

Problems Identified by Baggerly & Coombes (2)

• Specifically revealed a series of errors in a number of articles, including:

• Reversal of “sensitive/resistant” labels in training data

• Errors in test data, such as:• Only 84/122 test samples were distinct• Some samples labeled as both “sensitive” and “resistant”

• “Off by one” errors led to erroneous gene lists

• Some genes cited as evidence for biological plausibility were not output by software; 2 were not even on the arrays used.

(Extensively documented in Annals of Applied Statistics in 2009)

Precipitating Events at Duke (2)2008 Baggerly et al submit Letters to Editors regarding concerns

…rejected by Lancet Oncology, Nature Medicine

2009 Baggerly & Coombs publish in Annals of Applied Statistics

2009 ▪ NCI (McShane) asks University to examine validity of work and appropriateness of its extrapolation to the clinic

▪ Duke Cancer Protocol Review Committee endorses trials

▪ NCI recommends Duke investigate the 3 ongoing trials▪ NCI disapproves CALGB-30702 trial using genetic biomarker

▪ Duke suspends enrollment into these 3 trials▪ Duke IRB commissions two-person External Review panel

Baggerly sends Duke admin a report on key issuesNevins who asks it be withheld from panelDuke leadership honors Nevins request

▪ External Review panel: “Predictors are scientifically valid”▪ Duke re-starts the 3 clinical trials

Precipitating Events at Duke (3)

2010 ▪ NCI (McShane) rejects use of Lung Metagene Score (LMS) as being implemented in CALGB-30506 lung cancer trial

▪ NCI requests data from Potti regarding genetic biomarkers for premetrexed and cisplatin chemosensitivity

Meanwhile:

▪ Cancer Letter reports Potti incorrectly stated his credentials▪ Duke places Potti on admin leave; suspends the 3 trials again

▪ NCI and Duke request assistance from IOM: ~ To assess the scientific foundation of the 3 clinical trials ~ To identify appropriate evaluation criteria for

future tests based on omics technologies

▪ Duke informs NCI of flaws in key datasets used for validation▪ The 3 trials are terminated; Anil Potti resigns; The large scale retraction of publications begins…

Duke: Retracted Publications Key retracted papers by Nevins and Potti:

• 2006 Nature Medicine (Potti et al.) Cited 306 times

• 2006 New England Journal of Medicine (Potti et al.) Cited 350 times

• 2007 Lancet Oncology (Bonnefoi et al.) Cited 95 times

• 2007 Journal of Clinical Oncology (Dressman et al./ Hsu et al.) Cited 111 times / 60 times

Duke leadership identified 40 papers with Potti as co-author

• Two thirds will be partially or fully retracted

• Others may still be valid; pending evaluation (as of 8/11/2011)

• Surveyed 162 co-investigators

Omics Characteristics

• Complex, high dimensional data

• Many more variables than samples

• High risk that computational models will overfit data

Omics-based Test

• Composed or derived from multiple molecular measurements and interpreted by a fully specified computational model to produce a clinically actionable result

Errors & Problems may be Attributable to:

• Inherent complexity

• Inadequate attention to detail

• Intentional misrepresentation

• Inadequate validation

ma

IOM Omics Committee Charge

1. Obtain insights from the 3 cancer clinical trials conducted by Duke and using omics-based tests

2. Develop evaluation criteria to determine when omics-based tests are fit for use in a clinical trial.

3. Recommend ways to ensure adherence to the development framework.

IOM Multidisciplinary Committee* GILBERT S. OMENN (Chair), University of Michigan Medical School

CATHERINE D. DEANGELIS, Johns Hopkins School of Medicine

DAVID L. DEMETS, University of Wisconsin-Madison * THOMAS R. FLEMING, University of Washington GAIL GELLER, Johns Hopkins University

JOE GRAY, Oregon Health & Science University Knight Cancer InstituteDANIEL F. HAYES, University of Michigan Comprehensive Cancer Center I. CRAIG HENDERSON, University of California San Francisco* LARRY KESSLER, University of Washington School of Public Health STANLEY LAPIDUS, SynapDx CorporationDEBRA LEONARD, Weill Medical College of Cornell UniversityHAROLD L. MOSES, Vanderbilt-Ingram Cancer Center WILLIAM PAO, Vanderbilt University School of Medicine REBECCA D. PENTZ, Emory School of Medicine* NATHAN D. PRICE, Institute for Systems BiologyJOHN QUACKENBUSH, Dana-Farber Cancer Institute ELDA RAILEY, Research Advocacy Network

DAVID RANSOHOFF, University of North Carolina School of Medicine  E. ALBERT REECE, University of Maryland School of Medicine * DANIELA M. WITTEN, University of Washington

Chapter 1: Overview of the statement of task and scope

Chapter 2: Science, technology, and discovery process for omics-based tests

Chapter 3: Test development and analytical and clinical/biological validation

Chapter 4: Evaluation of tests in clinical trials and ultimately for clinical use

Chapter 5: Roles of investigators, institutions, journals, funders, and FDA

Chapter 6: Overview of lessons learned from the case studies

Appendix A: Summary of 8 case studies

Appendix B: Summary of the Duke University omics-based tests

Overview of IOM Omics Report (270 pages)

Some Specific Issues at Duke • Genomic trials conducted outside the Duke Cancer Center

• Trials conducted within the Institute for Genomics, which is outside the medical center

• Lacked clinical trials infrastructure and oversight

• Genomic predictors were published in leading journals, even though these biomarkers were not appropriately validated• Test data not locked down• Algorithm not locked down• Not validated on blinded data set• Some of the predictors gave incorrect risk scores

(backwards)

• Clinical trials launched at Duke used these invalid predictors that• Aided in the selection of patients for their risk• Aided in making an ‘optimal’ choice among available drugs

Some Specific Issues at Duke

• Duke IRB interpreted FDA lack of response as an approval for not needing an IDE• Inadequate communication with regulatory authorities• Did not recognize an algorithm is a device

• Statistical input largely was from a junior faculty member whose funding heavily dependent on the Nevins-Potti lab; • offered ownership in the spin-off companies• did not have independence necessary to raise issues

• Institution did not recognize the seriousness of the issues• Did not want to challenge a proven senior investigator• Believed the issues to be statistical subtleties• Did not fully understand issues until a meeting with NCI

late in the process

Goals of Committee’s Recommendations

GOAL I: Define best practices for discovery and translation of an omics-based test into a clinical trial.

[Recommendations 1-3]

GOAL II: Recommend actions to ensure adoption of and adherence to the development and evaluation process.

[Recommendations 4-7]

Three Stages of Omics Test Development

1. Discovery

2. Test Validation

3. Evaluation for Clinical Utility and Use

Omics-Based Test Development Framework

Three Stages of Omics Test Development

1. Discovery

2. Test Validation

3. Evaluation for Clinical Utility and Use

Test Validation Phase

FDA Drug & Device Regulation

• Device laws & regulations for devices not same as for drugs; ie “devices ≠ drugs”

• Few of us are expert for all aspects of drug development and review

• Even fewer are expert in medical device development & approval

• Not being familiar with device regulations can lead to problems

FDA Investigational Device Exemption (IDE)• Investigational device exemption (IDE) refers to the

regulations under 21 CFR 812. • An approved IDE means that the IRB (and FDA for significant

risk devices) has approved the sponsor’s study application and all the requirements under 21 CFR 812.20 are met.

• Significant risk device (SR device)• (1) is intended as an implant and presents a potential for

serious risk to the health, safety, or welfare of a subject; • (2) is for use in supporting or sustaining human life and

represents a potential for serious risk to the health, safety, or welfare of a subject;

• (3) is for a use of substantial importance in diagnosing, curing, mitigating, or treating disease or otherwise preventing impairment of human health and presents a potential for serious risk to the health, safety, or welfare of a subject; or

• (4) otherwise presents a potential for serious risk to a subject.

IDE Approval• In order to conduct a significant risk device study, prior to enrolling

subjects, a sponsor must:• Submit to the FDA, a complete IDE application (§812.20) to

FDA for review and obtain FDA approval of the IDE (usually within 30 days);

• Submit to the IRBs, the investigational plan and report of prior investigations (§812.25 and §812.27) to the IRB at each institution where the study is to be conducted for approval

• FDA recommends meeting with them prior to the IDE submission• Determination meeting

• discuss the type of valid scientific evidence to demonstrate that the device is effective for its intended use

• Agreement meeting• If an agreement regarding the parameters of an

investigational plan (including a clinical protocol), the terms of the agreement are put in writing for the FDA the record.

IDE Implementation

Labeling - The device must be labeled (§812.5) and must bear the statement "CAUTION - Investigational Device. Limited to investigational use."

Distribution - distributed to qualified investigators §812.43(b).Informed Consent - Each subject must be provided with and sign an

informed consent form before being enrolled (21 CFR 50)Monitoring - All investigations must be properly monitored with

approved protocols under §812.46.Prohibitions - Commercialization, promotion, and misrepresentation

of an investigational device are prohibited (§812.7).Records and Reports - Sponsors and investigators are required to

maintain specified records and make reports to investigators, IRBs, and FDA (§812.140 and §812.150).

When is an Algorithm a Device?(Based on Talk by Seth Mailhot, Attorney, at UW Conf in 9/14)

Definition of a medical device:

• Sec. 201(h) defines a device as an instrument, apparatus, implement, machine, contrivance, . . . or other similar or related article, including any component, part, or accessory, which is intended: for use in the diagnosis of disease or other conditions, in the cure, mitigation, treatment, or prevention of disease, to affect the structure or function of the body

• Applying definition of a device to software

Software that is not a device

• Used to log, track, evaluate, make decisions or suggestions related to general health

• Performs functionality of an electronic health record

• Used for training purposes

• Not marketed for a specific medical indication

New 2014 (April) Report

• Issued by FDA, FCC and Office of National Coordinator for Health Information Technology

Health IT in 3 Functional Categories Administrative Health Management Medical Device Health IT

• FDA will focus on the latter category Examples

• Computer aided detection• Real time alarms• Assist in treatment decisions

Software Related Categories• Medical Device Data Systems

Eg CV or cancer risk assessment, diabetes management

• Mobile Medical Apps Eg Performs patient specific analysis,

diagnosis or treatment recommendations• Laboratory Information Systems• Picture Archiving and Communication

Systems (PACS)• Remote Patient Monitoring

FDA Regulation Implications

• Device/algorithm registration• IDE approval process

• Conduct studies according to IDE• Premarket submission• Design controls (21 CFR 820.30)• Good manufacturing • Implementation of algorithm• Reporting of adverse events• Reporting of recalls

Verification & Validation

• Provide objective evidence• Algorithm is valid Software implementation is tested Platform on which software runs is reliable

• Conducted via• Code & document inspection• Walkthroughs, simulations

• Confirmation that Algorithm/Software conforms to user needs &

intended uses User site testing

Test Validation Phase• Regulatory oversight for omics based tests differs from drugs

• An omics-based test validation consists of both

• the data-generating assay and

• the fully specified computational model

• Software implementation

• Needs to be validated on an independent blinded data set FDA approval or clearance as a device

Use a Laboratory Developed Test (LDT) process / FDA

Three Stages of Omics Test Development

1. Discovery

2. Test Validation

3. Evaluation for Clinical Utility and Use

Evaluation for Clinical Utility and Use

Clinical Utility: “Evidence of improved measurable clinical outcomes . . . compared with current management without [omics] testing.”

Clinical utility is not assessed by FDA or in the LDT process

Lack of FDA review does not mean lack of clinical utility

Process of gathering evidence to support clinical use should begin before test is introduced into clinical practice

Evaluation for Clinical Utility and Use

RECOMMENDATION 3:

a. Investigators should communicate early with the FDA

regarding Investigational Device Exemption (IDE) process.

b. Omics-based tests should not be changed during the clinical trial without a protocol amendment and discussion with the FDA. A substantive change to the test may require restarting study.

Goals of Committee’s Recommendations

GOAL II: Recommendations to ensure adoption of andadherence to the development and evaluation process

Investigators and institutions are responsible for the scientific culture in which omics-based tests are developed.

Recommendations also for funders, FDA & journals

Recommendations are not intended to create new barriers, but rather to maintain the integrity of the scientific process

Summary: Lessons Learned (1)

• Importance of:

Data provenance and data management

Locking down the computational model required

Making data, code, and other information publicly available

Independent confirmation of the Test & Test validation

Effective multidisciplinary collaboration

Institutional oversight, including fresh review of the science when serious criticisms are raised or clinical trials or spinoff companies are proposed

Consultation with FDA and submission of IDE

Summary: Lessons Learned (2)

• Limitations of:

Peer review process

Ability of funders and journals to address scientific controversies

Ability of institutions to objectively review work of their own faculty when issues or controversies arise

To download or read the report online:

www.nap.edu