Embed Size (px)
Citation preview
EVALUATION DIGITALER THERAPEUTIKA
NEUE STUDIENDESIGNS UND REAL WORLD EVIDENCE
Lars G. Hemkens
Heidelberg, 27. Februar 2020
web version
University of Basel, Department of Clinical Research
…
“Women havefewer teeth thanmen“
Aristoteles; HA II 3, 501b19
“Pneumonia is one ofthe diseases in whicha timely venesection
may save life“
Sir William Osler; Principles and Practice of Medicine;1892
Bloodletting for pneumonia
Mercury for syphilis
Lobotomy for psychiatric diseases
Heroin for cough
Bedrest in myocardial infarction
Low-fiber diet in diverticulosis
Babys sleep on stomach
Surgery for peptic ulcers
Intensive glucose lowering in type 2 diabetes
Hormone replacement therapy in women
Vitamine E in cardiac diseases
Antiarrhythmic in myocardial infarction
Stents in stable CVD
„Approximately 90% of new drugs entered into clinical development on
promising preclinical findings
fail to yield sufficient efficacy and
safety to receive …(FDA) license“
Benjamin et al. PLoS Biol 2017 (citing Hay et al. Nature Biotech 2014; 32: 40-51).
)
BIG DATAROUTINELY COLLECTED DATA
Insurance data / claims data
Electronic Health Records
Registries
Microsoft
Bing
Amazon
Netflix
Wearables
Fitbit
Firefox
IoT (Internet of Things)
Real World Data
“Real-world data are the data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. RWD can come from a number of sources, for example:
Electronic health records (EHRs)
Claims and billing activities
Product and disease registries
Patient-generated data including in home-use settings
Data gathered from other sources that can inform on health status, such as mobile devices”
www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
Accessed 23 Sep 2019
"Data collection under real-world circumstances maximizes representativeness and generalizability, minimizes costs and effort, and allows the capture of information in large populations and many clinical events in large data sets that are continuously updated and cover long periods."
Hemkens et al. CMAJ 2016
Safeguards of clinical trial research often lacking
Protocols
Registration
Research networks
Prospective Analysis
...
CONFOUNDING
From http://tylervigen.com/view_correlation?id=28807
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)
From http://tylervigen.com/view_correlation?id=28672
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)
From http://tylervigen.com/view_correlation?id=28669
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)
From http://tylervigen.com/view_correlation?id=28829
Thanks to Tyler Vigen for sharing (accessed 22 February 2020)
Nationwide Big Data
From www.tylervigen.com/spurious-correlations
Thanks to Tyler Vigen for sharing (accessed 8 March 2017)
Nationwide Big Data
From www.tylervigen.com/spurious-correlations
Thanks to Tyler Vigen for sharing (accessed 8 March 2017)
Byberg et al. Clinical and Translational Allergy 2016;6:33; Figure S1
URL: https://ctajournal.biomedcentral.com/articles/10.1186/s13601-016-0124-9
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made
Some Confounder RelationshipsBMI and physical activity in early childhood with atopy
Paternal factors?Ethnicity?Resp. infections?Socioeconomics?Patient preferences and values?
Wallach et al. (PrePrint/under review)www.researchsquare.com/article/8e138689-12d5-43fa-87c9-9766d9e11a53/v1
Some Confounder RelationshipsAlcohol and Cardiovascular disease
Chu et al. Int J Epidemiol. 2020
97 studies:
Pickyour
effect
Ioannidis JAMA 2005
"Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008)."
16 clinical questions / 16 RCD studies
36 subsequent RCTs
17 275 patients
835 deaths
Propensity Scores
Mortality
Hemkens et al. BMJ 2016;352:i493
URL: http://www.bmj.com/content/bmj/352/bmj.i493.full.pdf
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made
RCD studies find 31% larger benefits than later RCTs(ROR 1.31)
In various re-analyses ROR from 1.25 to 1.58
Hemkens et al. BMJ 2016;352:i493
URL: http://www.bmj.com/content/bmj/352/bmj.i493.full.pdf
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made
“Big Data presents manyexciting new opportunities, but there is no underlyingrevolution in the logic of howclaims about the effects oftreatment should be justified.” S Goodman, S Schneeweiss, M Baiocchi JAMA 2017
JAMA 2017
…
Non-Randomized
vs.
Randomized
RCD Active
Learning from Winners
41 SHADES OF BLUE
„..a team at Google
couldn’t decide between two blues,
so they’re testing
41 shades between each blue
to see which one performs better...
I can’t operate in an environment like that.“
Douglas Bowman, Google’s previous visual design leader left Google
https://stopdesign.com/archive/2009/03/20/goodbye-google.html
Today,
Microsoft and several other leading companies
including Amazon, Booking.com, Facebook, and Google each conduct
more than 10000 online controlled experiments
annually…
Kohavi R, Thomke HS. Havard Buisness Review 2017
„The idea increased
Bing’s revenue by 12%(over $120M at the time)“
R. Kohavihttp://bit.ly/2017ABTestingTutorial
“Only one third of the ideas tested at Microsoft
improved the metric(s) they
were designed to improve”
Kohavi, R. et al. 2013 Online Controlled Experiments at Large Scale.
Retrieved from http://bit.ly/ExPScale
Kohavi et al. Trials 2020;21:150
URL: https://trialsjournal.biomedcentral.com/track/pdf/10.1186/s13063-020-4084-y
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made
Kohavi et al. Trials 2020;21:150
URL: https://trialsjournal.biomedcentral.com/track/pdf/10.1186/s13063-020-4084-y
CC-BY 4.0: http://creativecommons.org/licenses/by/4.0/ No changes were made
Learning Health Care System
RCD for RCT
RCD for RCTIdentification/recruitment Screen EHR-database for eligible
patients Inform Study design Power Calculation/ Feasibility
Send recruitment message
RCD for RCTExtended follow-up Long-term outcomes Expanding previous trials
(Registry Linkage)
RCD for RCTOutcome Assessment Length of hospital stay Adverse events Complications ICU stay Re-admission Ressource use / Costs
Many of the most patient-relevant outcomes!
RCD for RCTIntervention EHR-based diagnostic support
tools (e.g. AI, algorithms) Alert systems Prescription feedback
Learning healthcare system
Mc Cord et al. CMAJ Open 2019https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375253/pdf/cmajo.20180096.pdf
RCD for RCT – Kosten
Schnelle AntwortenGlaubwürdige Evidenz
Geringe Kosten
???
Späte AntwortenGlaubwürdige Evidenz
Hohe Kosten
Randomisierte Studien
The Magic of Randomization
WHY NOT ???MODELLIERUNG RANDOMISIERUNG
Datenmenge deutlich mehr
Datenqualität höhere Anforderung
Statistische Modelle komplexer
Annahmen / Verständnis Wirkprinzip komplexer
Risiko für Fehler/Bias deutlich höher
Einverständnis erforderlich
Kosten eher höher
Vorteil Modellierung Vorteil Randomisierung
IMMER: Registrierung, Studienprotokoll, Analyseplan, Ethikvotum, Follow-up, Publikationen…
“Obstacles to randomized trials
should be removed to protectpatients…
Promotion of nonrandomized analyses
… is a false solution to the problems
caused by the bureaucratic burdensimposed on … trials...” R Collins, L Bowman, M Landray, R Peto, NEJM 2020
“The Magic of Randomization versus the Myth of Real-World Evidence” NEJM 382;7 February 13, 2020
“Sometimes when you innovate, you make mistakes.
It is best to admit them quickly, and get on with improving your
other innovations“
Steve Jobs
JPA Ioannidis (U Stanford)
D Contopoulus-Ioannidis (U Stanford)
R Salman (U Edinburgh)
H Ewald (U Basel)
KA Mc Cord (U Basel)
M Briel (U Basel)
C Pauli-Magnus (U Basel)
HM Verkoijen (U Utrecht)
U Dirnagl (Charite / BIH / QUEST)
B Kasenda (iomedico/U Basel)
TV Pereira (U Toronto)
F Naudet (METRICS / INSERM)
O Fröbert (U Orebro)
B Speich (U Oxford)
R Kohavi (Microsoft/Airbnb)
Special Thanks
@LGHemkens
PD Dr. med. Lars G. Hemkens MPHDepartment of Clinical ResearchUniversity Hospital Basel Switzerland
QUEST (Quality | Ethics | Open Science | Translation) Visiting FellowBerlin Institute of Health (Charité and Max Delbrück Center) | Berlin | Germany
Weiterführende Informationen
Fröbert et al. (TASTE trial). Thrombus aspiration during ST-segment elevation myocardial infarction. N Engl J Med. 2013
www.ncbi.nlm.nih.gov/pubmed/23991656
Nationwide registry based RCT (Scandinavia); 0% Loss-to-follow-up; >7000 pts; costs about 350‘000 EUR
Hemkens et al. Personalized Prescription Feedback Using Routinely Collected Data to Reduce Antibiotic Use in Primary Care:
A Randomized Clinical Trial. JAMA Intern Med. 2017 www.ncbi.nlm.nih.gov/pubmed/28027333
Nationwide admin/insurance data RCT (Switzerland); 2900 GPs, >10‘000‘000 consultations; 2 yrs; costs about 300‘000 USD
The Magic of Randomization versus the Myth of Real-World Evidence. Collins R, Bowman L, Landray M, Peto R.
N Engl J Med. 2020 www.ncbi.nlm.nih.gov/pubmed/32053307
„Nonrandomized observational analyses have been promoted as alternatives to randomized clinical trials. However,
randomization ensures balance between groups, whereas nonrandomized studies are often biased by between-group
differences. Efforts to reduce the cost and complexity of clinical trials are preferable to relying on observational studies.“
Using electronic health records for clinical trials: Where do we stand and where can we go?
Mc Cord KA, Hemkens LG. CMAJ. 2019 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4868623/pdf/188e158.pdf
How Routinely Collected Data for Randomized Trials Provide Long-term Randomized Real-World Evidence.
Hemkens LG. JAMA Netw Open. 2018 Dec 7;1(8):e186014
https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2719124
Nonrandomized studies using causal-modeling may give different answers than RCTs: a meta-epidemiological study.
Ewald et al. J Clin Epidemiol 2020 https://www.ncbi.nlm.nih.gov/pubmed/31704350
Creating a Learning Health System through Rapid-Cycle, Randomized Testing Horwitz LI, Kuznetsova M, Jones SA.
N Engl J Med 2019; www.ncbi.nlm.nih.gov/pubmed/31532967 „Randomized tests of quality-improvement interventions were
embedded within an existing health care system. Quality and efficiency were improved by abandoning ineffective interventions
and continuing successful quality-improvement strategies.“
