Clinical Trials &

Circular Reasoning and Anti-intellectual Pomposity Detectors

(CRAP Detectors)

D Massel, MD FRCPC

Professor Emeritus

Western University

Conflict of Interest

None

CRAP #1: Disclosure

… is generally considered preferable to nondisclosure, because it makes explicit and transparent details that are important to the interpretation, credibility, and value of the information.

BUT, disclosure has limits as a strategy for mitigating bias.

1) Disclosure alone merely shifts “secret bias” to “open bias”; it sometimes involves so much information about ties to the industry, for example, that the reader is blinded by the sheer “signal to noise ratio”;

2) Disclosure may be perceived as absolving a person from their responsibility for managing their conflict.

3) Evidence suggests that disclosure to be not only ineffective but also regressive. Disclosure can actually lead doctors to give biased advice, either through strategic exaggeration (whereby more biased advice is provided to counteract anticipated discounting), or “moral licensing” such that advice is legitimized because advisees “have been warned” (that is, caveat emptor or “buyer beware”)

Experiments have shown that bias is considerably greater when conflicts of interest are disclosed. Worse, because it it has been shown that advisees (i.e., patients) both think that their advisers (i.e., doctors) would never intentionally mislead them and tend not to discount advice in light of conflicts, disclosure policies will never be the solution and are very likely exacerbating the problem of bias in medicine.

Studies reporting the results of industry sponsored clinical trials present a more favourable picture of the effects of drugs and medical devices than those reporting on non-industry sponsored trials.

Results and conclusions sections in these papers were less likely to agree.

Studies sponsored by industry reported greater benefits and fewer harmful side effects.

Can influence guidelines and formal recommendations.

CRAP #2: Industry Ties

• ‘Positive’ Trials

• ‘Negative’ Trials

• Non-inferiority Trials

• How to be a good consumer

Agenda

P < 0.05, could it be….

1. The truth

2. Chance

3. Bias – look for ‘fatal flaws’

Positive Trials

CRAP #3 : Is there a control group?

“A study without controls is better than nothing, but not by much”

Archie Cochrane

Positive Trials

CRAP #4 : Was there true randomization?

Not only randomization but concealment of allocation, that is it should not be

possible to guess what the next patient might get.

Did the randomization ‘work’?

Positive Trials

CRAP #4 : Was there true randomization?

• FHRT

• Positive inotropes for CHF

• Vitamins

• Lidocaine for PVCs in the setting of MI

•EC IC bypass for stroke prevention

UnPositive Trials

“A difference is a difference only if it makes a difference”

CRAP #5

Was the difference important?

Were all important endpoints (good and bad) included?

Were the events adjudicated blindly?

Positive Trials

CRAP #5

Just because a result is significant it does not mean it is important.

If it takes a trial of 20,000 patients to show a small difference, is it worth it

to the individual?

Positive Trials

Clinical vs Statistical Importance

CRAP #6: Composite endpoints, proceed with caution

They mislead if component end points are :

1) of widely differing importance to patients 2) the number of events in the components of greater importance is small, and 3) the magnitude of effect differs markedly across components

Most often the least important part is where the difference lies, but bundled giving the impression that the more serious events are also reduced.

Watch double counting – esp. as dead patients cannot have non-fatal events.

Positive Trials

CRAP #6: Composite endpoints, proceed with caution

Definitions of MACE include end points that reflect both the safety (death, MI, stroke) and effectiveness(target vessel revascularization [TVR], restenosis, recurrent ischemia, rehospitalization)

1) each component should be of comparable clinical importance 2) each component should occur with similar frequency 3) Each component should be similarly sensitive to treatment intervention

Positive Trials

The Stent PAMI (Stent–Primary Angioplasty for Myocardial Infarction)was designed to assess whether primary stenting was superior to primary balloon angioplasty in patients with acute STEMI.

The combined primary end point of death, re-MI, disabling stroke, or ischemia driven TVR at 1 year occurred in fewer patients in the stent than in the angioplasty group (12.6% vs. 20.1%, p�0.005) ….

Differences:

• TVR (p = 0.0005) • ReMI (p = 0.7) • Stroke (p = 0.83) • Death (p = 0.07) -negatively impacted.

Positive Trials

One could state that the composite was significantly lower among patients randomized to:

Treatment A vs B (21.9% vs. 13.8%, p �0.001) death [inc-ns], disablingstroke [inc-ns], re-MI [nd-ns], TVR [dec-sig]).

Treatment A vs B (21.9% vs. 13.8%, p 0.001) death [inc-ns], disabling

stroke [inc-ns], re-MI [nd-ns], TVR [dec-sig]).

Positive Trials – Composite Confusion

CRAP #7 : Subgroup Analysis – Interesting and Misleading

High chance of chance

Pre-planned

Formal test of interaction

Be especially wary of if the primary hypothesis is not significant

Positive Trials

ISIS-2

# deathsPLACEBO

# deathsASPIRIN 2p

N=8587 N=8600

Gemini or Libra 147 150 0.5

All others 869 654 <0.0000001

All participants 1016 804 <0.0000001

1. PRAISE Trial of amlodine in CHF. BUT, patients with heart failure due to non-ischemic cardiomyopathy had nearly a 50% reduction in the risk of death or cardiac hospital admission.

• PRAISE – 2 done in that subgroup – NS

Subgroups

• ELITE Trial -to investigate the safety and tolerability of losartan with captopril in CHF. Serum Cr was the primary endpoint. BUT, a surprising mortality reduction with losartan.

•• ELITE – 2 showed no difference, • indeed losartan looked worse

Subgroups

CRAP #7 : Subgroup Analysis

“The only thing worse than doing a subgroup analysis is believing the results”

Be especially wary if more noise is made out of the subgroup rather than the overall trial result, particularly if surprising.

Positive Trials

Many ways this might be considered

Was not interesting enough

Intervention is inferior or harmful

It is just failed to be ‘significant’

– similar, the same, or not different, or are they equivalent

Negative Trials

Trial methodology is critical.

Possibly simply a lack of events (too small).

Can’t ‘accept’ the null hypothesis

Forget the p-value and look at the CIs…Could an important clinical difference have been missed.

Negative Trials

Crap #8

Were there factors (loss to follow-up, poor compliance, co-intervention,

contamination or bias) that may have pushed the results closer together?

Negative Trials

Crap #9

Could an important difference have been missed?

Look at the 95% CI.

Negative Trials

Negative Trials

Can have a Positive Impact

Abandon useless or harmful therapies.

Embrace non-harmful, but helpful treatments.

Negative Trials

Can have a Positive Impact

Digoxin in CHF – no mortality benefit but did reduce rehospitalization

PRAISE-2 - amlodipine in CHF – no increase in mortality

AIM – HIGH – niacin in low-LDL patients – no benefit

Negative Trials

… is designed to show that a new treatment is

‘not unacceptably worse’ than the current standard

therapy…

Non-Inferiority Trials

Choice of the Non-Inferiority Margin

Sample Size

Bio-Creep

Switching From Superiority to Non-Inferiority

Non-inferiority Trials

Non-inferiority Trials

1. In 1492 received funding from Ferdinand II and Is abela from Spain.

2. His goal was to find a shorter route to the Indie s

3. He postulated that the world was a small sphere a nd that he would be able to reach his goal by going west over the Atlantic Ocean

4. 1492/8/3 departs from Palos, Spain

5. 1492/10/12 the ‘New World’ sighted at 2:00 a.m. b y Rodrigo de Triana

6. The widely published report of his voyage of 1492 made Columbus famous throughout Europe and secured for him the title of ‘Admiral of the Ocean Sea’ and further royal patronage.

Christopher Columbus &North America

Christopher Columbus &North America

• In 1484 he was unable to convince King John II of P ortugal of his venture; tried England and France and finally in 1492 received ‘al ternate’ funding from Ferdinand II and Isabela from Spain (bad science)

• His goal was to find a shorter route to the Indies (or to convert the Chinese to Christianity (? 2 primary outcome events)

• The spherical world he postulated was far too smal l (wrong ‘statistical model’)

• He did not know where he was when he got there (a surprising non pre-specified outcome!)

• His finding of a ‘new world’ superceded an obvious f ailed enterprise (classic post-hoc rationalization of a ‘negative’ s tudy)

Weak Statistical Evidence

DID YOU KNOW THAT……

if you read 2 articles per day out of the

6,000,000 medical articles published annually,

in 1 year you would fall 82 centuries behind in your

reading.

Miser WF, 1999

Choose important journals to you and get the TOC sent along.

Be wary, RUTHLESS and guilt-free.

• Read the title – interesting? If not, move on to the next.

• Read the abstract – useful? If not, move on.

• Read the methods – quality? If not, move on.

• Read the study.

• Reviews articles and Guidelines only if pertinent.

Move on to the next journal.

How to consume the literature

without being consumed

Listen to Podcasts

Medscape, Dynamed, BCMA, Skyscape, Centre for EBM

Evidence-based Medicine

ACP Journal Club

How to consume the literature

without being consumed

Be selective

Be efficient

Be ruthless

Make use of technology

Make use of summaries esp. if peer-reviewed

Have a healthy degree of skepticism but be pragmatic

Final thoughts