CAMARADES: Bringing evidence to translational medicine Study Quality and Publication Bias in Experimental Studies of Neurological Diseases Emily S Sena,

CAMARADES: Bringing evidence to translational medicine

Study Quality and Publication Bias in Experimental Studies of Neurological Diseases

Emily S Sena, Hanna M Vesterinen, Kieren J Egan and Malcolm R Macleod Centre for Clinical Brain Sciences, University of Edinburgh


CAMARADES

• Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies

• Look systematically across the modelling of a range of conditions

• Data Repository– 7 Neurological Diseases– 2500 studies – 4700 in vivo experiments – from over 60,000 animals


Hypotheses:

• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity

• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power

• These factors combine to compromise the utility of animal models and contribute to translational failure


Outline

• Translational failure

• Internal validity of experiments

– Reporting of measures to minimise bias

• External validity of experiments

• Publication bias


Outline







1026

1026 interventions in experimental stroke

Tested in experimentsO’Collins et al, 2006


1026603


Tested in focal ischaemia O’Collins et al, 2006


1026883374


Effective in focal ischaemia O’Collins et al, 2006


1026883550

97 18


Tested in clinical trial O’Collins et al, 2006


1026883550

97 171 3


Effective in clinical trial O’Collins et al, 2006


Animal data in stroke

• There are huge amounts of often confusing data

• Systematic review can help to make sense of it

• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit

• Investigating the sources behind this variation may be helpful in translation

Hypothermia: a systematic search identified 222 experiments in 3353

animals

Bett

er

Wors

e

Van der Worp et al Brain 2007


Outline







Potential sources of bias in animal studies

• Internal validity

• External validity– Are the models we use good models? – Publication bias

Problem Solution

Selection Bias Randomisation

Performance Bias Allocation Concealment

Detection Bias Blinded outcome assessment

Attrition bias Reporting drop-outs/ ITT analysis


Internal Validity NXY-059 in Stroke

Infarct Volume– 11 publications– 408 animals– Improved outcome by 44% (35-53%)

Macleod et al, 2008


Internal validity in PD models

Blinded outcome assessment

Rooke et al, 2011

• 253 Publications

• 601 Experiments

• 4181 Animals


1117 publications

388 report neurobehavioural outcome for 38 interventions tested >5

127 included in meta-analysis for 36 interventions

Study quality

Experimental Autoimmune Encephamyelitis (EAE)

(Vesterinen et al, 2010)

255 missing essential data


EAE : Sample Size


Chances that data from any given animal will be non-contributory

Number of animals Power % animals wasted

4 18.6% 81.4%

8 32.3% 67.7%

16 56.4% 43.6%

32 85.1% 14.9%

assume simple two group experiment seeking 30% reduction in infarct volume, observed SD

40% of control infarct volume


How do models of neurological disorders compare?

RandomisationBlinded Outcome

AssessmentSample Size calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

AD 15% 25% 0%

PD 12% 15% 0%

EAE 8% 15% <1%

Glioma 14% 0% 0%

Sena et al TiNS 2007


Reporting of measures to avoid bias

• Sample size calculation 2/311 (1%)

• Randomisation 46/292 (16%)

• Blinding 46/312 (15%)




Outline







External ValidityHypertension in studies of tPA in experimental stroke

Comorbidity

“Normal” BP

Effi

cacy

-2%25%

Sena et al JCBFM 2010

• High prevalence of hypertension in patients with stroke

• tPA is substantially less effective in hypertensive animals


External ValiditySimulation of the asymptomatic phase of AD


External Validity:Time to Treatment in EAE

• Median: 0 days (IQR -11 to 4)• 1% did not report time of administration

Before EAE 48%

Day of Induction 22%

After EAE 30%

Day of Symptom Onset 2%


Outline







Publication Bias

• Neutral and negative studies– remain unpublished

– less likely to be identified in systematic review

– leads to the overstatement of efficacy in meta-analysis.


Effect Size

-150 -100 -50 0 50 100 150

Pre

cisi

on

0.0

0.1

0.2

0.3

0.4

0.5

Effect Size/Standard Error

-10 -5 0 5 10 15 20 25

Pre

cisi

on

-0.1

0.0

0.1

0.2

0.3

0.4

0.5


Publication bias in experimental stroke

• Trim and Fill suggested 16% of experiments

remain unpublished

• Best estimate of magnitude of problem

– Overstatement of efficacy 31%

• Only 2% publications reported no significant

treatment effects


Hypotheses:

• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity

• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power

• These factors combine to compromise the utility of animal models and contribute to translational failure


What happens …

• Small (underpowered), poorly conducted (randomisation, blinding) studies reach spurious (falsely positive) conclusions but are published because they are seen to be “interesting”.

• Small (perhaps) poorly conducted (sometimes) studies not reaching the same conclusions are not published.

• Investigators become conditioned by the apparent success that comes from conducting small underpowered studies

• Investigators keep trying to replicate the “positive” studies


What should we do?

• Be rigorous in demanding the highest quality standards in the conduct and reporting of studies

• Develop model specific GLP guidelines • Develop a registry of animal studies to prevent

unnecessary replication• Where effect sizes are small (preclinical testing),

develop tools for multicentre animal studies


Thanks to …

• Edinburgh– Peter Sandercock

• Utrecht– Bart van der Worp

• Melbourne– David Howells– Geoff Donnan

• Nottingham– Philip Bath

Documents

CAMARADES: Bringing evidence to translational medicine Study Quality and Publication Bias in Experimental Studies of Neurological Diseases Emily S Sena,