CAMARADES: Bringing evidence to translational medicine
Study Quality and Publication Bias in Experimental Studies of Neurological Diseases
Emily S Sena, Hanna M Vesterinen, Kieren J Egan and Malcolm R Macleod Centre for Clinical Brain Sciences, University of Edinburgh
CAMARADES: Bringing evidence to translational medicine
CAMARADES
• Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies
• Look systematically across the modelling of a range of conditions
• Data Repository– 7 Neurological Diseases– 2500 studies – 4700 in vivo experiments – from over 60,000 animals
CAMARADES: Bringing evidence to translational medicine
Hypotheses:
• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity
• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power
• These factors combine to compromise the utility of animal models and contribute to translational failure
CAMARADES: Bringing evidence to translational medicine
Outline
• Translational failure
• Internal validity of experiments
– Reporting of measures to minimise bias
• External validity of experiments
• Publication bias
CAMARADES: Bringing evidence to translational medicine
Outline
• Translational failure
• Internal validity of experiments
– Reporting of measures to minimise bias
• External validity of experiments
• Publication bias
CAMARADES: Bringing evidence to translational medicine
1026
1026 interventions in experimental stroke
Tested in experimentsO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026603
1026 interventions in experimental stroke
Tested in focal ischaemia O’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883374
1026 interventions in experimental stroke
Effective in focal ischaemia O’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883550
97 18
1026 interventions in experimental stroke
Tested in clinical trial O’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883550
97 171 3
1026 interventions in experimental stroke
Effective in clinical trial O’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
Animal data in stroke
• There are huge amounts of often confusing data
• Systematic review can help to make sense of it
• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit
• Investigating the sources behind this variation may be helpful in translation
Hypothermia: a systematic search identified 222 experiments in 3353
animals
Bett
er
Wors
e
Van der Worp et al Brain 2007
CAMARADES: Bringing evidence to translational medicine
Outline
• Translational failure
• Internal validity of experiments
– Reporting of measures to minimise bias
• External validity of experiments
• Publication bias
CAMARADES: Bringing evidence to translational medicine
Potential sources of bias in animal studies
• Internal validity
• External validity– Are the models we use good models? – Publication bias
Problem Solution
Selection Bias Randomisation
Performance Bias Allocation Concealment
Detection Bias Blinded outcome assessment
Attrition bias Reporting drop-outs/ ITT analysis
CAMARADES: Bringing evidence to translational medicine
Internal Validity NXY-059 in Stroke
Infarct Volume– 11 publications– 408 animals– Improved outcome by 44% (35-53%)
Macleod et al, 2008
CAMARADES: Bringing evidence to translational medicine
Internal validity in PD models
Blinded outcome assessment
Rooke et al, 2011
• 253 Publications
• 601 Experiments
• 4181 Animals
CAMARADES: Bringing evidence to translational medicine
1117 publications
388 report neurobehavioural outcome for 38 interventions tested >5
127 included in meta-analysis for 36 interventions
Study quality
Experimental Autoimmune Encephamyelitis (EAE)
(Vesterinen et al, 2010)
255 missing essential data
CAMARADES: Bringing evidence to translational medicine
EAE : Sample Size
CAMARADES: Bringing evidence to translational medicine
Chances that data from any given animal will be non-contributory
Number of animals Power % animals wasted
4 18.6% 81.4%
8 32.3% 67.7%
16 56.4% 43.6%
32 85.1% 14.9%
assume simple two group experiment seeking 30% reduction in infarct volume, observed SD
40% of control infarct volume
CAMARADES: Bringing evidence to translational medicine
How do models of neurological disorders compare?
RandomisationBlinded Outcome
AssessmentSample Size calculation
Stroke 36% 29% 3%
MND 31% 20% <1%
AD 15% 25% 0%
PD 12% 15% 0%
EAE 8% 15% <1%
Glioma 14% 0% 0%
Sena et al TiNS 2007
CAMARADES: Bringing evidence to translational medicine
Reporting of measures to avoid bias
• Sample size calculation 2/311 (1%)
• Randomisation 46/292 (16%)
• Blinding 46/312 (15%)
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
Outline
• Translational failure
• Internal validity of experiments
– Reporting of measures to minimise bias
• External validity of experiments
• Publication bias
CAMARADES: Bringing evidence to translational medicine
External ValidityHypertension in studies of tPA in experimental stroke
Comorbidity
“Normal” BP
Effi
cacy
-2%25%
Sena et al JCBFM 2010
• High prevalence of hypertension in patients with stroke
• tPA is substantially less effective in hypertensive animals
CAMARADES: Bringing evidence to translational medicine
External ValiditySimulation of the asymptomatic phase of AD
CAMARADES: Bringing evidence to translational medicine
External Validity:Time to Treatment in EAE
• Median: 0 days (IQR -11 to 4)• 1% did not report time of administration
Before EAE 48%
Day of Induction 22%
After EAE 30%
Day of Symptom Onset 2%
CAMARADES: Bringing evidence to translational medicine
Outline
• Translational failure
• Internal validity of experiments
– Reporting of measures to minimise bias
• External validity of experiments
• Publication bias
CAMARADES: Bringing evidence to translational medicine
Publication Bias
• Neutral and negative studies– remain unpublished
– less likely to be identified in systematic review
– leads to the overstatement of efficacy in meta-analysis.
CAMARADES: Bringing evidence to translational medicine
Effect Size
-150 -100 -50 0 50 100 150
Pre
cisi
on
0.0
0.1
0.2
0.3
0.4
0.5
Effect Size/Standard Error
-10 -5 0 5 10 15 20 25
Pre
cisi
on
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
CAMARADES: Bringing evidence to translational medicine
Publication bias in experimental stroke
• Trim and Fill suggested 16% of experiments
remain unpublished
• Best estimate of magnitude of problem
– Overstatement of efficacy 31%
• Only 2% publications reported no significant
treatment effects
CAMARADES: Bringing evidence to translational medicine
Hypotheses:
• In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity
• In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power
• These factors combine to compromise the utility of animal models and contribute to translational failure
CAMARADES: Bringing evidence to translational medicine
What happens …
• Small (underpowered), poorly conducted (randomisation, blinding) studies reach spurious (falsely positive) conclusions but are published because they are seen to be “interesting”.
• Small (perhaps) poorly conducted (sometimes) studies not reaching the same conclusions are not published.
• Investigators become conditioned by the apparent success that comes from conducting small underpowered studies
• Investigators keep trying to replicate the “positive” studies
CAMARADES: Bringing evidence to translational medicine
What should we do?
• Be rigorous in demanding the highest quality standards in the conduct and reporting of studies
• Develop model specific GLP guidelines • Develop a registry of animal studies to prevent
unnecessary replication• Where effect sizes are small (preclinical testing),
develop tools for multicentre animal studies
CAMARADES: Bringing evidence to translational medicine
Thanks to …
• Edinburgh– Peter Sandercock
• Utrecht– Bart van der Worp
• Melbourne– David Howells– Geoff Donnan
• Nottingham– Philip Bath