Systematic reviews of animal studies

Malcolm Macleod

Why do systematic reviews of animal studies?

• To summarise existing data

• To help design clinical trials

• To understand where evidence is lacking

• To understand the limitations of animal models

Basic requirements

• Understanding the model – or knowing someone who does

• Clear a priori hypotheses

• A clear search strategy with inclusion and exclusion criteria

• Information and data management

The importance of hypotheses

• Observational research

• Susceptible to identification of statistically significant but biologically meaningless spurious associations

• If there isn’t a question for which you wish to know the answer, why are you doing it?

Examples

• “Hypothermia improves outcome in animal models of stroke”

• “The efficacy of hypothermia in animal models of stroke depends on the degree of cooling”

• “Evidence for the efficacy of NXY-059 is confounded by poor study quality”

• “The evidence for efficacy of stroke dugs in animals is confounded by publication bias”

Search strategy• What are you looking for …

– Controlled studies testing the effect of hypothermia in an animal model of focal cerebral ischaemia brought about by occlusion of a cerebral artery, where outcome was measured as infarct size or neurobehavioural score.

• Exclusion criteria– hypothermia was accomplished with use of a pharmacological agent

that may also have an intrinsic neuroprotective property; – cooling was used to counteract (spontaneous) hyperthermia after

MCAO; – brain cooling lasted <10 min, for example to counteract heating in

models of photochemically induced cerebral infarction; – data were presented in a way not suitable for use in a meta-analysis

(e.g. no information on group size, mean or variance); or if – mortality was the only outcome measure.

Example search strategyStudies of hypothermia in animal models of acute ischaemic stroke

were identified from …

(i) PubMed, EMBASE and BIOSIS up till December 31, 200> with the search strategy [[[<cerebral> OR <brain> OR <neuron> OR <neuronal> OR <nervous>] AND [<ischemia> OR <ischaemia>]] OR <stroke>] AND [<hypothermia> OR <temperature>] (limit: animals)

(ii) hand searching of abstracts of scientific meetings of the International Society of Cerebral Blood Flow and Metabolism, the International Stroke Conference [‘Joint (International) Conference on Stroke and Cerebral Circulation’ before 2000] and the European Stroke Conference during the same time period;

(iii) reference lists of identified publications; and (iv) requests to authors of identified publications

Source selection

• Download search results to reference management system (eg RefMan)

• 2 investigators independently select sources against inclusion/ exclusion criteria

• May have to retrieve full text

• Discrepancies resolved by negotiation or in discussion with third investigator

Data extraction

• Publication meta-data

Data extraction

• Publication meta-data

• Outcome data

Data cleaning

Data analysis

• Standardised mean difference analysis• Weighted mean difference analysis• Normalised mean difference analysis

• Fixed Effects• Random Effects

• Meta-regression

difference

s.d. e.s. = difference/s.d.“sd units”

SMD

0%unlesioned

100%lesioned

d1

e.s. = d1/100%“percent improvement in outcome”

NMD

n1 n2

e.s. = n1-n2“real” unitsWMD

0 mm3 250 mm3

Testing significance

• Partitioning of heterogeneity:

– Observed heterogeneity = within group heterogeneity + between group heterogeneity

– Test against chi squared distribution with n-1 degrees of freedom

• Observational, so set high statistical bar

Data analysis

Demonstration

So, what can it do …?

• Describe a literature ….

Randomisation

Blinded Outcome

Assessment

Sample Size calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

PD 12% 15% 0%

EAE 2% 11% 0%

Glioma 14% 0% 0%

Estimate efficacy …

• Hypothermia– 101 papers– 277 experiments– 3353 animals

Describe efficacy in subgroups…

Show potential sources of bias …

NXY 0599 publications

29 experiments408 animals

Improved outcome by 44% (35-53%)

Illustrate publication bias …

All

stud

ies

Nic

otin

amid

eT

hrom

boly

ticN

XY

-059

NO

S d

onor

Nos

Inhi

bito

rF

K50

6H

ypot

herm

iaM

elat

onin

Est

roge

nsT

irila

zad

IL1-

RA

Effi

cacy

0

20

30

40

50

60ObservedAdjusted

991 publications

Provide evidence to change practice …

1. Animals2. Sample size calculation 3. Inclusion and exclusion criteria4. Randomization5. Allocation concealment6. Reporting of animals excluded from analysis 7. Blinded assessment of outcome 8. Reporting potential conflicts of interest and study funding

Further resources

• http://www.camarades.info

• http://www.camarades.info/index_files/papers.htm

• malcolm.macleod@ed.ac.uk

• The CAMARADES podcast …– http://www.camarades.info/index_files/podcas

ts/podcast.xml