Reporting systematic reviews and meta-analyses (PRISMA) and observational studies (STROBE)

Reporting systematic reviews and meta-analyses (PRISMA) and

observational studies (STROBE)

Doug AltmanThe EQUATOR Network

Centre for Statistics in Medicine, Oxford, UKOctober 2012

2

Systematic review

A systematic review is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarise the findings of similar but separate studies.– A study of studies

Objective is to summarize evidence from multiple studies using explicit methods

It may include a quantitative synthesis (meta-analysis), depending on the available data

3

Key characteristics of SR

Focused well defined research question Clearly stated title and objectives Comprehensive strategy for identification of all

relevant studies (published & unpublished) Explicit (and justified) predefined inclusion &

exclusion criteria Critical appraisal of studies Clear analysis of the results of eligible studies

– Quantitative (meta-analysis)– Qualitative

Structured report

4

The QUOROM Statement[Moher et al 1999]

Guidance on what information should be included in published reports of meta-analyses of randomized trials

Checklist of items which should be reported Also recommended a flow diagram showing

flow of studies through the review – to be included in the published report

Evidence-based, whenever possible QUOROM developed in 1996 and published in

1999 Moher et al. Improving the quality of reporting of meta-

analyses of randomised controlled trials: the QUOROM statement. Lancet 1999;354:1896-1900

5

Checklist of items

6

Also recommended a flow diagram showing flow of studies through the

review

7

PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-

Analyses Update of QUOROM

Developed in 2005, Published in 2007

Consists of a 27 item checklist and a flow diagram

Reporting of systematic reviews and meta-analyses that evaluate healthcare interventions

Includes long explanatory document

Slide 8 of 43

9

# of records identified through database searching

# of additional records identified through other

sources

Total # of duplicates removed

# of records screened # of records excluded

Total # of articles assessed for eligibility

Total # of studies included in qualitative synthesis of

systematic review

# of articles excluded, with

reasons

# of studies included in quantitative synthesis of

systematic review

iden

tificatio

nscreen

ing

eligibility

includ

ed

10

Reporting vs conduct: study methods

METHODS – each aspect of the methods

Done well

Done poorly

Not done

Fully reported (=reproducible)

Ambiguously or incompletely reported

Not reported

11

PRISMA Practical

12

2 reviews in one article focus on progesterone

13

For each item …

Is there text relating to the item?

Does the text tell us what we need to know?

14

PRISMA: Item 6, eligibility criteria

METHODS Eligibility criteria Specify study characteristics used as criteria

for eligibility, giving rationale– PICOS

(participants, interventions, comparisons, outcomes and study design)

– Length of follow-up Specify report characteristics used as criteria

for eligibility, giving rationale– Years considered – Languages – Publication status

Can you locate any text about this issue in the report?

15

Wyatt et al

No explicit statement of study characteristics

“… clinical trials of progesterone and pro gestogens in the management of premenstrual syndrome.”

Dates given in search criteria

“All languages were included.”

PRISMA – item 8, search

“Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated”


Slide 16 of 43

Wyatt et alPage 2, Methods, Trials

“MeSH terms used were premenstrual syndrome, progesterone, and progestogen, as well as the individual drug names, together with title and abstract searches for keywords progesterone, pro gestogen, premenstrual syndrome, premenstrual ten sion (PMT), late luteal phase dysphoric disorder (LLPDD), premenstrual dysphoria (PMD), and pre menstrual dysphoric disorder (PMDD).”

Slide 17 of 43

PRISMA: Reporting search strategy

We realize that journal restrictions vary and that having the search strategy in the text of the report is not always feasible – Expensive real estate

We strongly encourage all journals, however, to find ways, such as a ‘‘Web extra,’’ appendix, or electronic link to an archive, to make search strategies accessible to readers

We also advise all authors to archive their searches so that:– others may access and review them (e.g.,

replicate them or understand why their review of a similar topic did not identify the same reports)

– future updates of their review are facilitated

19

PRISMA: Item 9, study selection

METHODS Study selection State the process for selecting studies

– Screening – Eligibility – Included in systematic review and, if applicable,

included in meta-analysis


20

Wyatt et al

“We searched medical databases for reports of published clinical trials of progesterone and progestogens in the management of premenstrual syndrome.”

“References cited in all trials were searched iteratively to identify missing studies. All languages were included.”

“We included trials that investigated the effect of progesterone or progestogens on premenstrual symptoms if they were randomised, placebo controlled, double blind studies that included patients with a pretreatment diagnosis of premenstrual syndrome, for which all data from the trials could be acquired.”

21

PRISMA Item 11, Data items

METHODS Data items 11 List and define all variables for which data

were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. (PICOS = participants, interventions, comparisons, outcomes and study design)


22

Wyatt et al

“We collected data on the dosage and preparation of treatment. The main outcome measure was a reduction in overall symptoms of premenstrual syndrome. Combined or overall symptoms was chosen in an attempt to overcome the clinical heterogeneity associated with the measurement and scoring of symptoms used in individual trials.”

PRISMA – item12, Risk of bias in individual studies

“Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis”– Can you locate any text about this issue in the report?

Slide 23 of 43

Wyatt et alPage 2, Methods, Quality assessment

“We assessed trial quality using a scale developed by Jadad et al,11 which assesses the randomisation, double blinding, reports of drop outs, and withdrawals for the trials

... our own quality scale, which assesses the quality of the trials for study design, reproducibility, and statistical analysis. This eight point scale comprised the following: confirmation that no other medications or oral contraceptives were being taken; a power calculation to justify patient numbers or more than 65 participants in each arm (enabling detection of a small effect size of 0.3, see below); a single, clearly stated dose of drug; reproducible measurement of premenstrual symptoms; clear presentation of results; a description of the number and reason for trial withdrawals; exclusion of, or a separate analysis of, participants with a major psychiatric disorder; and whether or not the trial was supported by independent funding.”

Slide 24 of 43

Reporting risk of bias

“Authors should report how they assessed risk of bias; whether it was in a blind manner; and if assessments were completed by more than one person, and if so, whether they were completed independently. Similarly, we encourage authors to report any calibration exercises among review team members that were done. Finally, authors need to report how their assessments of risk of bias are used subsequently in the data synthesis (see Item 16).”

Slide 25 of 43

Wyatt et alPage 2, Methods, Quality assessment

“We awarded one point for each category present in the trial.

Each trial was independently scored by two investigators and the third investigator arbitrated on any disagreements.

We used predetermined criteria for the recognition of the highest quality trials. A score of 3 or more was required in the Jadad score for the trial to be designated “high quality” and included in the meta analysis11; a score of less than 3 meant that the trial was designated “low quality.”

Slide 26 of 43

27

PRISMA Item 17

RESULTS Trial flow17 Give numbers of studies screened, assessed

for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.


Page 2, Results

“We identified 14 published trials that assessed the efficacy of progesterone in the management of premenstrual syndrome. We excluded four: two because of their low quality score on the Jadad scale, one because the data could not be extracted, and one because the trial failed to make a prospective diagnosis of premenstrual syndrome before randomisation. Ten trials remained ........”

No flow diagram

Slide 28 of 43

Caughey AB, Sundaram V, Kaimal AJ, Gienger A, Cheng YW, McDonald KM, Shaffer BL, Owens DK, Bravata DM. Systematic Review: Elective Induction of Labor Versus Expectant Management of Pregnancy. Annals of Internal Medicine 2009;151: 252-263

Slide 29 of 43

30

PRISMA Item 18

RESULTSStudy characteristics18 For each study, present characteristics for

which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.


31

Wyatt et al

32

Wyatt et al Problems in Table 1

Crossover trials – not mentioned in text Comparator not stated Variation in outcome measures – how

combined or chose one? Some statements disagree with Figure No details of elements of “quality” scores Unclear how handled trial with 3 arms

PRISMA – item 15, Risk of bias across studies

Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).


Slide 33 of 43

Page 2, methods, statistical analysis

“We used the method of Egger et al to detect bias (such as publication and location bias) in the included trials with a funnel plot. We assessed the asymmetry of the funnel plot quantitatively by plotting a linear regression of the standard normal deviate (standardised mean difference divided by SE) against precision (inverse of SE). A regression line that passes through the origin of the plot (within error limits) indicates symmetry and hence the absence of bias.”

Slide 34 of 43

35

PRISMA Item 20

RESULTS Results of individual studies20 For all outcomes considered (benefits or

harms) present, for each study: (a) simple summary data for each intervention group, (b) effect estimates and confidence intervals, ideally with a forest plot.

36

37

Altman & Cates complained about absence of any numerical results (BMJ

rapid response) “There are several aspects of this review that

readers cannot assess without summary data from each study.

For example, we would wish – to assess the strange heterogeneity P values for Figures 1

and 2 (the quoted P values of 0.999 are implausible given the clear graphical heterogeneity);

– to gain some insight into how the cross-over trials were included in the analysis (about which the authors say nothing at all) and whether the crossover and parallel group trials differed in their findings;

– to seek an explanation for the apparent discrepancy for three trials (references 19, 31 and 32) between the ‘reported results’ in Table 1 and the results shown in Figures 1 and 2;

– to assess the claim that random and fixed models give the same answer in the face of graphical heterogeneity;”

– … and 5 more points

38

Altman & Cates complained about the absence of any

numerical results “In addition, we note that the authors make no

comment about the varied nature of the outcome measures in these trials, nor do they say which outcome was used for those trials that presented more than one. It is hard to believe that all of the scales can be considered equally valid assessments of symptoms.

Also, we wonder if they can clarify the meaning in the figure legends of ‘standardised mean difference … for proportion of patients who showed improvement …’. We are puzzled by this terminology as the SMD gives no direct information about proportions of patients improving.”

39

Authors’ reply

“We have found this personal attack unpleasant and upsetting and have to question the use of unsupported attacks in the Rapid Response forum.”

“We have considerable experience in PMS (clinically as well as though our research) and believe ourselves competent to judge the clinical appropriateness of combining trial data.”

The only question they addressed was one we had not asked!

A 2nd request for the data to be provided went unanswered!

40

41 41

Observational studies

Transparent reporting is particularly important for observational studies– Vulnerable to bias and confounding– Findings are often over-interpreted– Findings often generate health scares

42

43

Scope of STROBE

Epidemiological research comprises several study designs and multiple topic areas

Initial restriction to three major areas– cohort, case-control, and cross-sectional studies

Later extensions to other study designs – STREGA for genetic association studies (published

2009)– STROBE-ME for molecular epidemiology – etc

44 44

Final STROBE checklist

TITLE and ABSTRACT INTRODUCTION Background/rationale Objectives METHODS Study design Setting Participants Variables Data sources/measurement Bias Study size Quantitative variables Statistical methods

RESULTS Participants Descriptive data Outcome data Main results Other analyses DISCUSSION Key results Limitations Interpretation Generalisability OTHER INFORMATION Funding

22 (34) items

45 45

Design-specific items

Participants Statistical methods Descriptive data Outcome data

46

STROBE Statement

Guidance on how to report observational studies well– Focus on 3 main study designs: cohort, case-control, cross-

sectional studies Published in Oct 2007: short paper and E&E Adopted by many journals

Find it on:www.equator-network.orgwww.strobe-statement.org

Case-control studies

Patients with a certain outcome or disease and an appropriate group of controls without the outcome or disease are selected – (usually with careful consideration of choice of controls,

possibly with matching)

Information is obtained on whether the participants have been exposed to the factor under investigation

47

48

STROBE exercise

49

50

51

STROBE Item 5. Setting

Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection


52

STROBE Item 5. Setting

Qiu et al

“This case-control study was conducted at the Materno Perinatal Institute of Lima and the Dos de Mayo Hospital in Lima, Peru, from May 2004 through October 2005. Both institutions are operated by the Peruvian government and are primarily responsible for providing maternity services to low income women residing in Lima.”

53

STROBE Item 6a. Participants

Give the eligibility criteria, and the sources and methods of case ascertainment and control selection. Give the rationale for the choice of cases and controls


54

STROBE Item 6a. Participants

Qiu et al“Cases were selected from those women with a

diagnosis of preeclampsia. Potential preeclampsia cases were identified by daily monitoring of all new admissions to antepartum wards, emergency room wards, and labor and delivery wards of the study hospitals. Study subjects were recruited during their hospital stay. Study personnel made periodic visits to each ward in a fixed order for the purposes of identifying potential cases and controls for the present study. Preeclampsia was defined by …”

“Controls were women with pregnancies uncomplicated by pregnancy-induced hypertension or proteinuria. Each day during the enrollment period, controls were numbered in the order in which they were admitted and identified. Subsequently, they were approached in the order in which research personnel identified them.”

55

STROBE Item 10. Study size

Explain how the study size was arrived at


56

STROBE Item 10. Study size

Qiu et al

?

57

STROBE Item 13. Participants

(a) Report numbers of individuals at each stage of study—e.g. numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed

(b) Give reasons for non-participation at each stage

(c) Consider use of a flow diagram

58 58

What we’d like to see:

What do they give

us?

59

Qiu et al

60

STROBE Item 19. Limitations

Discuss limitations of the study, taking into account sources of potential bias or imprecision.

Discuss both direction and magnitude of any potential bias


61

Qiu et al

“First, our analyses are based on cross-sectionally collected data, which may be subject to recall bias. There has been one longitudinal study of Finnish women [6]; however, more longitudinal studies are needed to re-examine the potential causal relation between maternal experience of depression and preeclampsia risk in different populations.Second, we used a depression screening instrument to categorize participants according to depression/depressive symptoms. Participants did not have formal diagnostic examinations. As a result, some misclassification is possible. ... In addition, our assessment of maternal depression and depressive symptoms was limited to the duration of the pregnancy. Last, although we adjusted for multiple confounding factors, as with all observational studies, we cannot exclude the possibility of some residual confounding.”

62

63

Closing Comments on Checklists

They help AUTHORS ensure that they have addressed important issues in the report of their study

They help PEER REVIEWERS and EDITORS by reminding them what issues should be addressed

“Necessary but not sufficient!”

64

Closing Comments on Checklists

They help AUTHORS ensure that they have addressed important issues in the report of their study

They help PEER REVIEWERS and EDITORS by reminding them what issues should be addressed

“Necessary but not sufficient!”

Documents

Reporting systematic reviews and meta-analyses (PRISMA) and observational studies (STROBE)