Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones

$Page 1: Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones$
DOI:10.1111/j.1471-1842.2011.00972.x

Sensitivity and precision of adverse effects searchfilters in MEDLINE and EMBASE: a case study offractures with thiazolidinedionesSu Golder* & Yoon K. Loke†*Centre for Reviews and Dissemination (CRD), University of York, York, UK and †University of East Anglia, Norwich, UK

Abstract

Background: Search filters have been developed in MEDLINE and EMBASE to help overcome the chal-lenges of searching electronic databases for information on adverse effects. However, little evaluation oftheir effectiveness has been carried out.Objectives: To measure the sensitivity and precision of available adverse effects search filters in MED-LINE and EMBASE.Methods: A case study systematic review of fracture related adverse effects associated with the use ofthiazolidinediones was used. Twelve MEDLINE search strategies and three EMBASE search strategieswere tested.Results: Nineteen relevant references from MEDLINE and 24 from EMBASE were included in thereview. Four search filters in MEDLINE achieved high sensitivity (95 or 100%) with an improved levelof precision from searches without any adverse effects filter. High precision in MEDLINE could also beachieved (up to 53%) using search filters that rely on Medical Subject Headings. No search filter in EM-BASE achieved high precision (all were under 5%) and the highest sensitivity in EMBASE was 83%.Conclusions: Adverse effects search filters appear to be effective in MEDLINE for achieving either highsensitivity or high precision. Search filters in EMBASE, however, do not appear as effective, particularlyin improving precision.

Keywords: bibliographic databases, database searching, literature searching, medical subject headings(MeSH), MEDLINE, methodological filters, search strategies, searching.

Key Messages

Implications for Practice

d Adverse effects search filters may be applied in MEDLINE with an increase in precision withoutmajor loss of sensitivity.

d High precision of search strategies may be obtained in MEDLINE through use of search filterswhich rely solely on MeSH terms.

d Adverse effects search filters should be applied with caution in EMBASE as there may be too highloss of sensitivity without much improvement in precision.

Implications for Policy

d Further research on the yield of adverse effects search filters in multiple systematic review casestudies is required.

d Research is needed into the development of adverse effects search filters in EMBASE with highprecision.

Correspondence: Su Golder, Centre for Reviews and Dissemination (CRD), University of York, York, YO10 5DD, UK.

E-mail: [email protected]

ª 2011 The authors. Health Information and Libraries Journal ª 2011 Health Libraries Group

28 Health Information and Libraries Journal, 29, pp.28–38

Adverse effects search filters, S Golder & YK Loke 29

Introduction

To make informed balanced decisions, patients,clinicians, prescribers and other decision makersneed appropriate information on both the intendedbenefits and unwanted harms of a healthcareintervention.1,2

Currently, however, there is an emphasis on evi-dence of the beneficial effects and a relative lackof rigorous evaluations of adverse effects. Thiscreates a challenging conundrum.3 Decision mak-ers are obliged to struggle with a situation wherethey must weigh up quantitative efficacy dataagainst incomplete or inadequate adverse effectsinformation of uncertain quality.3,4 This imbalancein information may lead to interventions being pre-scribed inappropriately or in patients being harmedby potentially avoidable adverse effects.

To redress the over-emphasis on benefits, thedevelopment and improvement of methods toquantify adverse effects needs to be a key researchpriority. This will ensure that information on harmscan be considered at the same time and on an equalstanding with, information concerning benefit.

The first step towards quantifying adverseeffects is to retrieve good quality data on theirassociation and frequency with particular interven-tions. This requires the development of optimalsearch techniques to retrieve information onadverse effects.

Developing efficient search strategies (combina-tions of search terms) for use in databases (such asMEDLINE and EMBASE) which capture all therelevant literature on adverse effects is difficult.5,6

Specific difficulties arise when adverse effectsterms are added to the search strategy. This isbecause adverse effects are poorly reported, inade-quately indexed, inconsistently described and canbe new or unexpected at the time of searching.7,8

The most prominent difficulty in developingefficient search strategies for adverse effects is thatof poor reporting of adverse effects. Adverse out-comes are unlikely to be the primary outcome of astudy6,9 and few authors of trials devote substantialamounts of space to safety data.10 Adverse effectsare, therefore, often not reported in the title,abstract or indexing of a database record, makingthe creation of search strategies to capture adverseeffects difficult.6,8,9,11–15 Hence, to avoid missing

ª 2011 The authors. Health Information and Libraries Journal ªHealth Information and Libraries Journal, 29, pp.28–38

out any relevant studies, Derry et al. in 2001 pro-posed an arduous and highly resource intensiveapproach based on a broad search without adverseeffects terms, where a large number of potentiallyirrelevant full-text articles would need to be manu-ally screened. However, more efficient methods ofelectronic searching have since been developedand tested to try and ease the process of identify-ing and retrieving studies of specific interest. Thisincludes search strategies to retrieve particulartypes of information, for example, to retrieve spec-ified study designs (such as, RCTs) or subjectareas (such as, public health) or specific popula-tions (such as, the elderly)16 which has led to thedevelopment of predefined search strategies knownas search filters or search hedges.16–19 A search fil-ter is a predefined combination of search termsdesigned to retrieve information on a particulartopic. The filter may be created and evaluated invarious ways. For example, search terms in a filtermay be subjectively derived by contacting expertsin literature searching or the topic area. Alterna-tively, search terms may be objectively derivedusing word frequency analysis or statistical analy-sis on a set of relevant records. The best combina-tion of search terms can then be identified byrunning proposed search combinations and measur-ing how many relevant and irrelevant records areretrieved. Alternatively, word frequency or statisti-cal analysis, such as logistic regression, can beused to suggest the best combination of searchterms. Once a search filter has been developed, itmay or may not then be tested against a differentset of relevant records (a validation set).16

Despite the problems of searching for studies onadverse effects, attempts have been made todevelop a search filter. There are currently 12 pub-lished search filters for MEDLINE9,20–25 and threefor EMBASE22,24 (Box 1). All these filters, withthe exception of the filters by Buckingham et al.23,have been developed to maximise sensitivity, thatis, to retrieve a high proportion of all available rel-evant records. The search filters by Wieland andDickersin9,25 aim to identify as many of the rele-vant records as possible for a named specificadverse effect (breast cancer with oral contracep-tives), whereas the other filters aim to capture allor all serious adverse effects for a particular inter-ventions. The majority of the search filters have

2011 Health Libraries Group

Box 1 MEDLINE and EMBASE search strategies

MEDLINE (OVID: 1996 to July Week 1 2010) Searched: 21 ⁄ 07 ⁄ 10

Original Search

1. thiazolidinediones ⁄ OR rosiglitazone$.af. OR avandia.af. OR avandaryl.af OR avaglim.af. OR avandamet.af. OR

glitazone$.af. OR thiazolidinedion$.af. OR tzd OR ppar gamma agonist$.af. OR peroxisome proliferator activated

receptor gamma agonist$.af. OR pioglitazone$.af. OR actos.af. OR actoplus.af OR duetact.af OR competact.af. OR

glustin.af. OR nyracta.af. OR venvia.af. OR 111025 46 8.rn. OR 122320 73 4.rn.

2. exp Fractures, Bone ⁄ OR fracture$.af OR bone density ⁄ OR bone$.af OR bmd.af OR exp osteoporosis ⁄ OR

osteoporo$.af

3. 1 AND 2

Badgett 199920,21


glitazone$.af. OR thiazolidinedion$.af. OR tzd.af OR ppar gamma agonist$.af. OR peroxisome proliferator activated




osteoporo$.af

3. ((ae OR co OR po OR de).fs OR case report ⁄ ) AND humans ⁄4. 1 AND 2 AND 3

BMJ Clinical Evidence 200622






osteoporo$.af

3. (ae OR to OR po OR co).fs. OR (safe OR safety).ti,ab. OR side effect$.ti,ab. OR ((adverse OR undesirable OR

harm$ OR serious OR toxic) adj3 (effect$ OR reaction$ OR event$ OR outcome$)).ti,ab. OR exp product

surveillance, postmarketing ⁄ OR exp adverse drug reaction reporting systems ⁄ OR exp clinical trials, phase iv ⁄ OR exp

poisoning ⁄ OR exp substancerelated disorders ⁄ OR exp drug toxicity ⁄ OR exp abnormalities, drug induced ⁄ OR exp

drug monitoring ⁄ OR exp drug hypersensitivity ⁄ OR (toxicity OR complication$ OR noxious OR tolerability).ti,ab. OR

exp Postoperative Complications ⁄ OR exp Intraoperative Complications ⁄4. 1 AND 2 AND 3

Buckingham 2005a23 Without the quick filter (hedge)

1. thiazolidinediones ⁄ ae, ct, po, to

2. exp fractures, bone ⁄ ci, ep, et OR bone density ⁄ ab, ci, ep, et, tm OR exp osteoporosis ⁄ ab, ci, ep, et,tm

3. 1 AND 2

Buckingham 2005b23 With the quick filter (hedge)

1. thiazolidinediones ⁄ ae, ct, po, to

2. exp fractures, bone ⁄ ci, ep, et OR bone density ⁄ ab, ci, ep, et, tm OR exp osteoporosis ⁄ ab, ci, ep, et,tm

3. case control studies ⁄ OR cohort studies ⁄ OR risk ⁄4. 1 AND 2 AND 3

Golder 2006a24 Most sensitive search strategy






osteoporo$.af

3. exp Fractures, Bone ⁄ ci OR bone density ⁄ ci OR exp osteoporosis ⁄ ci

4. (ae OR co OR de).fs

Adverse effects search filters, S Golder & YK Loke


Health Information and Libraries Journal, 29, pp.28–38

30

Box 1 Continued.

5. (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs

OR (adverse adj2 (effect OR effects OR reaction OR reactions OR event OR events OR outcome OR

outcomes))).ti,ab

6. 1 AND 2 AND (3 OR 4 OR 5)

Golder 2006b24 Most sensitive search strategy excluding use of specified adverse effects






osteoporo$.af

3. (ae OR co OR de).fs



outcomes))).ti,ab

5. 1 AND 2 AND (3 OR 4) The addition of the fracture terms means that the Golder et al.24 ‘most sensitive search

strategy’ is essentially the same as the Golder 2006b ‘most sensitive search strategy excluding use of specified

adverse effects’

Wieland 2005a9,25 Exploding MeSH term search

1. humans ⁄ AND journal article.pt

2. exp fractures, bone ⁄ OR bone density ⁄ OR exp osteoporosis ⁄3. thiazolidinediones ⁄4. exp risk ⁄ OR exp follow-up studies ⁄ OR exp case-control studies ⁄5. 1 AND 2 AND 3 AND 4

Wieland 2005b9,25 Maximise Precision: MeSH term search with major topics and subheadings


2. *Fractures, Bone ⁄ OR * bone density ⁄ OR *osteoporosis ⁄3. thiazolidinediones ⁄4. risk ⁄ OR risk factors ⁄ OR follow-up studies ⁄ OR odds ratio ⁄5. 1 AND 2 AND 3 AND 4

Wieland 2005c9,25 MeSH term search without study methodology terms


2. *Fractures, Bone ⁄ OR * bone density ⁄ OR *osteoporosis ⁄3. thiazolidinediones ⁄4. 1 AND 2 AND 3

Wieland 2005d9,25 Text word search with automatic term mapping


2. exp fractures, bone OR fracture.tw OR fractures.tw OR exp bone and bones ⁄ OR bone.tw OR bones.tw OR bmd.tw

OR osteoporosis.tw

3. rosiglitazone.tw. OR avandia.tw. OR avandaryl.tw OR avaglim.tw. OR avandamet.tw. OR glitazone.tw. OR

glitazones.tw OR thiazolidinediones.tw. OR tzd.tw OR ppar gamma agonist.tw. OR peroxisome proliferator

activated receptor gamma agonist.tw. OR pioglitazone.tw. OR actos.tw. OR actoplus.tw OR duetact.tw OR

competact.tw. OR glustin.tw. OR nyracta.tw. OR venvia.tw.

4. exp risk ⁄ OR risk.tw OR follow-up.tw OR exp epidemiology ⁄ OR epidemiology.tw OR epidemiologic.tw

5. 1 AND 2 AND 3 AND 4

Wieland 2005e9,25 Text word search with truncation and double quotes


2. fracture$.tw OR bone$.tw OR bmd.tw OR osteoporo$.tw

3. rosiglitazone$.tw. OR avandia.tw. OR avandaryl.tw OR avaglim.tw. OR avandamet.tw. OR glitazone$.tw. OR

thiazolidinedion$.tw. OR tzd.tw OR ppar gamma agonist$.tw. OR peroxisome proliferator activated receptor gamma

agonist$.tw. OR pioglitazone$.tw. OR actos.tw. OR actoplus.tw OR duetact.tw OR competact.tw. OR glustin.tw.

OR nyracta.tw. OR venvia.tw.




31

Box 1 Continued.

4. risk.tw OR epidemiolog$.tw

5. 1 AND 2 AND 3 AND 4

Wieland 2005f9,25 Text word search without study methodology text words


2. fracture$.tw OR bone$.tw OR bmd.tw OR osteoporo$.tw

3. rosiglitazone$.tw. OR avandia.tw. OR avandaryl.tw OR avaglim.tw. OR avandamet.tw. OR glitazone$.tw. OR

thiazolidinedion$.tw. OR tzd.tw OR ppar gamma agonist$.tw. OR peroxisome proliferator activated receptor gamma

agonist$.tw. OR pioglitazone$.tw. OR actos.tw. OR actoplus.tw OR duetact.tw OR competact.tw. OR glustin.tw.

OR nyracta.tw. OR venvia.tw.

4. 1 AND 2 AND 3

EMBASE (OVID: 1996–2010 Week 28)

Searched: 21 ⁄ 07 ⁄ 10

Original Search

1. 2,4 thiazolidinedione derivative ⁄ OR exp glitazone derivative ⁄ OR rosiglitazone$.af. OR avandia.af. OR avandaryl.af

OR avaglim.af. OR avandamet.af. OR glitazone$.af. OR thiazolidinedion$.af. OR tzd OR ppar gamma agonist$.af.

OR peroxisome proliferator activated receptor gamma agonist$.af. OR pioglitazone$.af. OR actos.af. OR actoplus.af

OR duetact.af OR competact.af. OR nyracta.af. OR venvia.af. OR 111025 46 8.rn. OR 122320 73 4.rn.

2. exp fracture ⁄ OR fracture$.af OR bone density ⁄ OR bone$.af. OR bmd.af OR exp osteoporosis ⁄ OR osteoporo$.af

3. 1 AND 2

BMJ Clinical Evidence 20063


OR avaglim.af. OR avandamet.af. OR glitazone$.af. OR thiazolidinedion$.af. OR tzd.af OR ppar gamma

agonist$.af. OR peroxisome proliferator activated receptor gamma agonist$.af. OR pioglitazone$.af. OR actos.af.

OR actoplus.af OR duetact.af OR competact.af. OR nyracta.af. OR venvia.af. OR 111025 46 8.rn. OR 122320 73

4.rn.

2. exp fracture ⁄ OR fracture$.af OR bone density ⁄ OR bone$.af OR bmd.af OR exp osteoporosis ⁄ OR osteoporo$.af

3. (ae OR si OR to OR co).fs. OR (safe OR safety).ti,ab. OR side effect$.ti,ab. OR ((adverse OR undesirable OR harm$

OR serious OR toxic) adj3 (effect$ OR reaction$ OR event$ OR outcome$)).ti,ab. OR exp adverse drug

reaction ⁄ OR exp drug toxicity ⁄ OR exp intoxication ⁄ OR exp drug safety ⁄ OR exp drug monitoring ⁄ OR exp drug

hypersensitivity ⁄ OR exp postmarketing surveillance ⁄ OR exp drug surveillance program ⁄ OR exp phase iv clinical

trial ⁄ OR (toxicity OR complication$ OR noxious OR tolerability).ti,ab. OR exp postoperative complication ⁄ OR exp

Peroperative Complication ⁄4. 1 AND 2 AND 3

Golder 2006a24 Most sensitive search strategy



agonist$.af. OR peroxisome proliferator activated receptor gamma agonist$.af. OR pioglitazone$.af. OR actos.af.

OR actoplus.af OR duetact.af OR competact.af. OR nyracta.af. OR venvia.af. OR 111025 46 8.rn. OR 122320 73

4.rn.


3. exp fracture ⁄ si OR exp osteoporosis ⁄ si4. (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs


outcomes))).ti,ab

5. 1 AND 2 AND (3 OR 4)

Golder 2006b24 Most sensitive search strategy excluding use of specified adverse effects



agonist$.af. OR peroxisome proliferator activated receptor gamma agonist$.af. OR pioglitazone$.af. OR actos.af. OR

actoplus.af OR duetact.af OR competact.af. OR nyracta.af. OR venvia.af. OR 111025 46 8.rn. OR 122320 73 4.rn.





32

Box 1 Continued.

3. 2,4 thiazolidinedione derivative ⁄ ae,to OR exp glitazone derivative ⁄ ae,to



outcomes))).ti,ab

5. 1 AND 2 AND (3 OR 4)

MEDLINE Subheadings: ab, Abnormalities; ae, Adverse Effects; co, Complications; ct, contraindications; ci, Chemically

Induced; de, Drug Effects; ep, Epidemiology; et, Etiology; po, Poisoning; to, Toxicity; tm, Transmission. EMBASE

Subheadings: ae, Adverse Drug Reaction; co, Complication; to, Drug Toxicity; si, Side Effect Fields Searched: ab; abstract;

af, all fields; fs, floating subheading; rn, registry number; ti, title.


been developed using research techniques andtested.9,20,21,24,25 However, the precision of thesearch filters was not recorded in Badgett 1999,and the search filters by Golder et al. 2005 andWieland et al. 2005 were not validated with othercase studies.9,20,21,24,25 The other strategies avail-able are based solely on expert opinion and havenot yet been evaluated.26

The aim of this research was to explore the sen-sitivity (the ability to identify as many relevantarticles as possible), precision (the ability toexclude as many irrelevant articles as possible)and number needed to read (NNR) (the number ofrecords requiring sifting to identify one relevantarticle) of adverse effects search filters.

Methods

The included references from an ongoing update ofa previously published systematic review onfracture-related adverse effects associated with theuse of thiazolidinediones (rosiglitazone and pioglit-azone) formed the basis of analysis for this study.27

The search strategy used for the systematic reviewon thiazolidinediones-related fractures containedjust two facets, the intervention – thiazolidinedione(rosiglitazone and pioglitazone); and the outcomes –fractures or bone mineral density, with no genericadverse effects terms (such as ‘safety’ ‘side effect’or ‘adverse event’). This meant that the effect of theaddition of adverse effects terms or filters could beassessed in the present study. Indexing terms andterms in the title and abstract (including multiplesynonyms) were used for each facet. The searcheswere carried out in MEDLINE and EMBASE, aswell as numerous other sources.


The adverse effects search filters available forMEDLINE and EMBASE were then run to assesshow many of the relevant references would havebeen retrieved had these filters been applied, andhow many records in total would have beenretrieved. From the numbers of relevant recordsand total records retrieved, the sensitivity, preci-sion and NNR could be calculated for eachadverse effects search filter. Each search filter wasrun in turn, and a separate EndNote library createdfor the results from each strategy. The sensitivity,precision and NNR for the searches in each of thedatabases were measured.

Sensitivity of the search is the extent to whichthe search was able to pick up, and not miss outon relevant articles. This reflects how many of the‘included’ studies in a systematic review werepotentially identifiable through the use of a partic-ular search filter or database. If the search had lowsensitivity, it would miss out a fairly large propor-tion of relevant articles. In contrast, a highly sensi-tive search is constructed so that it can pick upmost of the relevant articles.

The precision of the search is analogous to theclinical term ‘positive predictive value’28. Here it isa measure of the ability of the search strategy toretrieve records that are genuinely relevant, suchthat they can be included in a systematic review.Precision can be calculated by taking the number ofrelevant (includable) studies from that particularsearch string as a proportion of the total number ofhits identified. A search with low precision wouldyield a large number of hits of which only a few aretruly relevant for the systematic review, whereas ahighly precise search would yield a cohort of stud-ies, most of which would be of genuine relevance.


Adverse effects search filters, S Golder & YK Loke34

However, most searches have to use a trade-offbetween sensitivity and precision, as highly sensi-tive searches may be imprecise, thus generating asubstantial workload for those screening articles insystematic reviews. This balance can be expressedas a product of the sensitivity and precision.Equally, the NNR is a more easily comprehensiblestatistic analogous to that of the clinical measure‘number needed to treat for benefit’ where a certainnumber of patients have to be treated for one bene-ficial outcome to occur 28. In the context of search-ing, NNR refers to number of references that haveto be checked to find one additional relevant articlethat is. Typically, in a systematic review, thiswould be the number of titles or abstracts retrievedfrom the electronic search that would have to bemanually checked and considered to pick up oneadditional relevant article from the set of retrievedcitations. A relatively high NNR means that a lot ofreferences would have to be checked, thus havingimportant resource implications in terms of timeand cost, whereas a low NNR means that relevantarticles can be more quickly identified without hav-ing to check large numbers of titles and abstracts.

The following definitions were used to calculatethe sensitivity, precision and NNR;

Sensitivity %ð Þ¼Number of included records retrivedTotal number of included records

�100

Precision %ð Þ¼Number of included records retrivedTotal number of records retrived

�100

NNR ¼ Total number of records retrievedNumber of included record sretrived

OR1=precision

In addition, sensitivity · precision29 was calcu-lated to allow an equilibrium between sensitivityand precision to be assessed.

As the adverse effect used in this investigation(fractures) was known before the searches wereconducted, some adaptations to the generic adverseeffects search strategies by Badgett and Chi-quette20,21 and Golder et al.24 were made (Box 1).Generic search filters are designed to identify anyadverse effects of an intervention. However, theadverse effect of interest was known in thisinstance so the generic search strategies proposedby Badgett 1999 and Golder et al. 2005 (whichcontain only generic adverse effects related termssuch as ‘side effects’ and ‘adverse events’) werecombined (ANDed) with the fracture terms.

ª 2011 The authors. Health Info

The search strategies by Wieland and Dicker-sin9,25 were translated from the PubMed interface toOVID MEDLINE. Two of the search strategies inWieland and Dickersin9,25 do not contain any inter-vention terms. These search strategies were nottested in this case study because of the large volumeof records (over 8000 records with MeSH terms andover 40 000 records with textwords) that running asearch for ‘fracture’ and ‘risk’ terms alone generatesand thus the impractical nature of employing such asearch strategy for this type of systematic review.

Results

Fifty eight references (representing 41 studies)contained sufficient data to allow meta-analysis forthe outcomes of interest (fractures) and were thusincluded in the systematic review. Of these 58 ref-erences, 19 were identified in MEDLINE (12observational studies and seven RCTs) and 24were identified in EMBASE (14 observationalstudies and 10 RCTs).

MEDLINE

The original search strategy using only the drugterms for thiazolidinediones and the named adverseeffect (fractures) retrieved 251 records of which 19contained relevant adverse effects data (representing18 studies) (Table 1). If the Golder et al.24 (strategya, b see Box 1) search filter had been applied (inaddition to the thiazolidinediones and fractureterms), then the same number of relevant references(19) would have been identified with 50 fewerrecords to sift (Table 1). The search filters Badgettand Chiquette20,21, BMJ Clinical Evidence22 andWieland and Dickersin9,25 (strategy f see Box 1)each missed only one relevant reference (each filtermissed a different reference). In the case of theBadgett and Chiquette20,21 filter, the missed refer-ence would have been identified had the textword‘safety’ been included in the search strategy. Thepaper missed by the BMJ Clinical Evidence22 filterwould have been retrieved if the subheading ‘drugeffect’ had been included (de.fs), and in the case ofthe Wieland and Dickersin9,25 (strategy f see Box 1)filter, the missed reference would have been identi-fied had the MeSH term ‘Fracture, bone ⁄ ci’ beenincluded (ci is the subheading ‘chemically induced).

rmation and Libraries Journal ª 2011 Health Libraries Group


Table 1 Sensitivity, precision and number needed to read (NNR) of search filters in MEDLINE and EMBASE

Number of

Records

Retrieved

Number of

Relevant

Records

Sensitivity

(n = 19)

(%)

Precision

(%) NNR

Sensitivity*

Precision

(%)

MEDLINE

Original Search without any

adverse effects filter

251 19 100 7.57 13 7.57

Badgett and Chiquette20,21* 148 18 94.74 12.16 8 11.52

BMJ Clinical Evidence22 118 18 94.74 15.25 7 14.45

Buckingham et al.23 49 10 52.63 20.41 5 10.74

Buckingham et al.23

(strategy b see Box 1)

11 6 31.58 54.55 2 17.23

Golder et al. (2006)*24 201 19 100 9.45 8 9.45

Wieland and Dickersin9 22 10 52.63 45.45 2 23.92

Wieland and Dickersin25 10 4 21.05 40.00 3 8.42

Wieland and Dickersin9,25

(strategy c see Box 1)

28 11 57.89 39.29 2 22.74


(strategy d see Box 1)

45 8 42.11 17.78 6 7.49


(strategy e see Box 1)

58 11 57.89 18.97 5 10.98


(strategy f see Box 1)

120 18 94.74 15.00 7 14.21

EMBASE

Original Search without any

adverse effects filter

1017 24 100 2.36 42 2.36

BMJ Clinical Evidence22 689 20 83.33 2.90 34 2.42

Golder et al.*24 396 15 62.50 3.79 26 2.37

Golder et al.24

(strategy b see Box 1)

471 19 79.17 4.03 25 3.19

*Adapted search strategy.


High precision was achieved by the Buckinghamet al.23 (strategy b see Box 1) (strategy b, c see Box1) and Wieland 2005c9,25 filters. These search strate-gies all rely only on MeSH terms. The Buckinghamet al.23 (strategy b see Box 1) Wieland and Dicker-sin9,25 (strategy c see Box 1) filters also achieved thebest combination of sensitivity and precision.

EMBASE

The original search strategy using only the drugterms for thiazolidinediones and the named adverseeffect (fractures) retrieved 1017 records of which24 contained relevant adverse effects data (repre-senting 23 studies) (Table 1). Although fewerrecords required sifting with the addition of thesearch filters (328 less with the BMJ Clinical Evi-dence22 filter, 546 less with the Golder et al.24

(strategy b see Box 1) filter and 621 less with


Golder et al.24 filter), if any of the filters had beenapplied then not all the relevant records wouldhave been identified. Three of the four referencesmissed by all the search filters included the text-word ‘risk’ but did not include any other termsrelated to ‘adverse effects’ in the title, abstract orindexing. The other reference missed by all thesearch filters did not contain any generic adverseeffects terms (but did include fracture terms).

The additional reference missed by the Golderet al.24 filter was identified by the BMJ ClinicalEvidence22 search strategy by the textword ‘com-plications’. The additional 5 references missed bythe Golder et al.24 filter were identified by theBMJ Clinical Evidence22 filter with the subheading‘Adverse Drug Reaction’ ( ⁄ ae) (4 references) orthe textword ‘complications’ (1 reference).

Although the addition of the search filtersdid improve the precision of the searches, the



precision for all the searches remained low atbelow 5% (Table 1). The best combination of sen-sitivity and precision was achieved by the Golderet al.*24 filter; however, none of the filtersachieved a good balance because of low precision.

Limitations

The main limitation to the present study is thatonly one case study systematic review was used,limiting the generalisability of the results. In addi-tion, this case study was of a named adverseeffect, while a case study of a safety profile sys-tematic review, in which all adverse effects aresearched for, may have given different results.

Another limitation is the adaptations made to thesearch strategies by Badgett and Chiquette20,21 andGolder et al.24 These filters were originally createdand tested for use in searches where the adverseeffects are not known in advance of searching,while in the present study, these search filters wereused in addition to fracture terms. However, usingthe filters without the fracture terms would beimpractical given the vast number of irrelevantrecords that would have been retrieved (over 4000with the filter by Badgett and Chiquette20,21 andover 6000 with the Golder et al.24 filters).

Discussion

In MEDLINE, four search filters [Badgett & Chi-quette20,21, BMJ Clinical Evidence22, Golder etal.24 (strategy a, b see Box 1) and Wieland andDickersin9,25 (strategy f see Box 1)] achieved ahigh level of sensitivity (94.74 or 100%) with animproved level of precision from the originalsearch strategies. This implies that some of theMEDLINE adverse effects search filters could havebeen a useful addition to the search strategies.

The highest precision in MEDLINE wasachieved from those search filters that rely onMedical Subject Headings (MeSH). The searchstrategies by Buckingham et al.23 are designed toachieve high precision (whereas the other filtersare designed with systematic reviews in mind) soit is not that surprising that Buckingham et al.23

(strategy b see Box 1), filter achieved the highestprecision. However, as with all searching, therewas a trade-off between sensitivity and precision.


In EMBASE, the BMJ Clinical Evidence22 andGolder et al.24 (strategy b see Box 1) search filtersachieved a level of sensitivity that may be consid-ered acceptable by some authors of systematicreviews, with a reduction in the number of recordsto sift. However, none of the search filtersachieved 100% sensitivity, and precision was lowusing any of the filters. Although precisionremains low with the addition of adverse effectsfilters in EMBASE, in practical terms, hundredsless records required sifting. Therefore, dependingon the level of sensitivity judged to be acceptable,these filters could be of use in situations whereunmanageable numbers of records are retrievedotherwise.

In terms of precision, the results from searchingMEDLINE and EMBASE differed remarkably.Searches in MEDLINE achieved a much higherprecision (or lower NNR) than similar searches inEMBASE. This may be due to differences in thepractice of indexing in the two databases. Otherstudies have indicated that searching EMBASEresults in lower precision than MEDLINE.30,31

This research indicates the potential value ofadverse effects search filters. However, as 100%sensitivity was not achieved for any of the filters,they should be applied with caution. Authors ofsystematic reviews may therefore need to takepragmatic decisions when creating search strategiesfor adverse effects and sacrifice sensitivity for pre-cision. To compensate for this loss in sensitivity,searches of electronic databases could then be sup-plemented with other means of identifying paperssuch as reference checking, contacting industryand citation searches.

The complete search strategy for identifyingpapers in a systematic review on adverse effects isalso likely to depend on the inclusion criteria forthe review. For example, if the inclusion criteriaare limited to particular study designs, then thesearch strategy may need to reflect this. Thesesearch filters could therefore be adapted for thereview in question, so need not be used prescrip-tively but more as a basis for ideas of terms.

Conclusions

Adverse effects search filters in MEDLINE whencombined with specific adverse effects terms can




achieve a high level of sensitivity with animproved level of precision. In addition a highlevel of precision can be achieved in MEDLINEwith adverse effects indexing terms. However, itappears difficult to achieve similar high levels ofsensitivity and precision with search filters inEMBASE.

Further research is required with more casestudy systematic reviews for a range of interven-tions (pharmacological and non-pharmacological)and a diverse variety of adverse effects to testthe generalisability of these findings. Similarly, asadverse effects data are described from differentstudy designs (often using non-randomiseddesigns), it would help to design and evaluate thevalue of specific filters for particular types ofstudies such as case–control designs We suggestthat a broad range of reviews where no adverseeffects terms were used in the search processcould be assessed to determine whether the addi-tion of an adverse effect filter would haveimproved precision of the searches without lossof sensitivity. This should include not just sys-tematic reviews for specific named adverse effects(such as fractures here) but also reviews whichaim to provide a broad evaluation of the safetyprofile of an intervention, thereby capturing alladverse effects.

Acknowledgements

We would like to thank Dave Fox, Kath Wrightand Professor Lesley Stewart of CRD for com-ments on an early draft.

Source of funding

This research was undertaken by Su Golder as partof a Medical Research Council (MRC) fellowship.The views expressed in this presentation are thoseof the authors and not necessarily those of theMRC. The funders had no role in study design,data collection and analysis, decision to publish orpreparation of the manuscript.

Conflict of interest

No conflicts of interests are declared by theauthors.


References

1 Keech, A. C., Wonders, S. M., Cook, D. I. & Gebski, V. J.Balancing the outcomes: reporting adverse events. MedicalJournal of Australia 2004, 181, 215–218.

2 Strom, B. L. What is Pharmacoepidemiology? In: Strom, B.L. (ed). Pharmacoepidemiology. Chichester: John Wiley,2006. 3–11.

3 Loke, Y. K. Assessing the benefit-harm balance at thebedside. BMJ 2004, 329, 7–8.

4 Pirmohamed, M. & Darbyshire, J. Collecting and sharinginformation about harms: appropriate strategies tocommunicate this information are essential. BMJ 2004, 329,6–7.

5 Ross, S. D. Drug-related adverse events: a readers’ guide toassessing literature reviews and meta-analyses. Archives ofInternal Medicine 2001, 161, 1041–1046.

6 Derry, S., Loke, Y. K. & Aronson, J. K. Incompleteevidence: the inadequacy of databases in tracing publishedadverse drug reactions in clinical trials. BMC MedicalResearch Methodology 2001, 1, 7.

7 MacLaughlin, P. Integration of pre-clinical, clinical andpost-market safety data to help researchers assess drugcandidates and advance drug development. In: Pharma-Bio-Med 2007 Conference and Exposition; 2007 11–14November; Athens, Greece. 2007.

8 Etminan, M., Carleton, B. & Rochon, P. A. Quantifyingadverse drug events : are systematic reviews the answer?Drug Safety 2004, 27, 757–761.

9 Wieland, S. & Dickersin, K. Selective exposure reportingand Medline indexing limited the search sensitivity forobservational studies of the adverse effects of oralcontraceptives. Journal of Clinical Epidemiology 2005, 58,560–567.

10 Ioannidis, J. P. & Lau, J. Completeness of safety reportingin randomized trials: an evaluation of 7 medical areas.JAMA 2001, 285, 437–443.

11 Loke, Y. K., Edwards, J. & Deny, S. Conventional searchstrategies cannot easily identify those trials of drug therapywhich provide quantitative adverse effects data. 7th AnnualCochrane Colloquium Abstracts, October 1999 in Rome 1999.

12 Loke, Y. K. The applications and limitations of systematicreview and meta-analysis in assessing adverse drugreactions. London: King’s College, 2003.

13 Windsor, D. A. Adverse-reactions literature: a bibliometricanalysis. Methods of Information in Medicine 1977, 16,52–54. Available at: http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=ISTA1202362&site=ehost-live.

14 Nuovo, J. & Sather, C. Reporting adverse events inrandomized controlled trials. Pharmacoepidemiology andDrug Safety 2007, 16, 349–351.

15 Bernal-Delgado, E. & Fisher, E. S. Abstracts in high profilejournals often fail to report harm. BMC Medical ResearchMethodology 2008, 8, 14.

16 Jenkins, M. Evaluation of methodological search filters – areview. Health Information and Libraries Journal 2004, 21,148–163.



17 White, V. J., Glanville, J. M., Lefebvre, C. & Sheldon, T.A. A statistical approach to designing search filters to findsystematic reviews: objectivity enhances accuracy. Journalof Information Science 2001, 27, 357–370.

18 Wilczynski, N. L., Morgan, D., Haynes, R. B. & HaynesTeam An overview of the design and methods for retrievinghigh-quality studies for clinical care. BMC MedicalInformatics and Decision Making 2005, 21, 20.

19 Glanville, J. M., Lefebvre, C., Miles, J. N. & Camosso-Stefinovic, J. How to identify randomized controlled trials inMEDLINE: ten years on. Journal of the Medical LibraryAssociation 2006, 94, 130–136.

20 Badgett, R., Chiquette, E., Anagnostelis, B. & Mulrow, C.Locating reports of serious adverse drug reactions(PowerPoint presentation). 1999. Available at: http://medinformatics.uthscsa.edu/#FILTERS (accessed 7 January2005).

21 Badgett, R., Chiquette, E., Anagnostelis, B. & Mulrow, C.Locating reports of serious adverse drug reactions. 7thAnnual Cochrane Colloquium Abstracts, October 1999:Rome, Italy. 1999. Available at: http://gateway.nlm.nih.gov/MeetingAbstracts/ma?f=102193751.html (accessed 2December 2011).

22 BMJ Clinical Evidence. Search filters. BMJ. 2006. Availableat: http://www.clinicalevidence.com/ceweb/about/search_filters.jsp (accessed 13 May 2005).

23 Buckingham, J., Fisher, B. & Saunders, D. OvidMedline ⁄ Pubmed search strategies for aticles oncausation ⁄ harm ⁄ etiology. Alberta: evidence Based MedicineTool Kit; 2005. Available at: http://www.ebm.med.ualberta.ca/HarmSearch.html (accessed 13 May 2005).

24 Golder, S., McIntosh, H. M., Duffy, S. & Glanville, J.Developing efficient search strategies to identify reports of


adverse effects in MEDLINE and EMBASE. HealthInformation and Libraries Journal 2006, 23, 3–12.

25 Wieland, S. & Dickersin, K. Erratum: selective exposurereporting and Medline indexing limited the search sensitivityfor observational studies of the adverse effects of oralcontraceptives (Journal of Clinical Epidemiology (June2005) 58:6 (560–567)). Journal of Clinical Epidemiology2005, 58, 1077.

26 Golder, S. & Loke, Y. K. Search strategies to identifyinformation on adverse effects: a systematic review. Journalof the Medical Library Association 2009, 97, 84–92.

27 Loke, Y. K., Singh, S. & Furberg, C. D. Long-term use ofthiazolidinediones and fractures in type 2 diabetes: a meta-analysis. CMAJ 2008, 180, 32–39.

28 Bachmann, L. M., Estermann, P., Kronenberg, C. & terRiet, G. Identifying diagnostic accuracy studies inEMBASE. Journal of the Medical Library Association2003, 91, 341–346.

29 Tanon, A., Champagne, F., Contandriopoulos, A.-P., Pomey,M.-P., Vadeboncoeur, A. & Nguyen, H. Patient safety andsystematic reviews: finding papers indexed in MEDLINE,EMBASE and CINAHL. Qual Saf Health Care 2010, 19,452–461.

30 Topfer, L., Parada, A., Menon, D., Noorani, H., Perras, C. &Serra-Prat, M. Comparison of literature searches on qualityand costs for health technology assessment using theMEDLINE and EMBASE databases. International Journal ofTechnology Assessment in Health Care 1999, 15, 297–303.

31 Woods, D. & Trewheellar, K. Medline and Embasecomplement each other in literature searches. BMJ 1998,316, 1166.

Received 13 May 2010; Accepted 16 November 2011



Documents

Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones