Searching beyond the RCT - looking for sibling studies on qualitative, economic

and process research

Searching beyond the RCT - looking for sibling studies on qualitative, economic

and process research

Faten Hamad and Christine UrquhartFaten Hamad and Christine Urquhart

Introduction

Systematic reviews – producing the evidence

…We have the evidence on what works (or doesn’t) – what happens next?

Finding qualitative, economic and process research evidence – preliminary findings of research to identify sibling studies associated with particular randomised controlled trials.

Aims and objectives of presentation are to:

Discuss searching strategies to find evidence beyond the randomised controlled trial

Present preliminary findings

Assess your reactions to the findings!

Background

Sibling studies is the name which has been chosen to indicate the relationship that groups a set of related studies (randomised control trial, qualitative, process and economic evaluations).

An intervention may be viewed as:

“a complex system where intervention itself is a fragile creature that is delivered in a social system

of interacting elements, such as an individual’s capacity, interpersonal relationships, institutional

setting and infrastructure.”

Search strategies and search filters

Existing work by Cochrane Information Retrieval Methods Group and others (e.g. Hedges team)

Aim to maximise recall (sensitivity), keeping precision reasonable, and ensuring specificity

Search strategies vary with the database – depends on index terms, which terms have to be added as free text terms…

Relevance judgements

Seed studies selection

We chose a range of topic areas, and dates for the large RCTs and chose five seed studies with different

characteristics. We will discuss the result of the following two of seed studies:

-Telemedicine and diabetes (a known RCT, with many known direct siblings, that could be used to validate

and checking the search strategies for their sensitivity).

- Tamoxifen for breast cancer prevention (known qualitative sibling, two RCTs involved).

Search strategies and databases under investigation

Different search strategies and different databases were explored for the reason of exploring the differences in search performance in regard to different study areas.

- Author-Subject search (using a very simple subject term combination) with each of the author names in the seed article in turn, in MEDLINE on PubMed.

-Related article search in MEDLINE on PubMed (for the seed article).

- E-library search (with a combination of ISI (WoS), OCLC WorldCat, OCLC Articles First, EBSCO Business Complete, and EBSCO International Bibliography) - simple subject term combination only, and limited to the first 300 documents retrieved.

Search strategies and databases under investigation-Cont.

- SCOPUS search (author-subject search as in MEDLINE on PubMed).

- CINAHL (author-subject search as in MEDLINE on PubMed).

- Cited reference in ISI, Web of Science (with the seed article as the reference).

- Cited reference search in SCOPUS.

- Cited reference search in CINAHL.

Results Table 1 IDEATeL study: search strategy and retrieval performance

search strategyRelevant/R Total

retrievedUnique

Relevant/ROdds Ratio

Related Search(PubMed) 25 186 5 1.01219

Author +Subject(PubMed) 23 157 4 1.09189

Citation(Web of Science) 14 57 1 1.86855

Subject search(e-library) 39 296 33 1.19589

SCOPUS Author subject 39 52 5 23.64179

SCOPUS citation 32 64 12 7.13514

CINAHL Author subject 17 18 0 100.85393

CINAHL citation 4 7 0 6.90196

Total relevant retrieved without duplicates 106

Total retrieved without duplicate 634

Total non-relevant retrieved without duplicates 528

Table 2 IDEATeL study search strategies retrieval per study type and odds ratio calculations.

search strategy RCTs Qualitative Economical evaluation

Process evaluation

R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3 R1 R/N2 OR3

Related Search(PubMed) 8 39 0.1032 10 21 0.2396 3 8 0.1887 4 13 0.1548

Author-Subject(PubMed) 8 39 0.1271 9 22 0.2534 2 9 0.1377 4 13 0.1906

Citation(Web of Science) 4 43 0.1990 8 23 0.7442 2 9 0.4755 0 17 0

Subject search(e-library) 25 22 0.2034 20 11 0.3254 9 2 0.8055 6 11 0.0976

SCOPUS Author - subject 17 30 2.9205 12 19 3.2551 2 9 1.1453 8 9 4.5812

SCOPUS citation 19 28 1.5692 8 23 0.8044 2 9 0.5139 3 14 0.4955

CINAHL Author subject 10 37 24.054 4 27 13.185 1 10 8.9 2 15 11.867

CINAHL citation 1 46 0.7391 3 28 3.6429 0 11 0 0 17 0

1 : Relevant retrieved. 2: Relevant not retrieved. 3: Odds Ratios.

Table 3 Tamoxifen study: search strategy and retrieval performance

search strategyRelevant/R Total Unique

Relevant/ROdds Ratio

Related Search(PubMed) 17 200 13 0.58277

Author+Subject(PubMed) 17 451 7 0.24573

Citation(Web of Science) 5 53 2 0.60155

Subject search (e-library) 59 288 54 2.316121

SCOPUS Author subject 72 229 47 4.76069

SCOPUS citation 4 59 2 0.41722

CINAHL Author subject 19 41 7 5.49701

CINAHL citation 1 4 0 1.87527

Total relevant R without duplicates 156

Total retrieved without duplicate 1028

Total non-relevant R without duplicates 872

Table 4 Tamoxifen study search strategies retrieval per study type and odds ratio calculations.

search strategy RCTs Qualitative Economic Process evaluation

R1 R/N2

OR3 R1 R/N2 OR3 R1 R/N2

OR3 R1 R/N2 OR3

Related Search(PubMed) 12 83 0.1098 3 31 0.0735 1 0 DIV/0

1 25 0.0304

Author-Subject(PubMed) 12 83 0.0463 1 33 0.0097 0 1 0 4 22 0.0582

Citation(Web of Science) 2 93 0.0677 2 32 0.1966 0 1 0 1 25 0.1258

Subject search(e-library) 26 69 0.1629 18 16 0.4864 0 1 0 13 13 0.4323

SCOPUS Author-subject 52 43 0.6470 10 24 0.2229 0 1 0 10 16 0.3344

SCOPUS citation 0 95 0 4 30 0.3685 0 1 0 0 26 0

CINAHL Author subject 15 80 1.1676 1 33 0.1887 0 1 0 3 23 0.8123

CINAHL citation 1 94 0.0750 0 34 0 0 1 0 0 26 0

1: Relevant retrieved. 2: Relevant not retrieved. 3: Odds Ratios.

Discussion

CINAHL author-subject search was the most effective search that can precisely retrieve direct and indirect siblings of a certain seed study in general with higher score in the case of the IDEATeL study in specific.

- IDEATeL, the number of retrieved records was 41 (with 19 relevant);

- Tamoxifen, the number retrieved was 18 (with 17 relevant);

CINAHL author- subject odds ratios were quite different, but this does not affect the fact this search strategy was the best search strategy, in terms of the chances of finding relevant material from a search, with proportionally fewer irrelevant items retrieved.

The SCOPUS author–subject strategy was the second best search performance.

For both the seed studies, the e-library subject search retrieved most of the unique studies.

Simple subject terms and / or the combination of simple subject terms and author names for each seed study, appeared to be the most effective method of retrieving most of the siblings, outperforming citation searching (apart from SCOPUS citation with the IDEATeL seed study).

Conclusions and future works

The analysis indicates that there is neither a winner in the search strategies nor for the databases.

The CINAHL author-subject performed well (in terms of precision).

The SCOPUS author-subject search performed the next best, (higher recall for the relevant studies for both seed studies).

The e-library author-subject search produced a good number of relevant studies (and a high proportion of unique items as

well).

Conclusions and future works

Further research will examine how expanding the number of databases and changing the selection will affect the relative performance of the e-library Meta-lib search.

Further work is needed to identify how grey literature, conference proceedings and thesis and dissertation material can be obtained efficiently, as Web of Knowledge found some of the direct sibling publications for the IDEATel studies that could not be obtained on any of the other databases used.

Thank you for Listening