http://qhr.sagepub.com/Qualitative Health Research

http://qhr.sagepub.com/content/16/1/162The online version of this article can be found at:

 DOI: 10.1177/1049732305284027

2006 16: 162Qual Health ResLeslie A. Walters, Nancy L. Wilczynski and R. Brian Haynes

Developing Optimal Search Strategies for Retrieving Clinically Relevant Qualitative Studies in EMBASE  

Published by:

http://www.sagepublications.com

can be found at:Qualitative Health ResearchAdditional services and information for    

  http://qhr.sagepub.com/cgi/alertsEmail Alerts:

 

http://qhr.sagepub.com/subscriptionsSubscriptions:  

http://www.sagepub.com/journalsReprints.navReprints:  

http://www.sagepub.com/journalsPermissions.navPermissions:  

http://qhr.sagepub.com/content/16/1/162.refs.htmlCitations:  

What is This? 

- Nov 29, 2005Version of Record >>

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

ARTICLEQUALITATIVE HEALTH RESEARCH / January 2006Walters et al. / OPTIMALSEARCH STRATEGIES

Developing Optimal Search Strategies forRetrieving Clinically Relevant QualitativeStudies in EMBASE

Leslie A. WaltersNancy L. WilczynskiR. Brian Haynes(for the Hedges Team)

Qualitative researchers address many issues relevant to patient health care. Their studiesappear in an array of journals, making literature searching difficult. Large databases such asEMBASE provide a means of retrieving qualitative research, but these studies representonly a minuscule fraction of published articles, making electronic retrieval problematic. Lit-tle work has been done on developing search strategies for the detection of qualitative studies.The objective of this study was to develop optimal search strategies to retrieve qualitativestudies in EMBASE for the 2000 publishing year. The authors conducted an analytic sur-vey, comparing hand searches of journals with retrievals from EMBASE for candidatesearch terms and combinations. Search strategies reached peak sensitivities at 94.2% andpeak specificities of 99.7%. Combining search terms to optimize the combination of sensitiv-ity and specificity resulted in values over 89% for both. The authors identified searchstrategies with high performance for retrieving qualitative studies in EMBASE.

Keywords: information storage and retrieval; qualitative research; medical informatics

Qualitative research can play an integral role in our understanding of thehuman experience of the consumers and providers of health care (Evans,

2002) while also tackling questions of how research evidence is turned into clinicalpractice, including matters that cannot be addressed by quantitative studies alone(Green & Britten, 1998). Qualitative data can further assist researchers in differenti-ating between interventions in a randomized control trial by assessing why theeffects have taken place (Grant, 2004).

For qualitative research to advance, researchers must be able to retrieve studiespublished to date easily (Wilczynski & Haynes, 2002). Furthermore, if qualitativeresearch is to inform health care practice, practitioners must be able to retrieve reli-ably and quickly sound studies that are directly relevant to the problem they are try-ing to solve (Wilczynski & Haynes, 2003). In both instances, online literature

162

AUTHORS’ NOTE: This research was funded by the National Library of Medicine, USA. The HedgesTeam includes Angela Eady, Brian Haynes, Susan Marks, Ann McKibbon, Doug Morgan, Cindy Walker-Dilks, Stephen Walter, Stephen Werre, Nancy Wilczynski, and Sharon Wong.

QUALITATIVE HEALTH RESEARCH, Vol. 16 No. 1, January 2006 162-168DOI: 10.1177/1049732305284027© 2006 Sage Publications

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

databases such as EMBASE, MEDLINE, PsychINFO, and CINAHL are most likelyto be the easiest routes of access, although the information might be buried withinthe volume of literature contained in these databases (Haynes & Wilczynski, 2004).

Ideally, users should be able to optimize their searches so that all relevant arti-cles for their needs are retrieved but irrelevant studies are not (Wilczynski &Haynes, 2002). In practice, searches for qualitative research are limited by theimperfections of indexing that exist and the very dilute concentration of qualitativestudies relative to reports of quantitative research. Indeed, little is known about theretrieval characteristics of search terms or the success of those looking forqualitative research reports.

We have previously shown that search filters (“hedges”) can optimize theretrieval of relevant studies for specific types of research from large electronic data-bases (Haynes & Wilczynski, 2004; Haynes, Wilczynski, McKibbon, Walker, &Sinclair, 1994; Wilczynski & Haynes, 2002, 2003; Wilczynski, McKibbon, & Haynes,2001; Wilczynski, Walker, McKibbon, & Haynes, 1993; Wong & Wilczynski, 2004;Wong, Wilczynski, Haynes, & Ramkissonsingh, 2003). To develop and test searchfilters for qualitative studies, index terms and textwords related to research designfeatures were run as search strategies. We treated the search strategies as “diagnos-tic tests” for sound studies and then treated the manual review of the literature asthe gold standard. In this article, we report our evaluation of the retrieval propertiesof search filters for identifying qualitative studies of relevance to health carequestions in EMBASE.

METHOD

In this study, we compared the retrieval performance of search terms and phrases inEMBASE with a manual review of each article for each issue of 55 journal titles forthe year 2000.

Journal Selection

The 55 journal titles reviewed were from a subset of 170 journals chosen on recom-mendations of clinicians and librarians, Science Citation Index Impact Factors pro-vided by the Institute for Scientific Information, and ongoing assessment of theiryield of studies and reviews of scientific merit and clinical relevance for the disci-plines of internal medicine, general medical practice, mental health, and generalnursing practice (Health Information Research Unit, 2005). Of the 170 journals, 135were indexed in EMBASE. Search strategies were developed on a 55-journalEMBASE subset, chosen on the basis of having the highest number of appropriatestudies across purpose categories. We previously developed search strategies inMEDLINE using all 161 journals indexed in MEDLINE (Wong, Wilczynski, Haynes,& Ramkissonsingh, 2004). We found that the developed search strategies are robustin smaller journal subsets and that computation time is substantially decreased. Wealso found that when strategies were developed in 60% of the database and vali-dated in the remaining 40%, there were no statistical differences in performance(Wong, Wilczynski, & Haynes, 2004). Thus, we developed search strategies forEMBASE using all data from 55 journals.

Walters et al. / OPTIMAL SEARCH STRATEGIES 163

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

Hand Search

Six research staff were rigorously trained before reviewing the 2000 literature, andinterrater agreement for application of all criteria exceeded 80% beyond chance(Haynes, Wilczynski, McKibbon, et al., 1994; Wilczynski, McKibbon, et al., 2001;Wilczynski, Walker, et al., 1993; Wong, Wilczynski, & Haynes, 2004; Wong, Wilczynski,Haynes, & Ramkissonsingh, 2003). The research assistants then hand-searchedjournals for the year 2000 and applied criteria to each item in each issue to determineif the article fit one or more of seven purpose categories (e.g., treatment, diagnosis,qualitative). Purpose category definitions and methodologic rigor criteria havebeen published previously (Haynes, Wilczynski, McKibbon, et al., 1994; Wilczynski,McKibbon, et al., 2001; Wilczynski, Walker, et al., 1993; Wong, Wilczynski, &Haynes, 2004; Wong, Wilczynski, Haynes, & Ramkissonsingh, 2003). Original stud-ies or review articles were classified as qualitative if the content related to how peo-ple experience certain situations and if the data collection and analytical methodsused were appropriate for qualitative data.

Search Term Selection

We compiled an initial list of index terms and textwords, then sought input from cli-nicians and librarians in the United States and Canada through interviews ofknown searchers, and requests at meetings and conferences. We asked individualsto identify which terms or phrases they used when searching for studies of a quali-tative nature as well as studies of prognosis, treatment, causation, diagnosis, eco-nomics, clinical prediction guides, reviews, and costs. We compiled a list of 5,385terms, of which 4,843 were unique and 3,524 returned results.1 Examples of thesearch terms tested are ethnographic theme, face to face interview, data saturation, andfocus group, all as textwords; videorecording, the index term, and the index term infor-mation processing, exploded.

Testing Search Terms and Analysis

We ran index terms and textwords related to study design features as search strate-gies, treating the search strategies as “diagnostic tests” for qualitative studies andthe manual review of the literature as the gold standard. Searches were run usingOvid’s search interface for EMBASE, and we used Ovid syntax in all strategies. Thesensitivity, specificity, precision, and accuracy of EMBASE searches were deter-mined. Sensitivity for a given topic is defined as the proportion of qualitative arti-cles retrieved; specificity is the proportion of nonqualitative research articles notretrieved; precision is the proportion of retrieved articles that are qualitative; andaccuracy is the proportion of all articles that are correctly classified. We incorpo-rated individual search terms with sensitivity greater than 25% and specificitygreater than 75% for a given purpose category into the development of search strat-egies that included a combination of two or more terms. All combinations of termsused the Boolean OR, for example, “random OR controlled.” (We did not use theBoolean AND, because this strategy invariably compromised sensitivity.) For thedevelopment of multiple-term search strategies to optimize either sensitivity orspecificity, we tested all two-term search strategies with sensitivity at least 75% and

164 QUALITATIVE HEALTH RESEARCH / January 2006

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

specificity at least 50%. For optimizing accuracy, two-term search strategies withaccuracy greater than 75% were considered for multiple-term development. Wetested 1,290 search strategies in the development of qualitative hedges.

FINDINGS

We downloaded indexing information from EMBASE for 27,769 articles from the 55journals hand-searched. Of these, 86 articles (0.31%) were classified as qualitativestudies. Thus, we tested the strategies for their ability to retrieve articles of a qualita-tive nature from all other articles. In Table 1, we show the best single term for highsensitivity, high specificity, and best balance of sensitivity and specificity. Two of thetop-performing single terms had specificities greater than 97.5%. The single term“interview:.mp.” yielded the best sensitivity, 70.9%, with a specificity of 97.5%.When specificity was maximized with the single term “qualitative.tw.” at 99.7%, anexpected reduction in sensitivity occurred, lowered to 57%. The best optimizationof sensitivity and specificity occurred with the single term “exp health care facilitiesand services,” which had a sensitivity of 62.8% and a specificity at 74%.

Combination of terms with the best results for sensitivity, specificity, and opti-mization of sensitivity and specificity are shown in Table 2. Compared to singleterms, the best combined terms reached higher peak sensitivity (94.2%) while main-taining comparable specificity (57.0%). The three-term strategy “interview:.tw. ORqualitative.tw. OR exp health care organization” yielded the best sensitivity, 94.2%,and had a specificity of 90%. Compared with the best sensitivity single term, “inter-view:.mp.” (70.9% sensitivity, 97.5% specificity), the best three-term strategyyielded an absolute increase in sensitivity of 23.3% but with an absolute decrease inspecificity of 7.5%. The two-term strategy “qualitative.tw. OR qualitative study.tw.”and the single-term strategy “qualitative.tw.” both yielded the best specificity,superior to any of the three-term strategies, at 99.7%, however, with an evident

Walters et al. / OPTIMAL SEARCH STRATEGIES 165

TABLE 1: Single Term With the Best Sensitivity (Keeping Specificity ≥ 50%), Best Specificity (Keep-ing Sensitivity ≥ 50%), and Best Optimization of Sensitivity and Specificity (Based on theLowest Possible Absolute Difference Between Sensitivity and Specificity) for DetectingQualitative Studies in EMBASE in 2000

Search Term, Sensitivity (%), Specificity (%), Precision (%), Accuracy (%),OVID Searcha 95% CI, n = 86 95% CI, n = 27,683 95% CIb 95% CI, n = 27,769

Best sensitivity, 70.9 97.5 8.9 97.5interview:.mp. (60.1 to 80.2) (97.3 to 97.7) (6.3 to 10.4) (97.3 to 97.6)

Best specificity, 57.0 99.7 33.8 99.5qualitative.tw. (45.8 to 67.6) (99.6 to 99.8) (26.2 to 42.1) (99.4 to 99.6)

Best optimization of 62.8 74.0 0.8 74.0sensitivity and (51.7 to 73.0) (73.5 to 74.5) (0.6 to 1.0) (73.5 to 74.5)specificity, exphealth care facilitiesand services

NOTE: : = truncation; mp = multiple posting (term appears in title, abstract, or subject heading); tw =textword (word or phrase appears in title or abstract); exp = exploded subject heading.a. The search strategy is reported using Ovid’s search engine syntax for EMBASE.b. Denominator varies by row.

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

decrease in sensitivity, which was lowered to 57%. The best optimization ofsensitivity and specificity yielded a sensitivity of 89.5% and specificity of 89.9%.

DISCUSSION

We have developed search filters that can optimize the retrieval of qualitative stud-ies in EMBASE. Searchers should use our findings as a means of determining themost effective trade-off between sensitivity and specificity for their own needs.Searchers who choose the most sensitive strategy will need to sort through morearticles that are irrelevant to their search but will avoid missing key articles. If asearcher would rather retrieve a smaller fraction of relevant studies, consideringthat some key articles might be omitted through that search, then the most specificstrategy would be best.

The low precision documented in our study is the inevitable consequence ofimperfect specificity and the very low proportion of qualitative studies in the jour-nals included in our sample. Although precision was low in our study, it reached34.3% with the most specificity strategy.

We previously tested similar search strategies in MEDLINE, identifying 49,028articles from matching hand searches with downloaded data in MEDLINE, of which366 (0.75%) were qualitative, compared to 27,769 articles identified in EMBASE, of

166 QUALITATIVE HEALTH RESEARCH / January 2006

TABLE 2: Combination of Terms With the Best Sensitivity (Keeping Specificity ≥ 50%), Best Speci-ficity (Keeping Sensitivity ≥ 50%), and Best Optimization of Sensitivity and Specificity(Based on abs[sensitivity-specificity] 1%) for Detecting Qualitative Studies in EMBASEin 2000

Search Term, Sensitivity (%), Specificity (%), Precision (%), Accuracy (%),OVID Searcha 95% CI, n = 86 95% CI, n = 27,683 95% CIb 95% CI, n = 27,769

Best sensitivity, 94.2 90.0 2.8 90.0interview:.tw.OR qualitative.tw. (89.2 to 99.1) (89.6 to 90.3) (2.2 to 3.4) (89.6 to 90.3)OR exp health careorganization

Best specificity, 57.0 99.7 34.3 99.5qualitative.t.w. (46.5 to 67.4) (99.6 to 99.7) (26.5 to 42.0) (99.4 to 99.6)OR qualitativestudy.tw.

Best optimization of 89.5 89.9 2.7 89.9sensitivity and (83.1 to 96.0) (89.5 to 90.2) (2.1 to 3.3) (889.5 to 90.2)specificity,interview:.tw.OR exp health careorganizationOR experiences.tw.

NOTE: : = truncation; tw = textword (word or phrase appears in title or abstract); exp = exploded subjectheading.a. Search strategies are reported using Ovid’s search engine syntax for EMBASE.b. Denominator varies by row.

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

which 86 (0.31%) were considered qualitative. The top-performing single-termstrategy for sensitivity in MEDLINE, “interview:.mp.” was the same in EMBASEand had a peak specificity greater than 97%, also consistent with our EMBASEresult. Similarly, the three-term strategies in both MEDLINE (“interview:.tw. ORpx.fs. OR exp health services administration”) and EMBASE (“interview:.tw. ORqualitative.tw. OR exp health care organization”) yielded peak sensitivities greaterthan 94%.

A logistic regression approach to developing search strategies was done whenderiving similar hedges for MEDLINE. The analysis did not improve on searchstrategies developed using the Boolean approach described above (Wong,Wilczynski, & Haynes, 2004), so we did not repeat it for EMBASE.

It should be noted that our search strategies for retrieving qualitative studies donot distinguish between more and less rigorous qualitative studies. We were unableto find an agreement on criteria for methodological merit for qualitative research, asdebate continues on appraising the merit of qualitative studies (Barbour, 2001,Morse, Barrett, Mayan, Olson, & Spiers, 2002).

Our findings leave some room for improvement in search performance, and wehope that other information retrieval researchers will find better strategies. Mean-while, these strategies were developed using the Ovid search interface for EMBASEand have been prestored by Ovid for use. First, use search terms for the topic (forexample, “grief and adjustment”), then click the “target” (“More Limits”) on thesearch screen, find the Clinical Queries box and within that box the search strategiesfor EMBASE qualitative studies; select a sensitive or specific search strategy, thenproceed with the search.

CONCLUSION

We have demonstrated that several search strategies can achieve high performancelevels when retrieving qualitative studies from EMBASE. Depending on thesearcher’s needs, high sensitivity or high specificity can be traded off to achieve thedesired search results.

COMPETING INTERESTS OF AUTHORS

None of the authors has received reimbursements, fees, funding, or salary from anorganization that might gain or lose financially from the publication of this article.We do not hold stocks or shares in any company that might benefit from the publica-tion of this article. We also do not have any other financial or nonfinancial interestsin relation to this article.

NOTE

1. The list of terms tested can be obtained from the first author, bhaynes@mcmaster.ca, on request.

Walters et al. / OPTIMAL SEARCH STRATEGIES 167

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from

REFERENCES

Barbour, R. S. (2001). Checklists for improving rigour in qualitative research: A case of the tail waggingthe dog? British Medical Journal, 322, 1115-1117.

Evans, D. (2002). Database searches for qualitative research. Journal of the Medical Library Association,90(3), 290-293.

Grant, M. J. (2004). How does your searching grow? Asurvey of search preferences and the use if optimalsearch strategies in the identification of qualitative research. Health Information and Libraries Journal,21(1), 21-32.

Green, J., & Britten, N. (1998). Qualitative research and evidence based medicine. British Medical Journal,316, 1230-1232.

Haynes, R. B., & Wilczynski, N. L. (for the Hedges Team). (2004). Optimal search strategies for retrievingscientifically strong studies of diagnosis from MEDLINE. British Medical Journal, 328, 1040.

Haynes, R. B., Wilczynski, N., McKibbon, K. A., Walker, C. J., & Sinclair, J. C. (1994). Developing optimalsearch strategies for detecting clinically sound studies in MEDLINE. Journal of the American MedicalInformatics Association, 1, 447-458.

Health Information Research Unit. (2005). The “Hedges” project. Retrieved October 18, 2005, from http://hiru.mcmaster.ca/hedges/indexhiru.htm

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishingreliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), Arti-cle 2. Retrieved October 12, 2005, from http://www.ualberta.ca/~iiqm/backissues/1_2Final/pdf/morseetal.pdf

Wilczynski, N. L., & Haynes, R. B. (2002). Robustness of empirical search strategies for clinical content inMEDLINE. Journal of the American Medical Informatics Association, 9(Symposium Suppl.), 904-908.

Wilczynski, N. L., & Haynes, R. B. (for the Hedges Team). (2003). Developing optimal search strategies fordetecting clinically sound causation studies in MEDLINE. Journal of the American Medical InformaticsAssociation, 10(Symposium Suppl.), 719-723.

Wilczynski, N. L., McKibbon, K. A., & Haynes, R. B. (2001). Enhancing retrieval of best evidence forhealth care from bibliographic databases: Calibration of the hand search of the literature. Medinfo,10(Pt 1), 390-393.

Wilczynski, N. L., Walker, C. J., McKibbon, K. A., & Haynes, R. B. (1993). Assessment of methodologicsearch filters in MEDLINE. Proceedings of the Annual Symposium on Computer Applications in MedicalCare, 1993, 601-605.

Wong, S. S. L., Wilczynski, N. L., & Haynes, R. B. (for the Hedges Team). (2004). Developing optimalsearch strategies for detecting clinically relevant qualitative studies in MEDLINE. Medinfo, 2004,311-316.

Wong, S. S. L., Wilczynski, N. L., Haynes, R. B., & Ramkissoonsingh, R. (for the Hedges Team). (2003).Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE.Journal of the American Medical Informatics Association, 10(Symposium Suppl.), 728-732.

Leslie A. Walters, B.A., is a research coordinator at McMaster University in Hamilton, Ontario,Canada.

Nancy L. Wilczynski, M.Sc., is a research associate and doctoral candidate at McMaster University inHamilton, Ontario, Canada.

R. Brian Haynes, M.D., Ph.D., is Chair of the Department of Clinical Epidemiology and Biostatisticsand Chief of the Health Information Research Unit at McMaster University in Hamilton, Ontario,Canada.

168 QUALITATIVE HEALTH RESEARCH / January 2006

at MEMORIAL UNIV OF NEWFOUNDLAND on August 3, 2014qhr.sagepub.comDownloaded from