The Effectiveness of Symptom Checkers for Self-Diagnosis ... · PDF file checkers on appropriate triage advice ranged from 33% (95% CI 19 to 48) to 78% (95% CI 64 to 91) of SP evaluations

Confidential: For Review O

nly

The Effectiveness of Symptom Checkers for Self-Diagnosis

and Triage: Beyond “Googling” Symptoms

Journal: BMJ

Manuscript ID: BMJ.2015.025489.R1

Article Type: Research

BMJ Journal: BMJ

Date Submitted by the Author: 14-May-2015

Complete List of Authors: Semigran, Hannah; Harvard Medical School, Health Care Policy Linder, Jeffrey; Brigham and Women's Hospital, Medicine; Harvard Medical School, Gidengil, Courtney; RAND Corporation, ; Boston Children's Hospital, Infectious Diseases

Mehrotra, Ateev; Harvard Medical School, Health Care Policy; Beth Israel Deaconess Medical Center, General Internal Medicine and Primary Care

Keywords: Symptom checkers, Internet, Antibiotic prescribing, mHealth

https://mc.manuscriptcentral.com/bmj

BMJ

Confidential: For Review O

nly

1

The Effectiveness of Symptom Checkers for Self-Diagnosis and Triage:

An Audit Study

Hannah L. Semigran, 1

Jeffrey A. Linder, 2 Courtney Gidengil,

3,4 Ateev Mehrotra,

1,5

1 Department of Health Care Policy, Harvard Medical School, Boston, MA

2 Division of General Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA

3 RAND Corporation, Boston, MA

4 Division of Infectious Diseases, Boston Children’s Hospital, Boston, MA

5 Division of General Internal Medicine and Primary Care, Beth Israel Deaconess Medical Center,

Boston, MA

Corresponding Author:

Ateev Mehrotra, MD, MPH

Harvard Medical School

180 Longwood Avenue

Boston, MA 02115

617-432-3905

[email protected]

Word Count: 3,689

Tables: 4

This study was funded by the United States’ National Institute of Health, (National Institute of Allergy

and Infectious Disease - Grant # R21 AI097759-01).

Contributors: Author Contributions: Study concept and design: Mehrotra, Semigran, Gidengil, Linder.

Acquisition of data: Semigran. Analysis and interpretation of data: Semigran, Mehrotra. Drafting of the

manuscript: Semigran. Critical revision of the manuscript for important intellectual content: Gidengil,

Linder, Mehrotra. Statistical analysis: Semigran, Mehrotra. Administrative, technical, and material

support: Mehrotra. Study supervision: Mehrotra

Page 1 of 45

https://mc.manuscriptcentral.com/bmj

BMJ

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Confidential: For Review O

nly

2

Competing interests: All authors are affiliated with Harvard Medical School. Harvard Medical School’s

Family Health Guide is used as the basis for one of the symptom checkers evaluated. None of the

authors have been or plan to be involved in the development, evaluation, promotion or any other facet

of Harvard Medical School-related symptom checker. All authors have completed the ICMJE uniform

disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for

the submitted work; no financial relationships with any organisations that might have an interest in the

submitted work in the previous three years; no other relationships or activities that could appear to

have influenced the submitted work.

Ethical Approval: Not required

Data Sharing: No additional data available

Transparency: The senior author (the manuscript's guarantor) affirms that the manuscript is an honest,

accurate, and transparent account of the study being reported; that no important aspects of the study

have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered)

have been explained.

Page 2 of 45

https://mc.manuscriptcentral.com/bmj

BMJ

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Confidential: For Review O

nly

3

SUMMARY:

Objectives: Use of the internet for self-diagnosis is common. Symptom checkers are online tools which

use computer algorithms to help patients with self-diagnosis and/or self-triage. Despite the growth in

use of such tools, the clinical accuracy of symptom checkers has not been assessed. Our objective was

to determine the diagnostic and triage accuracy of online symptom checkers.

Design: Audit study of web-based symptom checkers using standardized patient (SP) vignettes to assess

their clinical performance

Participants: We identified 23 symptom checkers that were in English, free, publicly-available, and

provided advice across a range of conditions. We compiled 45 standardized patient (SP) vignettes

equally divided into three categories of triage urgency: emergent care required (e.g., pulmonary

embolism), non-emergent care reasonable (e.g., otitis media), and self-care reasonable (e.g., viral upper

respiratory illness)

Main outcome measures: For symptom checkers that provided a diagnosis, our main outcomes were

whether the symptom checker listed the correct diagnosis first or within the first 20 potential diagnoses

(n = 770 SP evaluations). For symptom checkers that provided a triage recommendation, our main

outcomes were whether the symptom checker recommended emergent care, non-emergent care, or

self-care (n = 532 SP evaluations).

Results: The 23 symptom checkers provided the correct diagnosis first in 34% (95% confidence interval

[CI], 31 to 37) of SP evaluations, listed the correct diagnosis within the top 20 diagnoses given in 58%

(95% CI 55 to 62) of SP evaluations, and provided the appropriate triage advice in 57% (95% CI 52 to 61)

of SP evaluations. Triage performance varied by condition urgency, with appropriate triage advice

provided in 80% (95% CI 75 to 86) of emergent cases, 55% (95% CI 47 to 63) of non-emergent cases, and

Page 3 of 45

https://mc.manuscriptcentral.com/bmj

BMJ

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Confidential: For Review O

nly

4

33% (95% CI 26 to 40) of self-care cases (p

Confidential: For Review O

nly

5

The public is increasingly using the internet to research their health concerns. For example, the United

Kingdom’s online patient portal for national health information, NHS Choices, reports over 15 million

visits per month. 1 More than a third of adults in the United States regularly use the internet to self-

diagnose what ails them, using it for both non-urgent and urgent symptoms such as chest pain. 2 3

While

there is a wealth of online resources to learn about specific conditions, self-diagnosis usually starts with

search engines like Google, Bing, or Yahoo. 2 However, internet search engines can lead users to

confusing and sometimes unsubstantiated information and people with urgent symptoms are not

directed to seek emergent care. 3-6

Recently, there has been a proliferation of more sophisticated

programs called symptom checkers that attempt to more effectively diagnose patients and direct them

to the appropriate care setting. 3 6-13

Using computerized algorithms, symptom checkers ask users a series of questions about their symptoms

or require users to input their symptoms themselves. The algorithms vary and may use branching logic,

Bayesian inference, or other methods. Private companies and other organizations, including the

National Health Service (NHS), the American Academy of Pediatrics, and the Mayo Clinic, have launched

their own symptom checkers. One symptom checker, iTriage, reports 50 million uses per year. 14

Typically symptom checkers are accessed via websites, but some are also available as apps for smart

phones or tablets.

Symptom checkers serve two main functions: facilitating self-diagnosis and assisting with triage. The

self-diagnosis function provides a list of diagnoses usually rank ordered by likelihood. The diagnosis

function is typically framed as helping educate patients on the range of diagnoses that might fit their

symptoms. The triage function informs patients whether they should seek care at all and, if so, where

(i.e. accident & emergency department, general practitioner’s (GP) clinic) and with what urgency (i.e.

emergently or within a few days). Symptom checkers may supplement or replace telephone triage lines,

Page 5 of 45

https://mc.manuscriptcentral.com/bmj

BMJ

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Confidential: For Review O

nly

6

which are common in primary care. 15-18

To ensure the safety of medical mobile apps, the United States

Congress is considering regulation of apps