The Effectiveness of Symptom Checkers for Self-Diagnosis ... · checkers on appropriate triage advice ranged from 33% (95% CI 19 to 48) to 78% (95% CI 64 to 91) of SP evaluations

Confidential: For Review O

nly

The Effectiveness of Symptom Checkers for Self-Diagnosis

and Triage: Beyond “Googling” Symptoms

Journal: BMJ

Manuscript ID: BMJ.2015.025489.R1

Article Type: Research

BMJ Journal: BMJ

Date Submitted by the Author: 14-May-2015

Complete List of Authors: Semigran, Hannah; Harvard Medical School, Health Care Policy Linder, Jeffrey; Brigham and Women's Hospital, Medicine; Harvard Medical School, Gidengil, Courtney; RAND Corporation, ; Boston Children's Hospital, Infectious Diseases

Mehrotra, Ateev; Harvard Medical School, Health Care Policy; Beth Israel Deaconess Medical Center, General Internal Medicine and Primary Care

Keywords: Symptom checkers, Internet, Antibiotic prescribing, mHealth

https://mc.manuscriptcentral.com/bmj

BMJ


nly

1

The Effectiveness of Symptom Checkers for Self-Diagnosis and Triage:

An Audit Study

Hannah L. Semigran,1

Jeffrey A. Linder,2 Courtney Gidengil,

3,4 Ateev Mehrotra,

1,5

1 Department of Health Care Policy, Harvard Medical School, Boston, MA

2 Division of General Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA

3 RAND Corporation, Boston, MA

4 Division of Infectious Diseases, Boston Children’s Hospital, Boston, MA

5 Division of General Internal Medicine and Primary Care, Beth Israel Deaconess Medical Center,

Boston, MA

Corresponding Author:

Ateev Mehrotra, MD, MPH

Harvard Medical School

180 Longwood Avenue

Boston, MA 02115

617-432-3905

[email protected]

Word Count: 3,689

Tables: 4

This study was funded by the United States’ National Institute of Health, (National Institute of Allergy

and Infectious Disease - Grant # R21 AI097759-01).

Contributors: Author Contributions: Study concept and design: Mehrotra, Semigran, Gidengil, Linder.

Acquisition of data: Semigran. Analysis and interpretation of data: Semigran, Mehrotra. Drafting of the

manuscript: Semigran. Critical revision of the manuscript for important intellectual content: Gidengil,

Linder, Mehrotra. Statistical analysis: Semigran, Mehrotra. Administrative, technical, and material

support: Mehrotra. Study supervision: Mehrotra

Page 1 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

2

Competing interests: All authors are affiliated with Harvard Medical School. Harvard Medical School’s

Family Health Guide is used as the basis for one of the symptom checkers evaluated. None of the

authors have been or plan to be involved in the development, evaluation, promotion or any other facet

of Harvard Medical School-related symptom checker. All authors have completed the ICMJE uniform

disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for

the submitted work; no financial relationships with any organisations that might have an interest in the

submitted work in the previous three years; no other relationships or activities that could appear to

have influenced the submitted work.

Ethical Approval: Not required

Data Sharing: No additional data available

Transparency: The senior author (the manuscript's guarantor) affirms that the manuscript is an honest,

accurate, and transparent account of the study being reported; that no important aspects of the study

have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered)

have been explained.

Page 2 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

3

SUMMARY:

Objectives: Use of the internet for self-diagnosis is common. Symptom checkers are online tools which

use computer algorithms to help patients with self-diagnosis and/or self-triage. Despite the growth in

use of such tools, the clinical accuracy of symptom checkers has not been assessed. Our objective was

to determine the diagnostic and triage accuracy of online symptom checkers.

Design: Audit study of web-based symptom checkers using standardized patient (SP) vignettes to assess

their clinical performance

Participants: We identified 23 symptom checkers that were in English, free, publicly-available, and

provided advice across a range of conditions. We compiled 45 standardized patient (SP) vignettes

equally divided into three categories of triage urgency: emergent care required (e.g., pulmonary

embolism), non-emergent care reasonable (e.g., otitis media), and self-care reasonable (e.g., viral upper

respiratory illness)

Main outcome measures: For symptom checkers that provided a diagnosis, our main outcomes were

whether the symptom checker listed the correct diagnosis first or within the first 20 potential diagnoses

(n = 770 SP evaluations). For symptom checkers that provided a triage recommendation, our main

outcomes were whether the symptom checker recommended emergent care, non-emergent care, or

self-care (n = 532 SP evaluations).

Results: The 23 symptom checkers provided the correct diagnosis first in 34% (95% confidence interval

[CI], 31 to 37) of SP evaluations, listed the correct diagnosis within the top 20 diagnoses given in 58%

(95% CI 55 to 62) of SP evaluations, and provided the appropriate triage advice in 57% (95% CI 52 to 61)

of SP evaluations. Triage performance varied by condition urgency, with appropriate triage advice

provided in 80% (95% CI 75 to 86) of emergent cases, 55% (95% CI 47 to 63) of non-emergent cases, and

Page 3 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

4

33% (95% CI 26 to 40) of self-care cases (p<0.001). Performance across the 23 individual symptom

checkers on appropriate triage advice ranged from 33% (95% CI 19 to 48) to 78% (95% CI 64 to 91) of SP

evaluations.

Conclusions: Symptom checkers had deficits in both triage and diagnosis. Triage advice from symptom

checkers is generally risk-averse, encouraging users to seek care for conditions where self-care is

reasonable.

Page 4 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

5

The public is increasingly using the internet to research their health concerns. For example, the United

Kingdom’s online patient portal for national health information, NHS Choices, reports over 15 million

visits per month.1 More than a third of adults in the United States regularly use the internet to self-

diagnose what ails them, using it for both non-urgent and urgent symptoms such as chest pain.2 3

While

there is a wealth of online resources to learn about specific conditions, self-diagnosis usually starts with

search engines like Google, Bing, or Yahoo.2 However, internet search engines can lead users to

confusing and sometimes unsubstantiated information and people with urgent symptoms are not

directed to seek emergent care.3-6

Recently, there has been a proliferation of more sophisticated

programs called symptom checkers that attempt to more effectively diagnose patients and direct them

to the appropriate care setting.3 6-13

Using computerized algorithms, symptom checkers ask users a series of questions about their symptoms

or require users to input their symptoms themselves. The algorithms vary and may use branching logic,

Bayesian inference, or other methods. Private companies and other organizations, including the

National Health Service (NHS), the American Academy of Pediatrics, and the Mayo Clinic, have launched

their own symptom checkers. One symptom checker, iTriage, reports 50 million uses per year.14

Typically symptom checkers are accessed via websites, but some are also available as apps for smart

phones or tablets.

Symptom checkers serve two main functions: facilitating self-diagnosis and assisting with triage. The

self-diagnosis function provides a list of diagnoses usually rank ordered by likelihood. The diagnosis

function is typically framed as helping educate patients on the range of diagnoses that might fit their

symptoms. The triage function informs patients whether they should seek care at all and, if so, where

(i.e. accident & emergency department, general practitioner’s (GP) clinic) and with what urgency (i.e.

emergently or within a few days). Symptom checkers may supplement or replace telephone triage lines,

Page 5 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

6

which are common in primary care.15-18

To ensure the safety of medical mobile apps, the United States

Congress is considering regulation of apps that “provide a list of possible medical conditions and advice

on when to consult a health care provider.”19 20

Symptom checkers have several potential benefits. They can encourage patients with a life-threatening

problem such as stroke or heart attack to seek emergency care.21

For patients with a non-emergent

problem that does not require a medical visit, these programs can reassure patients and recommend

they stay home. For approximately a quarter of acute respiratory illness visits for common conditions

like viral upper respiratory illness, patients do not receive any intervention beyond over-the-counter

therapy,22

and over half of patients receive unnecessary antibiotics.23-25

Decreasing unnecessary visits

saves patients’ time and money, deters overprescribing of antibiotics, and may decrease GP visits – a

critical issue given GP workload in the United Kingdom increased by 62% from 1995 to 2008.17

However,

there are several key concerns. If patients with a life-threatening problem are misdiagnosed and told to

not seek care, their health could worsen, increasing morbidity and mortality. Alternatively, if patients

with minor illnesses are told to seek care, in particular in an accident & emergency department, such

programs could increase unnecessary visits and therefore result in increased time and costs for patients

and society.

The impact of symptom checkers will depend to a large degree on their clinical performance, but to our

knowledge, no previous study has systematically evaluated the diagnostic and triage accuracy of

symptom checkers. To measure the accuracy of symptom checkers’ diagnosis and triage advice, we

audited 23 symptom checkers using 45 standardized patient (SP) vignettes. The vignettes reflected a

range of conditions from common to less common and low-acuity to life-threatening.

Methods

Search strategy for symptom checkers

Page 6 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

7

To identify symptom checkers, we searched for symptoms checkers between June 2014 and November

2014 that were in English, free, publicly-available, for humans (vs. veterinary), and did not focus on a

single type of condition (e.g. only orthopedic problems). To find symptom checkers that were available

as apps in the Apple App Store and Google Play, we used two search phrases (“symptom checker”,

“medical diagnosis”) used in a recent study on symptom checkers and examined by hand the first 240

search results.12

We used 240 because it has been used in prior studies that have searched app stores.26

To find online symptom checkers, we entered the same two search phrases in Google and Google

Scholar and examined the first 300 results. In prior research, the probability of relevant search results

declines substantially after the first 300 results.27

We supplemented our searches by asking two

symptom checker developers if they knew of other competing products.

In total, we identified 143 symptom checkers. We excluded 102 symptom checkers that used the same

medical content and logic as another tool (and therefore would have identical performance) (List

provided in Appendix). We excluded 25 symptom checkers that focused only on a single class of illness

(e.g. orthopedic problems), 14 that only provided medical advice (e.g. what symptoms are typically

associated with a certain condition) and did not provide diagnosis or triage advice, and 2 that were not

working. After these exclusions, we evaluated 23 symptom checkers.

Symptom checkers’ characteristics

We categorized symptom checkers by whether they facilitated self-diagnosis, self-triage, or both; type of

organization that operated the symptom checker; the maximum number of diagnoses provided, and

whether they were based on Schmitt or Thompson nurse triage guidelines, which are decision support

protocols commonly used in telephone triage for pediatric and adult consultations, respectively.28 29

We

grouped government and health plans together because both may have a financial incentive to deter

Page 7 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

8

unnecessary visits. We provide data when available about estimated total visitors to select symptom

checkers in the Appendix.

Clinical vignettes

To evaluate symptom checkers’ diagnosis and triage performance, we used 45 SP vignettes. We used

clinical vignettes to test performance because they are a common method to test physicians and other

clinicians on their diagnostic ability and management decisions. We purposefully selected SP vignettes

from 3 categories of triage urgency: (1) 15 SP vignettes for which emergent care is required, (2) 15 SP

vignettes for which non-emergent care is reasonable, and (3) 15 SP vignettes for which a medical visit is

generally unnecessary and self-care is sufficient. We chose vignettes across the severity spectrum

because patients are using symptom checkers for symptoms that require urgent and non-urgent care.3

We included vignettes for both common and uncommon conditions because we felt the clinical

community would be particularly interested in performance for less common but potentially life

threatening problems.

The SP vignettes were identified from various clinical sources, which also provided the associated

correct diagnosis. Symptom checkers ask users for a list of symptoms or ask a series of questions. Each

SP vignette was simplified into a core set of symptoms for easy entry and in some situations, we

supplemented the data provided by the vignette because a symptom checker asked about a symptom

not addressed in the vignette (details on source, core symptoms, and supplemental symptoms for each

vignette provided in Appendix).

We categorized the 45 vignette diagnosed as either “common” or “uncommon” diagnoses based on the

prevalence of the diagnosis among ambulatory visits in the United States (full details in Appendix).30

Assessing diagnosis and triage results

Page 8 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

9

Each SP vignette was entered into each website or app, and we recorded the resulting diagnoses and

triage advice. All vignettes were entered by an author (H.S.) who does not have clinical training. A

random sample of 25 SP vignettes were entered into symptom checkers by another person without

clinical training and there was high inter-rater reliability between the two in capturing the symptom

checker’s recommendations for diagnosis and triage (Cohen’s kappa 0.90). In some cases, we could not

evaluate a vignette because some symptom checkers only focus on children or adults or the symptom

checker did not list or ask for the key symptom in vignette. To avoid penalizing these symptom checkers,

we referred to SP vignettes that successfully yielded an output as “SP evaluations.”

To assess diagnostic accuracy, we noted whether the correct diagnosis was listed first or listed at all. For

several vignettes two symptom checkers presented a large number of diagnoses (up to 99). Because

such a long list of potential diagnoses is unlikely to be useful for a patient, we considered a diagnosis to

be listed at all only if it was within the first 20 diagnoses a symptom checker provided. It is possible that

many patients will only focus on the top diagnoses listed. Therefore, we also looked at whether the

correct diagnosis was listed in the first 3 diagnoses given. We judged the diagnosis incorrect if the

symptom checker indicated that the condition could not be identified.

We categorized the triage advice into three groups: (1) Emergent, which included advice to call an

ambulance, go to the accident & emergency department, or see your GP immediately; (2) Non-

Emergent, which included advice to call your GP or primary care provider, to see your GP or primary

care provider, go to an urgent care facility, go to a specialist, go to a retail clinic, or have an e-visit; and

(3) Self-Care, which included advice to stay at home or go to a pharmacy. If multiple triage locations

were suggested (e.g. accident & emergency department or a specialist), the most urgent suggestion was

used. We chose to do so because in almost all of the cases, the most urgent triage suggestion was listed

first. If a symptom checker was unable to reach a diagnosis decision for a given SP vignette but provided

Page 9 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

10

triage advice, we still assessed the appropriateness of this triage advice. Symptom checkers that

required users to select the correct diagnosis before giving triage advice were not included in assessing

triage accuracy with the exception of iTriage, which always suggested emergent triage advice.

Patient Involvement

There was no patient involvement in this study.

Analysis

We calculated summary statistics for diagnostic accuracy and triage advice with 95% confidence

intervals based on binomial distribution using Stata/MP 13.0. Given our focus on symptom checkers as a

whole, we did not make statistical comparisons of accuracy between individual symptom checkers. We

used chi-square tests to compare the diagnosis and triage accuracy by level and urgency and by type of

symptom checker. We conducted a sensitivity analysis of triage advice excluding several symptom

checkers that always or usually recommended emergent care.

Results

Study sample

The 23 identified symptom checkers were based in the United Kingdom, United States, Netherlands, and

Poland (Table 1): 11 symptom checkers provided both diagnoses and triage advice, 8 provided only

diagnoses, and 4 provided only triage advice. The 45 SP vignettes included 26 common and 19

uncommon diagnoses. Performance was assessed on a total of 770 SP evaluations for diagnosis and 532

SP evaluations for triage. Across the symptom checkers, 10 did not ask for demographics (age and

gender).

Accuracy of diagnosis

Page 10 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

11

Overall, the correct diagnosis was listed first in 34% (95% CI 31 to 37; Table 3) of SP evaluations.

Performance varied by urgency of condition. The correct diagnosis was listed first for 24% (95% CI 19 to

30) of emergent SP evaluations, 38% (95% CI 32 to 34) of non-emergent SP evaluations, and 40% (95% CI

34 to 47) of self-care SP evaluations (p<0.001 for comparison) (Table 3). There was no difference

between symptom checkers that did and did not ask for demographic information (34% [95% CI 30 to

39] vs. 34% [95% CI 28-39], p=0.88; Table 4). However, the correct diagnosis was listed first in SP

evaluations more often for common diagnoses compared to uncommon diagnoses (38% [95% CI 34 to

43] vs. 28% [95% CI 23 to 33], p = 0.004; Table 3).

Performance varied across symptom checkers (Table 2). Listing the correct diagnosis first in SP

evaluations ranged from 5% for MEDoctor (95% CI 0 to 13) to 50% for Doc Response (95% CI 33 to 67).

Few differences were observed by symptom checker characteristics (Table 5).

Across all symptom checkers the correct diagnosis was listed in the first 3 diagnoses in 51% (95% CI 47

to 54) of SP evaluations and in the first 20 diagnoses in 58% (95% CI 55 to 62) of SP evaluations (Table 3).

Diagnostic accuracy for listing the correct diagnosis in the top 3 and top 20 was higher for self-care

conditions vs. emergent conditions and was also higher for common conditions vs. uncommon

conditions. There was no significant difference in listing the correct diagnosis in the top 20 between

symptom checkers that listed more than 11 diagnoses versus those that only listed 1-3 diagnoses (59%

[95% CI 53 to 70] vs. 53% [95% CI 46 to 59], p=0.12; Table 4]. The accuracy of listing the correct diagnosis

in the top 20 across the 23 individual symptom checkers ranged from 34% (95% CI 17 to 52) to 84% (95%

CI 73 to 95) (Table 2).

Accuracy of triage advice

Appropriate triage advice was given in 57% (95% CI 52 to 61) of SP evaluations (Table 3). Performance

on triage advice was higher for emergent care SP evaluations versus non-emergent and self-care SP

Page 11 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

12

evaluations (80% [95% CI 75 to 86] vs. 55% [95% CI 47% to 63%] vs. 33% [95% CI 26 to 40], p<0.001).

Appropriate triage advice was higher for uncommon versus common diagnoses (63% [95% CI 57 to 70]

vs. 52% [95% CI 46 to 57], p=0.01).

iTriage, Isabel, and Healthwise advised emergent care for 100% of emergent SP evaluations. iTriage,

Symcat, Symptomate and Isabel always suggested users to seek care and therefore never advised self-

care (Table 2). After excluding these five symptom checkers, appropriate triage advice was 61% [95% CI

56 to 66]). (See Supplementary Table 5)

Symptom checkers that used the Schmitt or Thompson nurse-triage protocols were more likely to

provide appropriate triage decisions than those that did not (72% [95% CI 60 to 84] vs. 55% [95% CI 50

to 59], p=0.01; Table 4). Accurate triage advice varied by operator of symptom checker (provider groups

and physician associations 68% [95% CI 58 to 77], private companies 59% [95% CI 53 to 65], health

plans/governments 43% [95% CI 34 to 51], p<0.001).

Discussion

Principal Findings

Using SP vignettes we measured the diagnostic and triage accuracy of symptom checkers. Although

there was a range of performance across symptom checkers, overall they had deficits in both diagnosis

and triage accuracy. On average, symptom checkers provided the correct diagnosis within the first 20

listed in 58% of SP evaluations with the best-performing symptom checker listing the correct diagnosis

in 84% of SP evaluations. The correct triage decision was more than two times higher for SP evaluations

requiring emergent care (80%) versus those SP evaluations for which self-care was appropriate (34%).

Otherwise we found that symptom checkers advised the appropriate level of care about half the time,

but this varied by clinical severity.

Page 12 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

13

Comparisons with other studies

Our results on diagnostic accuracy and appropriate triage are roughly similar to prior work on the

performance of single symptom checkers for a limited set of diagnoses.6-8 31

An orthopedic symptom

checker listed the correct diagnosis for knee pain 89% of the time and Boots WebMD listed the correct

diagnosis 70% of the time for ear, nose, and throat symptoms.7 8

One study that also used two common

acute SP vignettes to evaluate WebMD reported a diagnostic accuracy rate of 50%.6

Whether this level of performance for diagnosis and triage is acceptable depends on the standard for

comparison. If symptom checkers are seen as a replacement for seeing a physician, they are likely an

inferior alternative. It is believed that physicians have a diagnostic accuracy rate of 85 to 90%, though in

some studies using clinical vignettes, performance was lower.32 33

However, in-person physician visits

might be the wrong comparison because patients are likely not using symptom checkers to obtain a

definitive diagnosis but for quick and accessible guidance. Also, instead of diagnostic accuracy the key

assessment of symptom checkers may be appropriate triage. Distinguishing between Rocky Mountain

spotted fever and meningitis may be less important than ensuring patients seek emergent care.

If symptom checkers are seen as an alternative for simply entering symptoms into an online search

engine such as Google, then symptom checkers are likely a superior alternative. A recent study found

that when typing acute symptoms that would require urgent medical attention into search engines to

identify symptom-related web sites, advice to seek emergent care was present only 64% of the time.3

Perhaps the most appropriate comparison to symptom checkers is telephone triage lines, which are

widely used in developed nations.15-18

In general patients use symptom checkers and telephone triage

for similar complaints.13

Also, many nurse phone triage lines use the same underlying clinical logic as the

symptom checkers evaluated in this study. For example, some health plan nurse triage lines use the

Healthwise symptom checker, and the Schmitt and Thompson protocols were originally developed for

Page 13 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

14

phone triage and now provide the underlying logic for several symptom checkers we evaluated. The

accuracy of telephone triage recommendations, as compared to in-person physician recommendations,

range from 61% in a study of pediatric abdominal pain to 69% in a multicenter observational study.34 35

A

recent study of NHS Symptom Checkers and NHS Direct’s telephone triage line found that triage advice

from both to be comparable.9 Given their similar clinical logic and performance in terms of triage,

symptom checkers may be viewed in general as a reasonable alternative to telephone triage. One

potential advantage of symptom checkers over telephone triage is cost.17

Because of their negligible

operation costs, symptom checkers could potentially be a more cost-effective way of providing triage

advice than nurse-staffed phone lines.

Implications for using symptom checkers

Both symptom checkers and telephone triage have been promoted as a means of reducing unnecessary

office visits.15-18 36

The impact of symptom checkers on how people seek care depends on how patients

respond to the advice provided which is unknown. In one study, users expressed skepticism about the

diagnosis ultimately suggested by a symptom checker.6 The risk-averse nature of symptom checker

triage advice is a concern. In two-thirds of SP evaluations where medical attention was not necessary,

we found symptom checkers encouraged care. Overly risk-adverse advice is not limited to symptom

checkers. Telephone triage advice can also encourage unnecessary care-seeking.31 34

For instance, the

NHS’s telephone triage line, which is not staffed by health professionals, has been implicated in

increasing accident & emergency room visits in the UK.37

Some patients researching health conditions

online are motivated by fear, and the listing of concerning diagnoses by symptom checkers could

contribute to hypochondriasis and “cyberchondria,” which describes the escalated anxiety associated

with self-diagnosis on the internet.38-42

Together, confusion, risk-adverse triage advice, and

cyberchondria could mean that symptom checkers encourage patients to receive care unnecessarily and

Page 14 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

15

thus increase health care spending. Understanding how patients interpret and use the symptom checker

advice and the impact of symptom checkers on care seeking should be a key focus for future research.

The symptom checkers in this study represent the first generation of such tools, and there is a number

of potential advances that may improve their performance in future versions. Incorporating local

epidemiological data may help inform diagnoses. For instance, addition of real-time information about

the local incidence of illness in the community greatly improved the performance of a Group A

Streptococcal pharyngitis diagnostic tool.10

Diagnosis and triage rates could also be improved if

symptom checkers incorporated population or individual clinical data from medical claims or the

electronic medical record. Demographic information is critical for both diagnostic and triage decisions

for programs like symptom checkers.11

One surprising finding in our study was that symptom checkers

that asked for demographic background information did not perform better. However, it is possible that

this demographic information was not effectively incorporated into the symptom checkers’ algorithms.

Strengths and limitations of this study

Despite the growth in use of symptom checkers, we believe ours is the first to assess clinical

performance across a large number of symptom checkers and wide range of conditions.

There were key limitations to this study. We cannot be sure we identified all publicly-available symptom

checkers, despite a thorough search of relevant databases and consultation with experts in this field. We

used clinical vignettes in which the symptoms and diagnoses were typically clear, and few had comorbid

conditions, resulting in a possible overestimation of the true clinical accuracy of symptom checkers.32

We also do not have data on the clinical performance of physicians for our SP vignettes, preventing a

direct comparison between symptom checkers and physicians beyond what is available in the literature.

When symptom checkers suggested several care sites (e.g. accident & emergency department or GP

Page 15 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

16

office), our triage assessment was based only on the highest acuity site of care listed and this may

contribute to our finding that triage advice is risk-averse.

Symptom checkers are part of a larger trend of both patients and physicians using the internet for many

health care tasks and therefore it appears likely that the use of symptom checkers will only increase.

Patients are chatting online with physicians,43

emailing their doctors for medical advice,44

receiving care

via E-Visits,45 46

and downloading health apps to smartphones.47

In addition to the public, physicians and

other practitioners are also using conceptually-similar tools to aid in the diagnosis and triage of their

patients.48 49

Physicians should be aware that an increasing number of their patients are using new internet-based

tools like symptom checkers, and that the diagnosis and triage advice patients receive may often be

inaccurate. For patients, our results imply that in many cases, symptom checkers can give the user a

sense of possible diagnoses but also provide a note of caution as the tools are frequently wrong and the

triage advice might be overly cautious. However, if the alternative to using a symptom checker is not

seeking any advice or simply using the internet, there may be value in their use. Further evaluations and

monitoring of symptom checkers will be important to assess whether they help people learn more and

make better decisions about their health.

Page 16 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

17

References

1. Gann B. Giving patients choice and control: health informatics on the patient journey. Yearb Med

Inform 2012;7(1):70-3.

2. Fox S, Duggan M. Health Online 2013. Internet and American Life Project. Washington D.C.: Pew

Research Center and California Health Care Foundation, 2013:4.

3. North F, Ward WJ, Varkey P, Tulledge-Scheitel SM. Should you search the Internet for information

about your acute symptom? Telemed J E Health 2012;18(3):213-8.

4. Black P. The dangers of using Google as a diagnostic aid. British Journal of Nursing 2009;18(19):1157.

5. Zhongbo C, Turner MR. The internet for self-diagnosis and prognostication in ALS. Amyotrophic

Lateral Sclerosis 2010;11:566.

6. Luger TM, Houston TK, Suls J. Older adult experience of online diagnosis: results from a scenario-

based think-aloud protocol. Journal of Medical Internet Research 2014;16(1):e16.

7. Bisson LJ, Komm JT, Bernas GA, Fineberg MS, Marzo JM, Rauh MA, et al. Accuracy of a computer-

based diagnostic program for ambulatory patients with knee pain. Am J Sports Med

2014;42(10):2371-6.

8. Farmer SE, Bernardotto M, Singh V. How good is Internet self-diagnosis of ENT symptoms using Boots

WebMD symptom checker? Clin Otolaryngol 2011;36(5):517-8.

9. Elliot AJ, Kara EO, Loveridge P, Bawa Z, Morbey RA, Moth M, et al. Internet-based remote health self-

checker symptom data as an adjuvant to a national syndromic surveillance system. Epidemiol

Infect 2015:1-7.

10. Fine AM, Nizet V, Mandl KD. Participatory medicine: A home score for streptococcal pharyngitis

enabled by real-time biosurveillance: a cohort study. Ann Intern Med 2013;159(9):577-83.

11. DocBot: A novel clinical decision support algorithm. Engineering in Medicine and Biology Society

(EMBC), 2014 36th Annual International Conference of the IEEE; 2014 26-30 Aug. 2014.

12. Lupton D, Jutel A. 'It's like having a physician in your pocket!' A critical analysis of self-diagnosis

smartphone apps. Soc Sci Med 2015;133:128-35.

13. North F, Varkey P, Laing B, Cha SS, Tulledge-Scheitel S. Are e-health web users looking for different

symptom information than callers to triage centers? Telemed J E Health 2011;17(1):19-24.

14. Reuters. Aetna Brings New iTriage Employer Technology to Mid-Sized Businesses, 2013.

15. Lattimer V, George S, Thompson F, Thomas E, Mullee M, Turnbull J, et al. Safety and effectiveness of

nurse telephone consultation in out of hours primary care: randomised controlled trial. The

South Wiltshire Out of Hours Project (SWOOP) Group. Bmj 1998;317(7165):1054-9.

16. Bunn F, Byrne G, Kendall S. The effects of telephone consultation and triage on healthcare use and

patient satisfaction: a systematic review. Br J Gen Pract 2005;55(521):956-61.

17. Campbell J, Fletcher E, Britten N, Green C, Holt T, Lattimer V, et al. Telephone triage for management

of same-day consultation requests in general practice (the ESTEEM trial): a cluster randomised

controlled trial and cost-sequence analysis. Lancet 2014;in press.

18. Richards D, Meakins J, Tawfik J, Godfrey L, Dutton E, Richardson G, et al. Nurse telephone triage for

same day appointments in general practice: multiple interrupted time series trial of effect on

workload and costs. British Medical Journal 2002;325(7374).

19. Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer

use. Journal of Medical Internet Research 2014;16(9):e210.

20. Medical Electronic Data Technology Enhancement for Consumers' Health (MEDTECH). 2nd Session

ed, 2014.

21. Saczynski J, Yarzebski J, Lessard D, Spencer F, Gurwitz J, Gore J, et al. Trends in pre-hospital delay in

patients with acute myocardial infarction (from the Worcester Heart Attack Study). American

Journal of Cardiology 2008;102(12):1591.

Page 17 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

18

22. Barnett ML, Linder JA. Antibiotic prescribing for adults with acute bronchitis in the United States,

1996-2010. Jama 2014;311(19):2020-2.

23. Barnett ML, Linder JA. Antibiotic prescribing to adults with sore throat in the United States, 1997-

2010. JAMA Intern Med 2014;174(1):138-40.

24. Gonzales R, Malone DC, Maselli JH, Sande MA. Excessive antibiotic use for acute respiratory

infections in the United States. Clin Infect Dis 2001;33(6):757-62.

25. Little P, Rumsby K, Kelly J, Watson L, Moore M, Warner G, et al. Information leaflet and antibiotic

prescribing strategies for acute lower respiratory tract infection: a randomized controlled trial.

Jama 2005;293(24):3029-35.

26. Bierbrier R, Lo V, Wu RC. Evaluation of the accuracy of smartphone medical calculation apps. Journal

of Medical Internet Research 2014;16(2):e32.

27. Orizio G, Merla A, Schulz PJ, Gelatti U. Quality of online pharmacies and websites selling prescription

drugs: a systematic review. Journal of Medical Internet Research 2011;13(3):e74.

28. Schmitt BD. Pediatric Telephone Protocols: Office Version. Elk Grove Village IL: American Academy of

Pediatrics, 2012.

29. Thompson DA. Adult Telephone Protocols, 3rd Edition. Elk Grove Village IL: American Academy of

Pediatrics, 2013.

30. CDC, NAMCS, NHAMCS. Annual Number and Percent Distribution of Ambulatory Care Visits By

Setting Type According to Diagnosis Group: United States, 2009-2010, 2010.

31. Poote A, French D, Dale J, Powell J. A study of automated self-assessment in primary care student

health centre setting. Journal of Telemedicine and Telecare 2014;20(3):125.

32. Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf 2013;22 Suppl 2:ii21-ii27.

33. Meyer AN, Payne VL, Meeks DW, Rao R, Singh H. Physicians' diagnostic accuracy, confidence, and

resource requests: a vignette study. JAMA Intern Med 2013;173(21):1952-8.

34. Staub GM, von Overbeck J, Blozik E. Teleconsultation in children with abdominal pain: a comparison

of physician triage recommendations and an established paediatric telephone triage protocol.

BMC Med Inform Decis Mak 2013;13:110.

35. Giesen P, Ferwerda R, Tijssen R, Mokkink H, Drijver R, van den Bosch W, et al. Safety of telephone

triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Qual

Saf Health Care 2007;16(3):181-4.

36. Stacey D, Graham I, O'Connor A, Pomey M. Barriers and facilitators influencing call center nurses'

decision support for callers facing values-sensitive decisions: A mixed methods study.

Worldviews on Evidence-Based Nursing 2005;2(4).

37. Donnelly L. A&E Crisis cause by NHS 111 phoneline, senior medic suggests. The Telegraph, January

14, 2015.

38. Usborne S. Cyberchondria: The perils of internet self-diagnosis. London: The Independent, February

17, 2009.

39. Hartzband P, Groopman J. Untangling the web - Patients, doctors, and the internet. New England

Journal of Medicine 2010;362(12):1064.

40. Brigo F, Igwe SC, Ausserer H, Nardone R, Tezzon F, Bongiovanni LG, et al. Why do people Google

epilepsy? An infodemiological study of online behavior for epilepsy-related search terms.

Epilepsy Behav 2014;31:67-70.

41. White RW, Horvitz E. Experiences with web search on medical concerns and self diagnosis. AMIA

Annu Symp Proc 2009;2009:696-700.

42. Husain I, Spence D. Can healthy people benefit from health apps? Bmj 2015;350:h1887.

43. Eminovic N, Wyatt J, Tarpey A, Murray G, Ingram G. First evaluation of the NHS Direct Online Clinical

Enquiry Service: A nurse-led web chat triage service for the public. Journal of Medical Internet

Research 2004;6(2).

Page 18 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

19

44. Eysenbach G, Diepgen T. Patients looking for information on the internet and seeking teleadvice.

Archives of Dermatology 1999;135(2).

45. Mehrotra A, Paone S, Martich GD, Albert SM, Shevchik GJ. A comparison of care at e-visits and

physician office visits for sinusitis and urinary tract infection. JAMA Intern Med 2013;173(1):72-

4.

46. DeJong C, Santa J, Dudley RA. Websites that offer care over the Internet: is there an access quality

tradeoff? Jama 2014;311(13):1287-8.

47. Edney A. The FDA Sets Its Sights on Medical Apps: Bloomberg Businessweek, September 20, 2013.

48. Cook DA, Enders F, Linderbaum JA, Zwart D, Lloyd FJ. Speed and accuracy of a point of care web-

based knowledge resource for clinicians: a controlled crossover trial. Interact J Med Res

2014;3(1):e7.

49. Sadeghi S, Barzi A, Sadeghi N, King B. A Bayesian model for triage decision support. Int J Med Inform

2006;75(5):403-11.

Page 19 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

20

Symptom

Checker Description

Maximum

No. of

Diagnoses*

Triage Options Provided

AskMD (USA) Online health and wellness platform from

Sharecare

(https://www.sharecare.com/askmd/get-

started)

15 n/a

BetterMedicine Health resource from HealthGrades;

symptom checker provides possible

diagnoses and information about these

conditions

(http://www.bettermedicine.com/symptom

-checker/)

46 n/a

DocResponse Symptom checker started by group of

certified physicians; user can choose from

internal medicine, dermatology, and

orthopedic views

(http://www.docresponse.com/)

5 n/a

Doctor

Diagnose

App offered on Google Play; provides

potential diagnoses and triage advice in

some cases

3 Seek immediate care, call your

doctor now, speak with your doctor,

home care

Drugs.com

(USA)

Online resource for drug and related health

information; uses content from Harvard

Health Publications

(http://www.drugs.com/symptom-

checker/)

10 ED, primary care doctor, home care

EarlyDoc

(Netherlands)

For triage criteria, uses Dutch College of

General Practitioners (NHG) TriageWijzer

and the Australian Triage Scale (used in

Australia and New Zealand to assess

urgency). (https://www.earlydoc.com/en/)

3 Don't wait and call a doctor now,

call a doctor preferably today, see

your doctor preferably on a

weekday, your complaints don't

seem urgent

Esagil (USA) Provides list of likely diagnoses (based on

the percent of entered symptoms that are

congruent with the diagnosis); the user can

also enter blood and urine analysis results

along with symptoms (http://esagil.org/)

65 n/a

Family Doctor

(USA)

Displays flow chart to track symptoms to a

diagnosis and triage option; produced by

the American Academy of Physicians

(http://familydoctor.org/familydoctor/en/h

ealth-tools/search-by-symptom.html)

7 ER, see your doctor, home care

FreeMD (USA) Takes user through a series of questions in a

"checkup" to finish with "what might be

wrong with you" and "where to go for

care"; owned by DSHI Systems, which

provides triage decision support solutions

from emergency medicine physicians to the

US government (Dep. of Veteran Affairs)

and private sector companies; program

called TriageXpert

(http://www.freemd.com/)

3 ED, urgent care, doctor's office,

doctor e-visit, retail clinic, dentist,

home care

Table 1|Symptom checkers included in the study

Page 20 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

21

Harvard

Medical School

Family Health

Guide (USA)

From Harvard Health Publications; this tool

is available both online and in print

(published 1999), and the online tool often

refers the user to the book in order to make

a diagnosis and triage decision

(http://www.health.harvard.edu/fhg/sympt

oms/symptoms.shtml)

4 ED, GP, home care

Healthline

(USA)

Health and wellness website that licenses

content to employers, health providers, and

health plans

(http://www.healthline.com/symptom-

checker)

76 n/a

Healthwise

(USA)

Non-profit provider for health content and

patient education; symptom checker

licensed to other organizations; we

accessed using the Province of Alberta's

website

(https://myhealth.alberta.ca/health/pages/

symptom-checker.aspx)

n/a Call 911 now, seek care now, seek

care today, try home care

Healthy

Children (USA)

From the American Academy of Pediatrics;

use's Barton D. Schmitt's "Pediatric

HouseCalls Symptom Checker" triage

protocol

(http://www.healthychildren.org/English/ti

ps-tools/symptom-checker)

n/a Call 911 now, call your doctor now

(night or day), call your doctor

within 24 hours, call your doctor

during weekday office hours, parent

care at home

Isabel (UK) Created by the Isabel Medical Charity

(http://symptomchecker.isabelhealthcare.c

om/suggest_diagnoses_advanced/landing_

page)

10 Walk in care, family doctor,

emergency services

iTriage (USA) Owned by Aetna; provides clinical sites in

user's region with addresses and phone

numbers (https://www.itriagehealth.com/)

5 Emergency department, urgent

care, retail clinic, family practice,

internal medicine, specialties,

prescription medication, over the

counter medication

Mayo Clinic

(USA)

Health resource website from Mayo Clinic

(http://www.mayoclinic.org/symptom-

checker/select-symptom/itt-20009075)

20 n/a

MEDoctor

(USA)

Free differential diagnosis system from

MEDoctor, Inc.

(https://www.medoctor.com/)

3 n/a

NHS Symptom

Checkers (UK)

Available through England's National Health

Services (NHS) Choices website

(https://www.nhs.uk/symptomcheckers/pa

ges/symptoms.aspx)

n/a Emergency department, general

practitioner, home care

Steps2Care

(USA)

iPhone and Android app, provides symptom

care guides from Barton D. Schmitt's

pediatric telephone triage guidelines and

David A. Thompson's adult telephone triage

guidelines

(http://www.paramounthealthcare.com/ste

ps2care )

n/a Call 911 now, go to ER now, Call

doctor now or go to ER, Call doctor

within 24 hours, Call doctor during

office hours, self-care at home

Symcat (USA) Triage tool uses data linking specific patient 6 Primary care, retail clinic,

Page 21 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

22

*Symptom checkers that provided diagnostic advice presented a list of potential diagnoses. We

identified the maximum number of diagnoses presented across the applicable vignettes.

symptoms and physician diagnoses across

visits seen in the NAMCS survey

(http://www.symcat.com/)

emergency room, urgent care

Symptify (USA) Online self-assessment tool and other

health services including emergency contact

list, consultation list etc.

(https://symptify.com/)

9 ER, urgent care, home care,

inconclusive

Symptomate

(Poland)

Uses Bayesian network methodology and a

medical database for diagnoses

(https://symptomate.com/)

5 ER, specialist, GP

WebMD (USA) Medical reference and health care resource

website (http://symptoms.webmd.com)

99 n/a

Page 22 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only

23

Diagnosis

Triage

Symptom Checker

(n=23) Listed First Listed in Top 3 Listed in Top 20

All cases Emergent care required

Non-emergent care

reasonable Self-care reasonable

Rate* % (95%CI) Rate* % (95%CI) Rate* % (95%CI) Rate* % (95%CI) Rate* % (95%CI) Rate* % (95%CI) Rate* % (95%CI)

Ask MD 17/40 43 (26 to 59) 27/40 68 (52 to 83) 30/40 75 (61 to 89) - b

- b

- b -

b -

b -

b -

b -

b

BetterMedicine 11/45 24 (11 to 38) 13/45 29 (15 to 43) 17/45 38 (23 to 53) - b -

b -

b -

b -

b -

b -

b -

b

DocResponse 18/36 50 (33 to 67) 24/36 67 (50 to 83) 26/36 72 (57 to 88) - b -

b -

b -

b -

b -

b -

b -

b

Doctor Diagnose 16/39 41 (25 to 57) 17/39 44 (27 to 60) 18/39 46 (30 to 63) 10/16 63 (36 to 89) 8/10 80 (13 to 100) 2/3 67 (0 to 100) 0/3 0 (0 to 0)

Drugs.com 17/43 40 (24 to 55) 20/43 47 (31 to 63) 25/43 58 (43 to 74) 25/42 60 (44 to 75) 8/14 57 (27 to 87) 9/15 60 (32 to 88) 8/13 62 (31 to 92)

EarlyDoc 6/19 32 (9 to 55) 7/19 33 (9 to 57) 7/19 33 (9 to 57) 9/17 53 (26 to 79) 4/6 67 (12 to 100) 3/5 60 (0 to 100) 2/6 33 (0 to 88)

Esagil 9/44 20 (8 to 33) 15/44 34 (24 to 54) 22/44 50 (35 to 65) - b -

b -

b -

b -

b -

b -

b -

b

Family Doctor 20/43 47 (31 to 62) 24/43 56 (40 to 41) 24/43 56 (40 to 71) 22/41 54 (38 to 70) 6/12 50 (17 to 83) 7/14 50 (20 to 80) 9/15 60 (32 to 88)

FreeMD 16/44 36 (22 to 51) 20/44 45 (30 to 61) 21/44 48 (32 to 63) 26/44 59 (44 to 74) 10/15 67 (40 to 94) 13/15 87 (67 to 100) 3/14 21 (0 to 46)

HMS Family Health

Guide

13/38 34 (18 to 50) 20/38 52 (39 to 72) 21/38 55 (39 to 72) 32/40 78 (64 to 91) 13/14 93 (77 to 100) 11/13 85 (62 to 100) 8/13 62 (31 to 92)

Healthline 17/45 38 (23 to 53) 24/45 53 (37 to 68) 26/45 58 (43 to 73) - b -

b -

b -

b -

b -

b -

b -

b

Healthwise - a -

a -

a -

a -

a -

a 19/44 43 (28 to 58) 15/15 100 (100 to 100) 1/15 7 (0 to 21) 3/14 21 (0 to 46)

Healthy Children - a -

a -

a -

a -

a -

a 11/15 73 (48 to 99) 3/3 100 (100 to 100) 5/5 100 (100 to 100) 3/7 43 (0 to 92)

Isabel 20/45 44 (29 to 60) 31/45 69 (55 to 83) 38/45 84 (73 to 95) 23/45 51 (36 to 66) 15/15 100 (100 to 100) 8/15 53 (25 to 82) 0/15 0 (0 to 0)

iTriage 16/45 36 (22 to 51) 29/45 64 (49 to 78) 34/45 77 (64 to 90) 14/43 33 (19 to 48) 14/14 100 (100 to 100) 0/14 0 (0 to 0) 0/15 0 (0 to 0)

Mayo Clinic 7/41 17 (5 to 29) 24/41 59 (43 to 74) 31/41 76 (62 to 89) - b -

b -

b -

b -

b -

b -

b -

b

MEDoctor 2/37 5 (0 to 13) 16/37 43 (26 to 60) 16/37 43 (26 to 60) - b -

b -

b -

b -

b -

b -

b -

b

NHS - a -

a -

a -

a -

a -

a 23/44 52 (37 to 68) 13/15 87 (67 to 100) 3/15 20 (0 to 43) 7/14 50 (20 to 80)

Steps2Care - a -

a -

a -

a -

a -

a 30/42 71 (57 to 86) 12/14 86 (65 to 100) 10/14 71 (44 to 98) 8/14 57 (27 to 87)

Symcat 18/45 40 (25 to 55) 32/45 71 (57 to 85) 34/45 76 (62 to 89) 20/45 44 (29 to 60) 8/15 53 (25 to 82) 12/15 80 (57 to 100) 0/15 0 (0 to 0)

Symptify 13/45 29 (15 to 43) 16/45 36 (22 to 51) 20/45 44 (29 to 60) 28/40 70 (55 to 85) 11/12 92 (73 to 100) 10/14 71 (44 to 98) 7/14 50 (20 to 80)

Symptomate 10/32 31 (14 to 48) 11/32 34 (17 to 52) 11/32 34 (17 to 52) 9/14 64 (36 to 93) 7/9 76 (44 to 100) 2/3 67 (0 to 100) 0/2 0 (0 to 0)

WebMD 16/45 36 (21 to 50) 23/45 51 (36 to 66) 28/45 62 (47 to 77) - b -

b -

b -

b -

b -

b -

b -

b

Table 2|Accuracy of diagnosis decision and triage advice for each symptom checker

Abbreviations: HMS, Harvard Medical School; NHS, National Health Services

* Number of correct SP evaluations / Applicable SP evaluations. As noted in text, some SP vignettes could not be applied to a given symptom checker.

For example, we could not evaluate an adult SP vignette if it was a pediatric symptom checker. a Symptom checker does not provide diagnosis suggestions

b Symptom checker does not provide triage advice

Page 23 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


24

Diagnosis

Listed First Listed in Top 3

Listed in Top 20 Triage

No. of

vignettes

(%) Rate* %(95% CI) P value Rate* %(95% CI) P value Rate* %(95% CI) P value Rate* %(95% CI) P value

All Vignettes 45 (100) 262/770 34 (31 to 37) - 394/770 51 (47 to 54) - 449/770 58 (55 to 62) - 301/532 57 (52 to 61) -

Type of SP vignette

Emergent 15 (33) 64/263 24 (19 to 30)

<0.001

104/263 40 (34 to 46)

<0.001

132/263 50 (44 to 56)

0.003

147/183 80 (75 to 86)

< 0.001 Non-emergent 15 (33) 98/260 38 (32 to 44) 148/260 57 (51 to 63) 157/260 60 (54 to 66) 96/175 55 (47 to 63)

Self-care 15 (33) 100/247 40 (34 to 47) 142/247 57 (51 to 63) 160/247 65 (59 to 71) 58/174 33 (26 to 40)

Type of diagnosisa

Common 26 (58) 174/457 38 (34 to 43)

0.004

254/457 56 (52 to 61)

<0.001

283/457 62 (57 to 66)

0.01

162/313 52 (46 to 57)

0.01 Uncommon 19 (42) 88/313 28 (23 to 33)

140/313 45 (38 to 49)

166/313 53 (47 to 59)

139/219 63 (57 to 70)

* Number of correct SP evaluations / Applicable SP evaluations. As noted in text, some SP vignettes could not be applied to a given symptom checker.

For example, we could not evaluate an adult SP vignette if it was a pediatric symptom checker.

a Based on the annual number of ambulatory care visits in the United States, 2009-2010 (See Appendix for further description)

Table 3|Accuracy of diagnosis decision and triage advice for all symptom checkers, stratified by severity of the SP vignette and

by frequency of the condition’s diagnosis

Page 24 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


25

Diagnosis

Listed First Listed in Top 20 Triage

All Symptom

Checkers

No. of

symptom

checkers (%)

Rate* % (95% CI) P value

Rate* % (95% CI) P value

No. of

symptom

checkers (%) Rate* % (95% CI) P value

All Symptom

Checkers 23 (100)

19 (100)

262/770 34 (31 to 37) -

449/770 58 (55 to 62) -

15 (100) 301/532 57 (52 to 61) -

Uses nurse-triage books (Schmitt or Thompson)?

Yes 2 (9) 0 (0)

0 - a

- a

0 - a

- a

2 (13) 41/57 72 (60 to 84) 0.01

No 21 (91) 19 (100)

262/770 34 (31 to 37)

449/770 58 (55 to 62)

13 (87) 260/475 55 (50 to 59)

Asks demographic questions?

Yes 14 (61) 12 (63)

157/458 34 (30 to 39) 0.88

275/458 60 (56 to 65) 0.24

9 (60) 162/319 51 (45 to 56) <0.001

No 9 (39) 7 (37)

105/312 34 (28 to 39)

174/312 56 (50 to 61)

6 (40) 139/213 65 (59 to 72)

Site owner

Health plan or

government 3 (13)

1 (5)

16/44 36 (22 to 51)

0.9

34/44 77 (64 to 90)

0.01

3 (20) 56/131 43 (34 to 51)

< 0.001 Provider

group 4 (17)

5 (26)

56/167 34 (26 to 41)

104/167 62 (55 to 70)

4 (27) 65/96 68 (58 to 77)

Private

company 14 (61)

13 (68)

190/559 34 (30 to 38)

311/559 56 (52 to 60)

8 (53) 180/305 59 (53 to 65)

Maximum number of diagnoses listed

1-3 6 (32) 6 (32)

78/227 34 (28 to 41)

0.13

120/227 53 (46 to 59)

0.12

5 (33) 87/163 53 (46 to 60)

0.32 4-10 7 (37) 7 (37)

107/283 38 (32 to 43) 175/283 62 (56 to 68) 6 (40) 131/224 58 (52 to 65)

11+ 6 (32) 6 (32)

77/260 30 (24 to 35) 154/260 59 (53 to 65) 4 (27) - b -

b

* Number of correct SP evaluations / Applicable SP evaluations. As noted in text, some SP vignettes could not be applied to a given symptom

checker. For example, we could not evaluate an adult SP vignette if it was a pediatric symptom checker.

a Symptom checker does not provide diagnosis suggestions

b Symptom checker does not provide triage advice

Table 4|Accuracy of diagnosis given and triage advice for all symptom checkers given certain characteristics of the tools

Page 25 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nlyAppendix

This appendix includes more details on study sample exclusion criteria, additional methods, the

standardized patient (SP) vignettes that were used, diagnosis and triage accuracy for each symptom

checker, and the results of our sensitivity analyses.

Supplemental Table 1 categorizes the symptom checkers that were excluded from our study. After

identifying symptom checkers through the inclusion criteria described in the Methods, the symptom

checkers in this table were excluded on the basis of having the same underlying algorithm as another

tool in our sample or for other characteristics that we decided detracted from the ability of the

symptom checker to provide diagnostic and triage advice.

Our SP vignettes were gathered from several sources, which are listed in Supplemental Table 2. Each

vignette provided the age, gender, symptoms, and correct diagnosis for a given condition. This table also

notes where we added additional symptoms if the symptom checkers asked for them. Added symptoms

are italicized. The “simplified” symptoms were those inputted into each symptom checker.

To get a sense of the utilization of symptom checkers, we used Compete Pro to estimate the number of

unique visitors to symptom checker websites during the month of October 2014 in Supplemental Table

3. The limitations of this market analysis website, including its inability to track some websites outside of

the United States, those that were embedded within another website, and those with relatively low

traffic, allowed us to only estimate total use for seven symptom checkers.

Supplemental Table 4 has additional information for Table 2 in the manuscript. This includes the

accuracy of the diagnosis decision and triage advice for each symptom checker with the addition of the

stratification by the severity of the SP vignette.

Lastly, we performed sensitivity analyses shown in Supplemental Table 5 to assess the appropriateness

of the triage advice of the symptom checkers by excluding certain symptom checkers that were not as

variable in their triage advice. This includes iTriage, which always suggested that the user visit an

emergency department, and Symcat, Symptomate, and Isabel, all of which never suggest self-care.

Excluding these symptom checkers only had a modest impact on rates of appropriate triage advice.

Page 26 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nlyAdditional Methods

We stratified the performance of the symptom checkers by whether the diagnosis given by the SP

vignette was “common” or “uncommon.” We defined “common” diagnoses as those that accounted for

>0.3% of ambulatory visits (or >3,764,082 visits) in the United States in 2009-2010. These totals were

compiled from data gathered by the Center for Disease Control (CDC), the National Ambulatory Medical

Care Survey (NAMCS), and the National Hospital Ambulatory Medical Care Survey (NHAMCS).1

1. CDC, NAMCS, NHAMCS. Annual Number and Percent Distribution of

Ambulatory Care Visits By Setting Type According to Diagnosis Group:

United States, 2009-2010, 2010.

Page 27 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Supplemental Table 1: Symptom checkers excluded from study sample and reason for exclusion

Same Underlying Algorithm

Healthy Children (34) Steps2Care (28) iTriage (14)

Advocate Children’s Hospital AHN Health Finder Bayshore Community Hospital

Allied Pediatrics of New York Bon Secours Bryan Health

Children’s Medical Associates of Northern

Virginia Children’s Clinic of Raceland Crawford County Memorial Hospital

Children’s On Call College of Charleston HCA Far West

ChildrensMD Columbia St. Mary’s Inspira Health Network

ChildrensPGH East Tennessee Children’s Hospital Jersey Shore Medical Center

CIMG El Camino Hospital Meridian Health

COPA Eskenazi Health Mountainview Hospital

Docs2Go Indiana Univserity Health Ocean Medical Center

Greenwood Peditrics Intermountain health care OnPoint Urgent Care

HPN/SHL Lehigh Valley Health Network Riverside Community Hospital

Kid Aches Lourdes Hospital Riverview Medical Center

Kid Care St. Louis Children’s Mobile Middlesex Southern Ocean Medical Center

KidsDoc Mobile Nurse Sunrise Hospital and Medical Center

Lake Ray Hubbard Pediatrics Novant Health

MD 4Kids Pediatric Associate of Greater Salem NHS Symptom Checkers (6)

OU Medicine Providence Health and Services Health Direct Australia

PocketDoc Robert Wood Johnson University Hospital Martin Moth

REIS Pediatrics SCL Health NetDoctor

Sutter Health Mobile App Seton health care family NetDoctor

Swedish Kids South Texas Regional Medical Center North West Surrey

SymptomMD Spectrum Health Your.MD

UH Rainbow Babies and Children’s Hospital St. John Providence

Vanderbilt University Medical Center St. Vincent Health Healthwise (9)

Virtual Nurse UCLA Health Blue Shield of California

Page 28 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Wasatch Pediatrics Union County Hospital Group Health mobile

Wesley Kids UW Medicine Kaiser Permanente

Wesley Kids West Bloomfield Pediatrics Medical Mutual

Mercy

FreeMD (1)

Network of Care

EverydayHealth WebMD (2) Sutter Health website

MedicineNet The Hospital of Central Connecticut

Drugs.com (4) RxList University of Michigan Health System

Best Android Symptom Checker

GenieMD Isabel (2) Healthline (2)

King Abdullah bin Abdulaziz Arabic Health

Encyclopedia Patient.co.uk AARP

SmartHealth MSN Health and Fitness

Other Reasons for Exclusion

Tailored to specific condition (25) Symptom tracker (3) Medical advice only (14)

ADA Dental Symptom Checker Healee A.D.A.M. Symptom Checker

Capital Otolaryngology RheumaTrack About.com

Child Mind Institute Symple Alabama Blue Health Handbook

ColicCalm

Diagnosis And Therapy

Coping Cat Parents For pets (7) First Aid American Red Cross

Ebola Symptom Test Dog and Cat Dentist First Consult

First Aid and Symptom Checker PawNation How Stuff Works

Flu Alert Pet Education Medical Symptoms

Flu Facts PetCareRx Medical Wiz

Fortis Malar Hospital PetMD Parents.com

Hormone Balance Test WebDVM The Wellness Digest

MBH Symptom Checker ZooToo Urgent Care

MedZam Cold Flu

Xpress Urgent Care

MedZam Migraines Not working (2) Your Medial Encyclopedia

MedZam Restless Leg and Limb Dignity Health

Page 29 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


MedZam Strep Throat Saint Thomas Health

Meningitis

Myofascial Therapy.org Talk to a doctor (4)

Neocate Amwell

Pregnancy Test Doctor on Demand

Presbyterian/St. Luke’s MD Live

Shingles Symptom Checker RelyMD

SportsInjuryClinic

Trigger Point Products

USF Health

Page 30 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Diagnosis Vignette Simplified (added symptoms)

Acute liver failure¹ A 48-year-old woman with a history of migraine headaches presents to the emergency room with altered mental

status over the last several hours. She was found by her husband, earlier in the day, to be acutely disoriented and

increasingly somnolent. On physical examination, she has scleral icterus, mild right upper quadrant tenderness, and

asterixis. Preliminary laboratory studies are notable for a serum ALT of 6498 units/L, total bilirubin of 5.6 mg/dL, and

INR of 6.8. Her husband reports that she has consistently been taking pain medications and started taking additional

500 mg acetaminophen pills several days ago for lower back pain. Further history reveals a medication list with

multiple acetaminophen-containing preparations.

48 y/o f, confusion,

disorientation, increasingly

drowsy, mild right upper

quadrant tenderness, chronic

tylenol/acetaminophen -

recently took more

Appendicitis¹ A 12-year-old girl presents with sudden-onset severe generalized abdominal pain associated with nausea, vomiting,

and diarrhea. On exam she appears ill and has a temperature of 104°F (40°C). Her abdomen is tense with generalized

tenderness and guarding. No bowel sounds are present.

12 y/o f, sudden onset severe

abdominal pain, nausea,

vomiting, diarrhea, T=104

Asthma¹ A 27-year-old woman with a history of moderate persistent asthma presents to the emergency room with

progressive worsening of shortness of breath, wheezing, and cough over 3 days. She reports prior exposure to a

person who had a runny nose and a hacking cough. She did not receive significant relief from her rescue inhaler with

worsening symptoms, despite increased use. She has been compliant with her maintenance asthma regimen, which

consists of an inhaled corticosteroid and a leukotriene receptor antagonist for maintenance therapy and albuterol as

rescue therapy. Her cough is disrupting her sleep pattern and as a consequence she is experiencing daytime

somnolence, which is affecting her job performance.

27 y/o f, Hx of asthma, mild

shortness of breath,

wheezing, 3 days cough,

symptoms not responsive to

inhalers, recent cold

COPD flare (more severe)¹A 67-year-old woman with a history of COPD presents with 3 days of worsening dyspnea and increased frequency

of coughing. Her cough is now productive of green, purulent sputum. The patient has a 100-pack-year history of

smoking. She has had intermittent, low-grade fever of 100°F (37.7°C) for the past 3 days and her appetite is poor.

She has required increased use of rescue bronchodilator therapy in addition to her maintenance medications to

control symptoms.

67 y/o f, Hx of COPD, 3 days

worsening shortness of

breath, increase coughing,

green sputum, low grade

fever, increase use of rescue

bronchodilator therapy

Deep vein

thrombosis¹

A 65-year-old woman presents with unilateral leg pain and swelling of 5 days' duration. There is a history of

hypertension, mild CHF, and recent hospitalization for pneumonia. She had been recuperating at home but on

beginning to mobilize and walk, the right leg became painful, tender, and swollen. On examination, the right calf is 4

cm greater in circumference than the left when measured 10 cm below the tibial tuberosity. Superficial veins in the

leg are more dilated on the right foot and the right leg is slightly redder than the left. There is some tenderness on

palpation in the popliteal fossa behind the knee.

65 y/o f, 5 days swelling, pain

in one leg, recent

hospitalization, leg painful,

tender, swollen, red

Requires emergent care (n=15)

Supplemental Table 2: The 45 SP vignettes used to judge the symptom checkers’ accuracy and their condensed formats

Page 31 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Heart Attack² Mr. Y is a 64 year old Chinese male who presents with chest pain for 24 hours. One day prior to presentation, the

patient began to experience 8/10, non-radiating substernal chest pressure associated with diaphoresis and

shortness of breath. The pain intially improved with Tylenol, however over the following 24 hours, his symptoms

worsened. The patient went to his primary physician, where an EKG was performed which showed ST elevation in

leads V2-V6.

64 y/o m, 1 day chest pain

(8/10), non-radiating

substernal chest pressure,

sweating, shortness of

breath, (chest tightness )

Hemolytic uremic

syndrome¹

A 4-year-old boy presents with a 7-day history of abdominal pain and watery diarrhea that became bloody after the

first day. Three days before the onset of symptoms, he had visited the county fair with his family and had eaten a

hamburger. Physical examination reveals a mild anemia

4 y/o m, 7 day Hx of

abdominal pain, bloody

diarrhea, ate hamburger at

fair 3 days ago

Kidney stones¹ A 45-year-old white man presents to the emergency department with a 1-hour history of sudden onset of left-sided

flank pain radiating down toward his groin. The patient is writhing in pain, which is unrelieved by position. He also

complains of nausea and vomiting.

45 y/o m, 1 hour severe left-

sided flank pain radiating into

groin, nausea, vomiting, pain

unrelieved by position

Malaria¹ A 28-year-old man presents to his physician with a 5-day history of fever, chills, and rigors, not improving with

acetaminophen (paracetamol), along with diarrhea. He had been traveling in Central America for 3 months,

returning 8 weeks ago. He had been bitten by mosquitoes on multiple occasions, and although he initially took

malaria prophylaxis, he discontinued it due to mild nausea. He does not know the specifics of his prophylactic

therapy. On examination he has a temperature of 100.4°F (38°C), and is mildly tachycardic with a BP of 126/82

mmHg. The remainder of the examination is normal.

28 y/o m, 5 day Hx of fever,

chills, rigors, diarrhea, recent

travel abroad to area with

malaria, bitten by

mosquitoes, did not take

malaria prophylaxis

consistently

Meningitis¹ An 18-year-old male student presents with severe headache and fever that he has had for 3 days. Examination

reveals fever, photophobia, and neck stiffness.

18 y/o m, 3 days severe

headache, fever,

photophobia, neck stiffness

Pneumonia³ A 65-year-old man with hypertension and degenerative joint disease presents to the emergency department with a

three-day history of a productive cough and fever. He has a temperature of 38.3°C (101°F), a blood pressure of

144/92 mm Hg, a respiratory rate of 22 breaths per minute, a heart rate of 90 beats per minute, and oxygen

saturation of 92 percent while breathing room air. Physical examination reveals only crackles and egophony in the

right lower lung field. The white-cell count is 14,000 per cubic millimeter, and the results of routine chemical tests

are normal. A chest radiograph shows an infiltrate in the right lower lobe.

65 y/o m, Hx of hypertension

and degenerative joint

disease, 3 day Hx of

productive cough and fever

(101)

Page 32 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Pulmonary embolism¹ A 65-year-old man presents to the emergency department with acute onset of SOB of 30 minutes' duration. Initially,

he felt faint but did not lose consciousness. He is complaining of left-sided chest pain that worsens on deep

inspiration. He has no history of cardiopulmonary disease. A week ago he underwent a total left hip replacement

and, following discharge, was on bed rest for 3 days due to poorly controlled pain. He subsequently noticed swelling

in his left calf, which is tender on examination. His current vital signs reveal a fever of 100.4°F (38.0°C), heart rate

112 bpm, BP 95/65, and an O2 saturation on room air of 91%.

65 y/o m, shortness of breath

for 30 min, chest pain that

worsens with inspiration,

recent surgery, recent bed

rest, swelling in left calf,

which is tender, fever

Rocky Mountain

Spotted Fever4

An 8-year-old boy in Oklahoma is brought to the emergency department over the fourth of July weekend because of

fever, chills, malaise, athralgias, and a headache. Physical examination reveals a maculopapular rash that is most

prominent on his wrists and ankles.

8 y/o m, Fever, chills, joint

pain, headache, rash

wrists/ankles

Stroke¹ A 70-year-old man with a history of chronic HTN and atrial fibrillation is witnessed by a family member to have

nausea, vomiting, and right-sided weakness, as well as difficulty speaking and comprehending language. The

symptoms started with only mild slurred speech before progressing over several minutes to severe aphasia and

right arm paralysis. The patient is taking warfarin.

70 y/o m, nausea, vomiting,

right-sided weakness, rt arm

paralysis, difficulty speaking

and comprehension

Tetanus¹ A 63-year-old man sustained a cut on his hand while gardening. His immunization history is significant for not having

received a complete tetanus immunization schedule. He presents with signs of generalized tetanus with trismus

("lock jaw"), which results in a grimace described as "risus sardonicus" (sardonic smile). Intermittent tonic

contraction of his skeletal muscles causes intensely painful spasms, which last for minutes, during which he retains

consciousness. The spasms are triggered by external (noise, light, drafts, physical contact) or internal stimuli, and as

a result he is at the risk of sustaining fractures or developing rhabdomyolysis. The tetanic spasms also produce

opisthotonus, board-like abdominal wall rigidity, dysphagia, and apneic periods due to contraction of the thoracic

muscles and/or glottal or pharyngeal muscles. During a generalized spasm the patient arches his back, extends his

legs, flexes his arms in abduction, and clenches his fists. Apnea results during some of the spasms. Autonomic

overactivity initially manifests as irritability, restlessness, sweating, and tachycardia. Several days later this may

present as hyperpyrexia, cardiac arrhythmias, labile hypertension, or hypotension.

65 y/o m, cannot open

mouth, contraction of

muscles causing painful

spasms for minutes,

sweating, tachycardia, cut

hand while gardening, did not

get tetanus shot

Acute otitis media¹ An 18-month-old toddler presents with 1 week of rhinorrhea, cough, and congestion. Her parents report she is

irritable, sleeping restlessly, and not eating well. Overnight she developed a fever. She attends day care and both

parents smoke. On examination signs are found consistent with a viral respiratory infection including rhinorrhea and

congestion. The toddler appears irritable and apprehensive and has a fever. Otoscopy reveals a bulging,

erythematous tympanic membrane and absent landmarks.

18 mo f, 1 week rhinorrhea,

cough, congestion, irritable,

lack of appetite, fever, in

daycare

Requires non-emergent care (n=15)

Page 33 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Acute pharyngitis¹ A 7-year-old girl presents with abrupt onset of fever, nausea, vomiting, and sore throat. The child denies cough,

rhinorrhea, or nasal congestion. On physical exam, oral temperature is 101°F (38.5°C) and there is an exudative

pharyngitis, with enlarged cervical lymph nodes. A rapid antigen test is positive for group A Streptococcus (GAS).

7 y/o f, fever (101), nausea,

vomiting, sore throat, swollen

lymph nodes, tonsilar

exudate; no cough,

rhinorrhea, or nasal

congestion

Acute pharyngitis5 Mr. A is a 24 year-old man who presents to your office for complaints of sore throat, fever, and headache. His

symptoms started 2 days ago with acute onset of sore throat and fever to 102.2. He has had no cough. His physical

examination is normal, except for the presence of tonsillar exudates and some tender anterior cervical

lymphadenopathy. He is otherwise in good health, and is on no medications except for ibuprofen for fever. He has

no drug allergies. (, Centor score = 4 – treat, or test and treat)

24 y/o m, sore throat, fever

(102.2), headache, no

cough,tonsilar exudates

Acute sinusitis5 Mrs. S is a 35 year-old woman who presents with 15 days of nasal congestion. She has had facial pain and green

nasal discharge for the last 12 days. She has had no fever. On physical examination, she has no fever and the only

abnormal finding is maxillary tenderness on palpation. She is otherwise healthy, except for mild obesity. She is on no

medications, except for an over-the-counter decongestant. She has no drug allergies

35 y/o f, sx for 15 days, nasal

congestion, facial pain, green

nasal discharge, no fever

Back pain6 Consider a 35-year-old man who developed low back pain after shoveling snow 3 weeks ago. He presents to the

office for an evaluation. On examination there is a new left foot drop. In study 82% physicians recommend MRI

(sciatica/sprain)

35 y/o m, back pain following

shoveling, left foot drop,

symptoms 3 weeks of

duration (loss of sensation in

foot)

Cellulitis¹ A 45-year-old man presents with acute onset of pain and redness of the skin of his lower extremity. Low-grade fever

is present and the pretibial area is erythematous, edematous, and tender.

45 y/o m, pain and redness of

skin, low grade fever,

redness, edema, and

tenderness lower leg

COPD flare (milder)¹ A 56-year-old woman with a history of smoking presents to her primary care physician with shortness of breath and

cough for several days. Her symptoms began 3 days ago with rhinorrhea. She reports a chronic morning cough

productive of white sputum, which has increased over the past 2 days. She has had similar episodes each winter for

the past 4 years. She has smoked 1 to 2 packs of cigarettes per day for 40 years and continues to smoke. She denies

hemoptysis, chills, or weight loss and has not received any relief from over-the-counter cough preparations.

56 y/o f, Hx of smoking,

shortness of breath and

cough for several days,

rhinorrhea 3 days ago, white

sputum, no chills

Influenza¹ A 30-year-old woman presents in January with 2-day history of fever, cough, headache, and generalized weakness.

She was in her usual state of health before an abrupt onset of these symptoms. A few viral illnesses have affected

her during the current winter, but not to this severity. She reports sick contacts at work and did not receive the

seasonal influenza vaccine this season.

30 y/o f, 2 day fever, cough,

headache, weakness, did not

get flu shot

Page 34 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Mononucleosis¹ A 16-year-old female high school student presents with complaints of fever, sore throat, and fatigue. She started

feeling sick 1 week ago. Her symptoms are gradually getting worse, and she has difficulty swallowing. She has had a

fever every day, and she could hardly get out of bed this morning. She does not remember being exposed to

anybody with a similar illness recently. On physical examination she is febrile and looks sick. Enlarged cervical lymph

nodes, exudative pharyngitis with soft palate petechiae and faint erythematous macular rash on the trunk and arms

are found.

16 y/o f, 1 week Hx of fever,

sore throat, fatigue, difficulty

swallowing, fever, enlarged

lymph nodes, exudates,

macular rash on trunk/arms

Peptic Ulcer Disease¹ A 40-year-old man presents to his primary care physician with a 2-month history of intermittent upper abdominal

pain. He describes the pain as a dull, gnawing ache. The pain sometimes wakes him at night, is relieved by food and

drinking milk, and is helped partially by ranitidine. He had a similar but milder episode about 5 years ago, which was

treated with omeprazole. Physical examination reveals a fit, apparently healthy man in no distress. The only

abnormal finding is mild epigastric tenderness on palpation of the abdomen.

40 y/o m, 2 month Hx of

intermittent upper abdominal

pain, dulling and gnawing

ache, wakes at night and is

relieved by food/drinking

milk/ranitidine, prior episode

5 yrs ago

Pneumonia¹ A 6-year-old boy with a medical history significant for mild persistent asthma is brought to the clinic by his mother

with a history of a 5-day cough. His mother reports that the child's fever continues to be elevated despite

acetaminophen therapy. He has missed school for the past 3 days and he has a classmate sick with pneumonia. The

mother reports that the appetite is good for the child. His cough produced yellowish sputum at home. His vitals at

the clinic are: respiratory rate 19 breaths/min, heart rate 80 beats/min, and temperature 101.6°F (38.7°C). He

appears in no respiratory distress. His lung examination reveals bilateral rales and occasional wheeze. CXR reveals

lobar infiltrates without pleural effusions.

6 y/o m, Hx of asthma, 5 days

cough, fever, appetite good,

yellow sputum, t 101.6

Salmonella¹ A 14-year-old boy presents with nausea, vomiting, and diarrhea. Eighteen hours earlier, he had been at a picnic

where he ingested undercooked chicken along with a variety of other foods. He reports moderate-volume,

nonbloody stools occurring 6 times a day. He has mild abdominal cramps and a low-grade fever. He is evaluated at

an acute care clinic and found to be mildly tachycardic (heart rate 105 bpm) with a normal BP and a low-grade

temperature of 100.1°F (37.8°C). His physical exam is unremarkable except for mild diffuse abdominal tenderness

and mild increased bowel sounds. He is able to take oral fluids and is instructed on the appropriate oral fluid and

electrolyte rehydration.

14 y/o m, nausea, vomiting,

non-bloody diarrhea, mild

abdominal cramps (T=100.1),

mild abdominal tenderness,

diarrhea after attending a

picnic and eating

undercooked chicken,

Shingles¹ A 77-year-old man reports a 5-day history of burning and aching pain on the right side of his chest. This is followed

by the development of erythema and a maculopapular rash in this painful area, accompanied by headache and

malaise. The rash progressed to develop clusters of clear vesicles for 3 to 5 days, evolving through stages of

pustulation, ulceration, and crusting.

77 y/o m, 5 day burning and

aching on right side of chest,

erythema, maculopapular

rash, headache, malaise, rash

progressed to clear vesicles

after 3-5 days

Page 35 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Urinary tract

infection¹

A 26-year-old female newly wed presents complaining of painful urination, feeling of urgent need to urinate, and

more frequent urination for 2 days. She denies any fever, chills, nausea, vomiting, back pain, vaginal discharge, or

vaginal pruritus.

26 y/o f, painful urination,

urgent need to urinate, more

frequent urination for 2 days,

sexually active; no fever,

chills, nausea, vomiting, back

pain, vaginal discharge,

vaginal pruritus

Vertigo¹ A 65-year-old woman presents with a chief complaint of dizziness. She describes it as a sudden and severe spinning

sensation precipitated by rolling over in bed onto her right side. Symptoms typically last <30 seconds. They have

occurred nightly over the last month and occasionally during the day when she tilts her head back to look upward.

She describes no precipitating event prior to onset and no associated hearing loss, tinnitus, or other neurologic

symptoms. Otologic and neurologic examinations are normal except for the Dix-Hallpike maneuver, which is negative

on the left but strongly positive on the right side.

65 y/o f, dizziness, sudden

onset, recurrent, lasts <30

sec, consistent trigger, no

hearing loss, ringing in ears,

muscle weakness, loss of

sensation

Acute bronchitis¹ A 34-year-old woman with no known underlying lung disease 12-day history of cough. She initially had nasal

congestion and a mild sore throat, but now her symptoms are all related to a productive cough without paroxysms.

She denies any sick contacts. On physical examination she is not in respiratory distress and is afebrile with normal

vital signs. No signs of URI are noted. Scattered wheezes are present diffusely on lung auscultation.

34 y/o f, 12 day cough, initial

nasal congestion and sore

throat, cough, no fever

Acute bronchitis5 Mrs. L is a 61 year-old woman who presents with 4 days of a cough productive of yellow sputum. Her symptoms

started 4 days ago with rhinorrhea and productive cough. She initially had fevers as high as 101 for 2 days, but those

have now resolved. In the office, she has normal vital signs and a normal physical examination. She is otherwise

healthy except for high cholesterol for which she is being treated with atorvastatin. She has no drug allergies.

61 y/o f, 4 day cough, yellow

sputum, rhinorrhea, fever

(resolved)

Acute conjunctivitis¹ A 14-year-old boy with no significant past medical history presents 3 days after developing a red, irritated right eye

that spread to the left eye today. He has watery discharge from both eyes and they are stuck shut in the morning. He

reports recent upper respiratory symptoms and that several children at his day camp recently had pink eye. He

denies significant pain or light sensitivity and does not wear contact lenses. On examination, his pupils are equal and

reactive and he has a right-sided, tender preauricular lymph node. Penlight examination does not reveal any corneal

opacity.

14 y/o m, 3 days red, irritated

eye (spread from right to

left), discharge, URI

symptoms, no pain or light

sensitivity

Acute pharyngitis5 Mr. E is a 26 year-old man who presents to your office for complaints of sore throat, headache, and non-productive

cough. His symptoms started 2 days ago with acute onset of sore throat. He has been afebrile. His physical

examination is normal, except for some pharyngeal erythema. He is otherwise in good health, and is on no

medications except for acetaminophen for his sore throat and fever. He has no drug allergies.

26 y/o m, 2 day sore throat,

headache, cough, no fever

Self-care appropriate (n=15)

Page 36 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Allergic rhinitis¹ A 22-year-old student presents with a 5-year history of worsening nasal congestion, sneezing, and nasal itching.

Symptoms are year-round but worse during the spring season. On further questioning it is revealed that he has

significant eye itching, redness, and tearing as well as palate and throat itching during the spring season. He

remembers that his mother told him at some point that he used to have eczema in infancy.

22 y/o m, 5 year Hx of nasal

congestion, sneezing, nasal

itching worse during spring

season, eye itching, redness,

tearing, palate and throat

itching, Hx of eczema in

infancy

Back pain¹ A 38-year-old man with no significant history of back pain developed acute LBP when lifting boxes 2 weeks ago. The

pain is aching in nature, located in the left lumbar area, and associated with spasms. He describes previous similar

episodes several years ago, which resolved without seeing a doctor. He denies any leg pain or weakness. He also

denies fevers, chills, weight loss, and recent infections. Over-the-counter ibuprofen has helped somewhat, but he

has taken it only twice a day for the past 3 days because he does not want to become dependent on painkillers. On

examination, there is decreased lumbar flexion and extension secondary to pain, but a neurologic exam is

unremarkable.

38 y/o m, acute low back pain

after lifting, no leg pain or

weakness, no fevers, chills,

weight loss, or recent

infections

Bee sting without

anaphylaxis¹

A 9-year-old boy is brought to the ER after being stung by a bee at a picnic. He is crying hysterically. After 15 minutes

of calming him down, exam reveals a swollen tender upper lip but no tongue swelling, no drooling, no stridor, no

rash, and no other complaints.

9 y/o m, bee sting, swollen

and tender upper lip; no

tongue swelling, drooling,

stridor, rash, or other

complaints

Canker sore¹ A 17-year-old male student presents with recurrent mouth ulceration since his early schooldays. He has no

respiratory, anogenital, gastrointestinal, eye, or skin lesions. His mother had a similar history as a teenager. The

social history includes no tobacco use and virtually no alcohol consumption. He has no history of recent drug or

medication ingestion. Extraoral exam reveals no significant abnormalities and specifically no pyrexia; no cervical

lymph node enlargement; nor cranial nerve, salivary, or temporomandibular joint abnormalities. Oral exam reveals a

well-restored dentition and there is no clinical evidence of periodontal-attachment loss or pocketing. He has five 4

mm round ulcers with inflammatory haloes in his buccal mucosae.

17 y/o m with recurrent

mouth ulceration for year, no

respiratory, anogenital,

gastrointestinal, eye, or skin

lesions, mother has similar

Hx, no Hx of recent drugs or

medication

Candidal yeast

infection6

Consider a 40-year-old, monogamous, married woman who calls to report a 2-day history of vaginal itching and

thick white discharge. She has no abdominal pain or fever. (in study 50% recommended physician visit)

40 y/o f, 2 day vaginal itching,

thick white discharge, no

abdominal pain or fever

Page 37 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Constipation¹ A 5-month-old baby boy presents with difficulty and delay in passing hard stools. His mother reports that he strains

for several hours and may even miss a day, before passing stool with screaming and occasional spots of fresh blood

on the stool or diaper. He has recently been weaned from breastfeeding to cows' milk formula, which he had been

reluctant to drink initially. The child is thriving and now feeding normally. There was no neonatal delay in defecation

and no history of excessive vomiting or abdominal distension.

5 mo m, difficulty/delay in

passing hard stools, strains

for hours, may miss a day,

screams when passes stool

and occasional spots of

blood, weaned from

breastmilk to cows' milk, now

feeding normally

Eczema¹ A 12-year-old female presents with dry, itchy skin that involves the flexures in front of her elbows, behind her knees,

and in front of her ankles. Her cheeks also have patches of dry, scaly skin. She has symptoms of hay fever and has

recently been diagnosed with egg and milk allergy. She has a brother with asthma and an uncle and several cousins

who have been diagnosed with eczema.

12 y/o f, dry, itchy skin in

front of elbows, behind

knees, in front of ankles,

cheeks have patches of dry,

scaly skin, symptoms of hay

fever, egg and milk allergy,

brother has asthma and uncle

and cousins have eczema

Stye¹ A 30-year-old man presents with a painful, swollen right eye for the past day. He reports minor pain on palpation of

the eyelid and denies any history of trauma, crusting, or change in vision. He has no history of allergies or any eye

conditions and denies the use of any new soaps, lotions, or creams. On exam, he has localized tenderness to

palpation and erythema on the midline of the lower eyelid near the lid margin. The remainder of the physical exam,

including the globe, is normal.

30 y/o m, painful, swollen

right eye for past day, no Hx

of trauma, crusting, change in

vision, allergies, or eye

conditions, localized

tenderness, erythema

(redness)

Viral upper respiratory illnessMr. R. is a 56 year-old man who presents to you with 6 days of non-productive cough, nasal congestion, and green

nasal discharge. He has had intermittent fevers as high as 100.8. His physical examination is normal except for

rhinorrhea. He is otherwise healthy, except for chronic osteoarthritis of the right knee. He has no drug allergies.

56 y/o m, 6 day cough, nasal

congestion, green nasal

discharge, fever (100.8),

rhinorrhea

Page 38 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Viral upper respiratory illness¹A 30-year-old man presents with a 2-day history of runny nose and sore throat. He feels hot and sweaty, has a mild

headache, is coughing up clear sputum and complains of muscle aches. He would like antibiotics as he was

prescribed them last year when he had a similar condition. On examination, he is afebrile, has a normal pulse, a

slightly inflamed pharynx and nontender cervical lymphadenopathy. There is no neck stiffness and his chest is clear.

He has tried over-the-counter cough medications, but has not found these helpful. He smokes 10 cigarettes per day.

30 y/o m, 2 day HX of runny

nose, sore throat, hot,

sweaty, mild headache, cough

with clear sputum, muscle

aches, no fever or neck

stiffness

Vomiting7 Elizabeth’s 2-year-old son has a fever and vomited twice. Elizabeth worries about dehydration, so she gives Jack a

sippy cup of apple juice. He immediately vomits up the juice. Elizabeth debates what to do next. Should she try to

reach Jack’s pediatrician or should she take Jack to the ED? Instead, she calls her triage nurse line. Temperature =

100.5

2 y/o m, low grade fever (T =

100.5), vomited twice, vomits

up juice

Table References

1. Epocrates. https://online.epocrates.com/noFrame/.

2. Lue J. NYU Medical Grand Rounds Clinical Vignette. 2012;

http://www.medicine.med.nyu.edu/education/im-residency-homepage/research-

opportunities/clinical-vignettes. Accessed September 8, 2014.

3. Halm EA, Teirstein AS. Clinical practice. Management of community-acquired pneumonia. N Engl J

Med. Dec 19 2002;347(25):2039-2045.

4. Plantz SH, Adler JN, eds. NMS Emergency Medicine. Baltimore: Williams & Wilkins; 1998. National

Medical Series for Independent Study.

5. Gidengil CA, Linder J, Beach S, Setodjian C, Hunter G, Mehrotra A. Using clinical vignettes to predict

antibiotic prescribing for acute respiratory infections. In review.

6. Sirovich BE, Gottlieb DJ, Welch HG, Fisher ES. Variation in the tendency of primary care physicians

to intervene. Arch Intern Med. Oct 24 2005;165(19):2252-2256.

7. Boroughs DS, Dougherty JA, Goldsmith C. Telephone Triage: Help Is Just a Call Away.

http://ce.nurse.com/RVignette.aspx?TopicId=718. Accessed September 10, 2014.

Page 39 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Supplemental Table 3: Estimation of unique visitors to symptom checker websites in October 2014 using Compete Pro.¹

Symptom Checker Number of Unique Visitors

AskMD/Sharecare 4,220

EarlyDoc 16,826

FreeMD 37,545

Isabel 10,517

iTriage 198,398

Symcat 2,889

WebMD 975,127

Total 1,246,071

¹Compete. Site Comparison, 2014.

Page 40 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Do

cRe

spo

nse

Fa

mil

y D

oct

or

Isa

be

l

Ask

MD

/

Sh

are

care

Do

cto

r D

iag

no

se

Dru

gs.

com

Sy

mca

t

He

alt

hli

ne

Fre

eM

D

iTri

ag

e

We

bM

D

HM

S F

am

ily

He

alt

h G

uid

e

Ea

rly

Do

c

Sy

mp

tom

ate

Sy

mp

tify

Be

tte

r M

ed

icin

e

Esa

gil

Ma

yo

Cli

nic

ME

Do

cto

r

HM

S F

am

ily

He

alt

h G

uid

e

He

alt

hy

Ch

ild

ren

Ste

ps2

Ca

re

Sy

mp

tify

Sy

mp

tom

ate

Do

cto

r D

iag

no

se

Dru

gs.

com

Fre

eM

D

Fa

mil

y D

oct

or

Ea

rly

Do

c

NH

S

Isa

be

l

Sy

mca

t

He

alt

hw

ise

iTri

ag

e

Standardized Patient Vignette Emergent Care Non Emergent Care Self Care

Require emergent care

Acute liver failure 〇〇〇〇 U

Appendicitis 〇 ⊗ ⊗ 〇〇〇〇 ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ▨Asthma attack ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ 〇〇〇 ⊗ ⊗ ⊗

COPD more severe ⊗ 〇〇 ⊗ 〇 ⊗ ▨Deep Vein Thrombosis ⊗ ⊗ ⊗ ⊗ ⊗ 〇〇 ⊗ ⊗ 〇

Heart attack 〇 ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗

Hemolytic Uremic Syndrome ⊗ ⊗ 〇〇 U

Kidney Stones 〇 ⊗ ⊗ 〇 ⊗ ⊗ ⊗ 〇〇〇 ⊗ ▨Malaria 〇〇 N N

Meningitis ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ N

Pneumonia (more severe) 〇 ⊗ 〇〇 ⊗ 〇 ⊗ 〇〇〇 ▨Pulmonary Embolism ⊗ 〇 ⊗ ⊗ 〇〇〇 ⊗ ⊗ ⊗ ⊗ ▨Rocky Mountain Spotted

Fever⊗

Stroke ⊗ 〇 ⊗ 〇〇 ⊗ ▨Tetanus ⊗ N

Process couldn't be

started (ex. Too young)

Incorrect

diagnosis

Diagnosis given Triage Advice Given

Symptom Checker

Correct diangosis

listed in top 20

Correct

diagnosis

listed first

Supplemental Table 4: Diagnosis given and triage advice from each symptom checker, stratified by severity of the standardized patient

(SP) vignette. The symptom checkers are organized by the accuracy of listing the correct diagnosis first and the rate of appropriate triage

in descending order from left to right under each respective heading.

Diagnostic Accuracy Rate of appropriate triage

Page 41 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Do

cRe

spo

nse

Fa

mil

y D

oct

or

Isa

be

l

Ask

MD

/

Sh

are

care

Do

cto

r D

iag

no

se

Dru

gs.

com

Sy

mca

t

He

alt

hli

ne

Fre

eM

D

iTri

ag

e

We

bM

D

HM

S F

am

ily

He

alt

h G

uid

e

Ea

rly

Do

c

Sy

mp

tom

ate

Sy

mp

tify

Be

tte

r M

ed

icin

e

Esa

gil

Ma

yo

Cli

nic

ME

Do

cto

r

HM

S F

am

ily

He

alt

h G

uid

e

He

alt

hy

Ch

ild

ren

Ste

ps2

Ca

re

Sy

mp

tify

Sy

mp

tom

ate

Do

cto

r D

iag

no

se

Dru

gs.

com

Fre

eM

D

Fa

mil

y D

oct

or

Ea

rly

Do

c

NH

S

Isa

be

l

Sy

mca

t

He

alt

hw

ise

iTri

ag

e

Symptom Checker

Requires non-emergnt care

Acute otitis media ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ▩Acute pharyngitis ⊗ 〇 ⊗ ⊗ 〇 ⊗ 〇 ⊗ ⊗ ⊗ 〇 ▩ N

Acute pharyngitis ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ N

Acute sinusitis ⊗ 〇〇 ⊗ 〇〇 ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ▩Back pain with foot drop 〇 ⊗ ⊗ ⊗ ⊗ 〇 ▨Cellulitis ⊗ ⊗ 〇〇

COPD flare ⊗ ⊗ ⊗ ⊗ 〇〇 ⊗ ▨ N

Influenza 〇 ⊗ ⊗ 〇 ⊗ 〇 ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ▨Mononucleosis ⊗ 〇〇 ⊗ ⊗ 〇 ⊗ 〇〇 ⊗ ⊗ ▩Peptic Ulcer Disease ⊗ ⊗ 〇〇 ⊗ 〇 ⊗ ⊗ ⊗ ⊗ 〇 ⊗ 〇 ⊗ N

Pneumonia ⊗ 〇 ⊗ ⊗ 〇 ⊗ 〇〇 ⊗ ▩Salmonella 〇 ⊗ 〇 ⊗ ▩Shingles ⊗ ⊗ ⊗ 〇 ⊗ 〇〇〇 ⊗ ⊗ N

Urinary tract infection ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ ⊗ ▩Vertigo ⊗ ⊗ 〇 ⊗ ⊗ N


Page 42 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


Do

cRe

spo

nse

Fa

mil

y D

oct

or

Isa

be

l

Ask

MD

/

Sh

are

care

Do

cto

r D

iag

no

se

Dru

gs.

com

Sy

mca

t

He

alt

hli

ne

Fre

eM

D

iTri

ag

e

We

bM

D

HM

S F

am

ily

He

alt

h G

uid

e

Ea

rly

Do

c

Sy

mp

tom

ate

Sy

mp

tify

Be

tte

r M

ed

icin

e

Esa

gil

Ma

yo

Cli

nic

ME

Do

cto

r

HM

S F

am

ily

He

alt

h G

uid

e

He

alt

hy

Ch

ild

ren

Ste

ps2

Ca

re

Sy

mp

tify

Sy

mp

tom

ate

Do

cto

r D

iag

no

se

Dru

gs.

com

Fre

eM

D

Fa

mil

y D

oct

or

Ea

rly

Do

c

NH

S

Isa

be

l

Sy

mca

t

He

alt

hw

ise

iTri

ag

e

Symptom Checker

Acute bronchitis ⊗ 〇〇〇〇 ⊗ ⊗ ▧ N

Acute bronchitis 〇 ⊗ ⊗ 〇 ⊗ 〇 ⊗ 〇〇〇 ⊗ ⊗ ⊗ N

Acute conjunctivitis ⊗ ⊗ ⊗ ◯ ⊗ ◯ ⊗ ⊗ ⊗ ◯ ⊗ ⊗ ▨Acute pharyngitis ⊗ ⊗ 〇 ⊗ 〇 ⊗ ⊗ 〇 ⊗ ⊗ ▧ N

Allergic rhinitis ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ ⊗ ⊗ 〇〇 ⊗ ⊗ ⊗ ▧Back pain, unremarkable ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ⊗ 〇 ⊗ ⊗ ▧Bee sting without anaphylaxis ⊗ 〇 ⊗ 〇

Candidal yeast infection ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ 〇 ⊗ ▩Canker sore ⊗ ⊗ 〇〇〇〇 ⊗

Constipation 〇 ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ N U

Eczema ⊗ ⊗ ⊗ 〇〇 ⊗ 〇〇 ⊗ ⊗ H

Stye ⊗ 〇〇 ⊗ 〇〇 ⊗ 〇 ⊗ ⊗ ▧Viral upper respiratory illness 〇 ◯ ◯ 〇 ⊗ ⊗ 〇 ⊗ 〇〇 ◯ ⊗ 〇 ⊗ ⊗ ⊗ ▧Viral upper respiratory illness ⊗ 〇 ⊗ ⊗ 〇〇 ⊗ 〇〇 ⊗ 〇 ▧Vomiting ⊗ ⊗ 〇 ⊗ 〇 ⊗ 〇 ⊗ 〇 ⊗ ⊗ N N N


Page 43 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

Rate*

Appropriate triage %

(95% CI)

All symptom checkers 301/532 57 (52 to 61)

Without iTriage 287/489 59 (54 to 63)

Without Symcat, Symptomate, and Isabel 249/428 58 (53 to 63)

Without Symcat, Symptomate, Isabel, and iTriage 235/385 61 (56 to 66)

* Number of correct SP evaluations / Applicable SP evaluations. As noted in text, some SP vignettes

could not be applied to a given symptom checker. For example, we could not evaluate an adult SP

vignette if it was a pediatric symptom checker.

Supplemental Table 5: Sensitivity analysis for overall appropriateness of triage advice when

symptom checkers that always provide advice to go to the emergency department are

removed (iTriage) and when symptom checkers that never suggest self-care are removed

(Symcat, Symptomate, and Isabel).

Page 44 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nlySummary Box

Section 1: “What is already known on this topic”

The public is increasingly using the internet for self-diagnosis and triage advice, and there has

been a proliferation of computerized algorithms called symptom checkers that attempt to

streamline this process. Despite the growth in use of these tools, their clinical performance

has not been thoroughly assessed.

Section 2: “What this study adds”

Our study suggests that symptom checkers have deficits in both diagnosis and triage, and their

triage advice is generally risk-averse.

Page 45 of 45


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Documents

The Effectiveness of Symptom Checkers for Self-Diagnosis ... · checkers on appropriate triage advice ranged from 33% (95% CI 19 to 48) to 78% (95% CI 64 to 91) of SP evaluations