CoChat
Sign Up
Cited by 183Mobile Health and mHealth ApplicationsArtificial Intelligence in Healthcare and EducationArtificial Intelligence in Healthcare

How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs

Open access
Gilbert, Stephen, Mehl, Alicia, Baluch, Adel, Cawley, Caoimhe, Challiner, Jean, Fraser, Hamish, Millen, Elizabeth, Montazeri, Maryam +9 more·2020·BMJ Open
DOI 10.1136/bmjopen-2020-040269PMC PMC7745523

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives: To compare breadth of condition coverage, accuracy of suggested conditions and appropriateness of urgency advice of eight popular symptom assessment apps. Design: Vignettes study. Setting: 200 primary care vignettes. Intervention/comparator: For eight apps and seven general practitioners (GPs): breadth of coverage and condition-suggestion and urgency advice accuracy measured against the vignettes’ gold-standard. Primary outcome measures: (1) Proportion of conditions ‘covered’ by an app, that is, not excluded because the user was too young/old or pregnant, or not modelled; (2) proportion of vignettes with the correct primary diagnosis among the top 3 conditions suggested; (3) proportion of ‘safe’ urgency advice (ie, at gold standard level, more conservative, or no more than one level less conservative). Results: Condition-suggestion coverage was highly variable, with some apps not offering a suggestion for many users: in alphabetical order, Ada: 99.0%; Babylon: 51.5%; Buoy: 88.5%; K Health: 74.5%; Mediktor: 80.5%; Symptomate: 61.5%; Your.MD: 64.5%; WebMD: 93.0%. Top-3 suggestion accuracy was GPs (average): 82.1%±5.2%; Ada: 70.5%; Babylon: 32.0%; Buoy: 43.0%; K Health: 36.0%; Mediktor: 36.0%; Symptomate: 27.5%; WebMD: 35.5%; Your.MD: 23.5%. Some apps excluded certain user demographics or conditions and their performance was generally greater with the exclusion of corresponding vignettes. For safe urgency advice, tested GPs had an average of 97.0%±2.5%. For the vignettes with advice provided, only three apps had safety performance within 1 SD of the GPs—Ada: 97.0%; Babylon: 95.1%; Symptomate: 97.8%. One app had a safety performance within 2 SDs of GPs—Your.MD: 92.6%. Three apps had a safety performance outside 2 SDs of GPs—Buoy: 80.0% (p<0.001); K Health: 81.3% (p<0.001); Mediktor: 87.3% (p=1.3×10 -3 ). Conclusions: The utility of digital symptom assessment apps relies on coverage, accuracy and safety. While no digital tool outperformed GPs, some came close, and the nature of iterative improvements to software offers scalable improvements to care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2020-040269

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The accuracy observed in this study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in Gilbert et al. but lower than in another recent comparative study (72%).

    “Furthermore, the accuracy observed in our study is close to the performance rate of ADA (48.5%) across a broad spectrum of medical conditions in the study by Gilbert et al [34] but lower than in another recent comparative study (72%) [35].”

    From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]

    The Abstract Results report Ada’s condition-suggestion accuracy across the study’s vignettes, including 70.5% for top-three suggestions, and the Discussion describes the vignettes as covering varied conditions. The passages available from our copy do not state the cited 48.5% performance rate. The separate 72% study is attributed to citation [35], not Gilbert et al. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The top-5 condition suggestion performance in this study is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) or patients seeking emergency care (91.3%).

    “In two-thirds (34/49, 67%) of cases, 1 in 5 condition suggestions aligned with any of the interview-based diagnoses, which is somewhat below the range of performance rates of ADA in previous studies using case vignettes (77%-84%) [34,35] or patients seeking emergency care (91.3%) [58].”

    From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]

    The Methods define top-5 accuracy (M5), and the Discussion > Principal findings reports Ada’s top-5 performance in a vignette study. The passages do not give an Ada top-5 rate of 77%–84%, so they do not verify the numerical range attributed to this source. The 91.3% emergency-care figure has a separate citation in the citing sentence. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Evidence indicates that the use of algorithms over other methods, the inclusion of demographic information, or more rigorous questioning could explain the differences between symptom checkers' diagnostic performances.

    “On this matter, evidence indicates that the use of algorithms over other methods, the inclusion of demographic information [57], or more rigorous questioning [35] could explain the differences between symptom checkers’ diagnostic performances.”

    From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]

    The Discussion (Principal findings; page unavailable) says differences in app performance likely reflect, among other factors, each app’s question flow. This supports questioning as a possible explanation, but the paper does not say that more rigorous questioning explains the differences. Citation [35] is attached to questioning, not to the separately cited demographic-information point. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Previous studies have shown considerable differences between symptom checkers' diagnostic accuracy.

    “As previous studies have shown considerable differences between symptom checkers’ diagnostic accuracy [33,35], future studies could compare various symptom checkers for the formal diagnosis of mental disorders.”

    From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 30 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., Richter, C., Türk, E., Upadhyay, S., Virani, V., Vona, N., Wicks, P., & Novorol, C.
Affiliations
Brown University (US)
Corresponding author
Gilbert, Stephen
ORCID
Venue
BMJ Open
Publisher
BMJ
Published
2020-12
Volume
10
Issue
12
Pages
e040269
ISSN
2044-6055 (ISSN-L)
DOI
10.1136/bmjopen-2020-040269
PMC
PMC7745523
OpenAlex
W3112185662
Cited by
183
References
30
Citations by year
2026: 20 · 2025: 25 · 2024: 31 · 2023: 37 · 2022: 40 · 2021: 27 · 2020: 3
FWCI
27.92
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • HighWire Press Open Archive (submittedVersion)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
symptom checkers, clinical vignettes, general practitioners, Ada, Babylon, buoys, WebMD, primary care, diagnostic accuracy, patient safety, user demographics, mobile health apps, telemedicine, primary diagnosis
MeSH terms
Humans, Primary Health Care, General Practitioners, Symptom Assessment, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review