The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Accuracy of online symptom checkers and the potential impact on service utilisation
Open accessPublished version of a preprint
Abstract
Objectives: The aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems. Setting: Publicly available symptom checkers for patient use. Participants: Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher. Primary and secondary outcome measures: System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette. Results: Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems. Conclusions: There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As previous studies have shown considerable differences between symptom checkers’ diagnostic accuracy [33,35], future studies could compare various symptom checkers for the formal diagnosis of mental disorders.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489
Applications of Machine Learning Predictive Models in the Chronic Disease Diagnosis
Battineni, G., Sagaro, G. G., Chintalapudi, N., & Amenta, F. (2020) · Journal of Personalized Medicine · doi:10.3390/jpm10020021 · Cited by 418
Coster, J. E., Turner, J. K., Bradbury, D., & Cantrell, A. (2017) · Academic Emergency Medicine · doi:10.1111/acem.13220 · Cited by 379
Amante, D. J., Hogan, T. P., Pagoto, S., English, T. M., & Lapane, K. L. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4126 · Cited by 350
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · cited 1×
“Several previous studies [13, 14] into the effectiveness of algorithmic performance have found deficiencies in the diagnostic capabilities and a cautious approach to triage.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167
Meyer, A. N. D., Giardina, T. D., Spitzmueller, C., Shahid, U., Scott, T. M. T., & Singh, H. (2019) · Journal of Medical Internet Research · doi:10.2196/14679 · Cited by 162
O’Cathain, A., Connell, J., Long, J., & Coster, J. E. (2019) · Health Expectations · doi:10.1111/hex.12995 · Cited by 112
A machine learning approach to triaging patients with chronic obstructive pulmonary disease
Swaminathan, S., Qirko, K., Smith, T., Corcoran, E., Wysham, N. G., Bazaz, G., Kappel, G., & Gerber, A. N. (2017) · PLoS ONE · doi:10.1371/journal.pone.0188532 · Cited by 107
Applications of Machine Learning Approaches in Emergency Medicine; a Review Article.
Shafaf, N., & Malek, H. (2019) · Archives of Academic Emergency Medicine · doi:10.22037/aaem.v7i1.410 · Cited by 99
Are online symptoms checkers useful for patients with inflammatory arthritis?
Powley, L., McIlroy, G., Simons, G., & Raza, K. (2016) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-016-1189-2 · Cited by 74
Aboueid, S., Liu, R. H., Desta, B. N., Chaurasia, A. K., & Ebrahim, S. (2019) · JMIR Medical Informatics · doi:10.2196/13445 · Cited by 72
Accuracy of a Popular Online Symptom Checker for Ophthalmic Diagnoses
Shen, C., Nguyen, M., Gregor, A., Isaza, G., & Beattie, A. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.0571 · Cited by 40
Davies, B. M., Munro, C., & Kotter, M. (2018) · Journal of Medical Internet Research · doi:10.2196/10868 · Cited by 38
Online symptom checker diagnostic and triage accuracy for HIV and hepatitis C
Berry, A. C., Cash, B. D., Wang, B., Mulekar, M. S., Haneghan, A. B. V., Yuquimpo, K., Swaney, A., Marshall, M. C., & Green, W. K. (2019) · Epidemiology and Infection · doi:10.1017/s0950268819000268 · Cited by 28
Backman, A.-S., Lagerlund, M., Svensson, T., Blomqvist, P., & Adami, J. (2012) · Emergency Medicine Journal · doi:10.1136/emermed-2011-200464 · Cited by 14
Dall, T. M., West, T., Chakrabarti, R., et al. (2015) · doi:10.13140/rg.2.2.13111.57764 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 1
- J Beech, 2019, Closing the Gap: Key Areas for Action on the Health and Care Workforce
Cited by
Sorted by citations
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 351
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 213
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128
Advancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 121
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99
Metadata
- Authors
- Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T.
- Affiliations
- University of Exeter (GB)
- University of Leicester (GB)
- Corresponding author
- Ceney, Adam
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-07-15
- Volume
- 16
- Issue
- 7
- Pages
- e0254088
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC8282353
- OpenAlex
- W3182482959
- Cited by
- 125
- References
- 18
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 25 · 2023: 29 · 2022: 33 · 2021: 5
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health and Care Research (AI_AWARD02290)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- online symptom checkers, diagnostic accuracy, triage, health care utilization, clinical vignettes, guideline adherence, external validation, emergency care
- MeSH terms
- Humans, Patient Acceptance of Health Care, Quality of Health Care, Self Care, Triage, Symptom Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
