The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Accuracy of online symptom checkers and the potential impact on service utilisation
Open accessPublished version of a preprint
Abstract
Objectives: The aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems. Setting: Publicly available symptom checkers for patient use. Participants: Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher. Primary and secondary outcome measures: System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette. Results: Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems. Conclusions: There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As previous studies have shown considerable differences between symptom checkers’ diagnostic accuracy [33,35], future studies could compare various symptom checkers for the formal diagnosis of mental disorders.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489
Applications of Machine Learning Predictive Models in the Chronic Disease Diagnosis
Battineni, G., Sagaro, G. G., Chintalapudi, N., & Amenta, F. (2020) · Journal of Personalized Medicine · doi:10.3390/jpm10020021 · Cited by 418
Coster, J. E., Turner, J. K., Bradbury, D., & Cantrell, A. (2017) · Academic Emergency Medicine · doi:10.1111/acem.13220 · Cited by 379
Amante, D. J., Hogan, T. P., Pagoto, S., English, T. M., & Lapane, K. L. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4126 · Cited by 350
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · cited 1×
“Several previous studies [13, 14] into the effectiveness of algorithmic performance have found deficiencies in the diagnostic capabilities and a cautious approach to triage.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167
Meyer, A. N. D., Giardina, T. D., Spitzmueller, C., Shahid, U., Scott, T. M. T., & Singh, H. (2019) · Journal of Medical Internet Research · doi:10.2196/14679 · Cited by 162
O’Cathain, A., Connell, J., Long, J., & Coster, J. E. (2019) · Health Expectations · doi:10.1111/hex.12995 · Cited by 112
Flagged first, then by citations
Research works printed but not linked: 1
- J Beech, 2019, Closing the Gap: Key Areas for Action on the Health and Care Workforce
Cited by
Sorted by citations
Items 101–124 of 124
YouDiagnose Predictive Model Pilot Study to Compare Expert Human vs. Machine Prediction AccuracyPreprint
Misro, A. K., Sharma, V., & Kadoglou, N. (2023) · medRxiv · doi:10.1101/2023.04.18.23288678
Liu, V., Koskela, T., & Kaila, M. (2023) · doi:10.2196/preprints.49401
Freyer, O., Bergey, F., Cotte, F., Gilbert, S., & Mehl, A. (2023) · doi:10.2196/preprints.52486
Yap, A. Q. Y., Wong, K., Lai, N. M., Lim, Z. S., Thein, W. M., Hassan, K. N., Pala, N., Elliza, S. A., & Bahari, R. (2023) · doi:10.2196/preprints.53287
Pankow, A., Meißner-Bendzko, N., Kaufeld, J. K., Fouquette, L., Cotte, F., Gilbert, S., Türk, E., Das, A. M., Terkamp, C., Burmester, G.-R., et al. (2023) · doi:10.2196/preprints.55001
Liu, V., Koskela, T., & Kaila, M. (2023) · doi:10.2196/preprints.54818
Reinders, P., Augustin, M., & Otten, M. (2024) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.15454_g
Samadpour, H., Kalhori, S. R. N., Tahmasebi, M., Rakhshi, M., & Lotfi, F. (2024) · Research Square · doi:10.21203/rs.3.rs-5429142/v1
Sellin, J., Pantel, J. T., Börsch, N., Conrad, R., & Mücke, M. (2025) · Zeitschrift für Pneumologie · doi:10.1007/s10405-025-00606-y
SOCIAL MEDIA AND GERIATRIC HEALTH: PROMISES AND PITFALLS
Nagendra, L., Mamoun, M., & Gupta, L. (2025) · Anti-Aging Eastern Europe · doi:10.56543/aaeeu.2025.4.1.09
Gilbert, A., Diep, A. N., Boufraioua, M., Pétré, B., Donneau, A.-F., & Ghuysen, A. (2022) · Figshare · doi:10.6084/m9.figshare.21200636
Gilbert, A., Diep, A. N., Boufraioua, M., Pétré, B., Donneau, A.-F., & Ghuysen, A. (2022) · Figshare · doi:10.6084/m9.figshare.21200633
SymptoBuddy: A Progressive Web-Based AI Symptom Checker to Enhance Preliminary Health Assessment
Ishowo, A. K., Awotunde, J. B., Hitam, N. A., & Oloko, F. I. (2025) · International Conference on Artificial Intelligence, Metaverse and Cybersecurity (ICAMAC) · doi:10.1109/icamac67779.2025.11398697
A Mobile Application Approach to Deep Learning for Symptom-Based Disease Detection From User Textual
Garidzira, T. C., & Matobobo, C. (2025) · International Multidisciplinary Information Technology and Engineering Conference (IMITEC) · doi:10.1109/imitec67386.2025.11410504
Hong, S.-S., Son, L. K., & Kim, K. (2026) · Cognitive Research Principles and Implications · doi:10.1186/s41235-026-00717-x
Curry, E. N., Miller, N. E., North, F., Thompson, M. C., & Pecina, J. L. (2026) · Telemedicine Journal and e-Health · doi:10.1177/15305627261440766
AI-Based Diagnostic Platform Capabilities With Lyme Disease as a Use Case: Integrative Exploration
Maxwell, S. P., McNeely, C. L., Zeraatpisheh, A., Brooks, C., Maxwell, C., & Thomas, K. C. (2026) · Online Journal of Public Health Informatics · doi:10.2196/87529
Application of artificial intelligence chatbots for primary diagnosis at home
Hirosawa, T. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-43934-6.00006-7
Zhao, Z., & Arnesano, C. (2026) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml71097.2026.11686893
Usability and Acceptability of a Symptom Checker App by Patients Seeking Urgent Care
Fraser, H., Bacher, I., Khatri, I. S., Raman, S., Nagle, D., Crossland, D., Madsen, T., & Hilliard, R. (2026) · Applied Clinical Informatics · doi:10.1055/a-2962-9261
Alfano, M., Kellett, J., Lenzitti, B., & Helfert, M. (2026) · Digital Health · doi:10.1177/20552076261462696
Abundance, C., BRIAN, C., Collins, O., Otieno, A., Samwel, W., Obare, A., Obuya, M., Odhiambo, M., Ngala, E., Hillary, M., et al. (2026) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/26oct762
Metadata
- Authors
- Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T.
- Affiliations
- University of Exeter (GB)
- University of Leicester (GB)
- Corresponding author
- Ceney, Adam
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-07-15
- Volume
- 16
- Issue
- 7
- Pages
- e0254088
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC8282353
- OpenAlex
- W3182482959
- Cited by
- 125
- References
- 18
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 25 · 2023: 29 · 2022: 33 · 2021: 5
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health and Care Research (AI_AWARD02290)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- online symptom checkers, diagnostic accuracy, triage, health care utilization, clinical vignettes, guideline adherence, external validation, emergency care
- MeSH terms
- Humans, Patient Acceptance of Health Care, Quality of Health Care, Self Care, Triage, Symptom Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
