Accuracy of online symptom checkers and the potential impact on service utilisation
Open accessPublished version of a preprint
Abstract
Objectives: The aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems. Setting: Publicly available symptom checkers for patient use. Participants: Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher. Primary and secondary outcome measures: System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette. Results: Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems. Conclusions: There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Diagnostic performance of symptom checkers varies significantly between different symptom checkers and has been interpreted as low to moderate at best.
“Research indicates that, although symptom checkers seem to be easy to use and well-accepted by most users [30,31], the diagnostic performance varies significantly between different symptom checkers and has been interpreted as low to moderate at best [32,33].”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
The Results (Accuracy of diagnosis) report significant variation between checkers (P < 0.0001), and the Discussion (Summary) describes average diagnostic accuracy as poor. But “low to moderate at best” overlooks the most accurate checker, which listed the correct diagnosis first 72.0% of the time; the paper also says some systems achieved good accuracy. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPrevious studies have shown considerable differences between symptom checkers' diagnostic accuracy.
“As previous studies have shown considerable differences between symptom checkers’ diagnostic accuracy [33,35], future studies could compare various symptom checkers for the formal diagnosis of mental disorders.”
From Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients (2022) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
Applications of Machine Learning Predictive Models in the Chronic Disease Diagnosis
Battineni, G., Sagaro, G. G., Chintalapudi, N., & Amenta, F. (2020) · Journal of Personalized Medicine · doi:10.3390/jpm10020021 · Cited by 418 · see citing works
Coster, J. E., Turner, J. K., Bradbury, D., & Cantrell, A. (2017) · Academic Emergency Medicine · doi:10.1111/acem.13220 · Cited by 379 · see citing works
Amante, D. J., Hogan, T. P., Pagoto, S., English, T. M., & Lapane, K. L. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4126 · Cited by 350 · see citing works
Chambers, D., Cantrell, A. J., Johnson, M., Preston, L., Baxter, S. K., Booth, A., & Turner, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027743 · Cited by 218 · cited 1× · see citing works
“Several previous studies [13, 14] into the effectiveness of algorithmic performance have found deficiencies in the diagnostic capabilities and a cautious approach to triage.”
BackgroundGilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · see citing works
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182 · see citing works
A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis
Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey, D., Sangar, D., Butt, M., DoRosario, A., & Johri, S. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.543405 · Cited by 167 · see citing works
Meyer, A. N. D., Giardina, T. D., Spitzmueller, C., Shahid, U., Scott, T. M. T., & Singh, H. (2019) · Journal of Medical Internet Research · doi:10.2196/14679 · Cited by 162 · see citing works
A machine learning approach to triaging patients with chronic obstructive pulmonary disease
Swaminathan, S., Qirko, K., Smith, T., Corcoran, E., Wysham, N. G., Bazaz, G., Kappel, G., & Gerber, A. N. (2017) · PLoS ONE · doi:10.1371/journal.pone.0188532 · Cited by 101 · see citing works
O’Cathain, A., Connell, J., Long, J., & Coster, J. E. (2019) · Health Expectations · doi:10.1111/hex.12995 · Cited by 101 · see citing works
Applications of Machine Learning Approaches in Emergency Medicine; a Review Article.
Shafaf, N., & Malek, H. (2019) · Archives of Academic Emergency Medicine · doi:10.22037/aaem.v7i1.410 · Cited by 97 · see citing works
Are online symptoms checkers useful for patients with inflammatory arthritis?
Powley, L., McIlroy, G., Simons, G., & Raza, K. (2016) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-016-1189-2 · Cited by 74 · see citing works
Aboueid, S., Liu, R. H., Desta, B. N., Chaurasia, A. K., & Ebrahim, S. (2019) · JMIR Medical Informatics · doi:10.2196/13445 · Cited by 70 · see citing works
Accuracy of a Popular Online Symptom Checker for Ophthalmic Diagnoses
Shen, C., Nguyen, M., Gregor, A., Isaza, G., & Beattie, A. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.0571 · Cited by 40 · see citing works
Davies, B. M., Munro, C., & Kotter, M. (2018) · Journal of Medical Internet Research · doi:10.2196/10868 · Cited by 38 · see citing works
Online symptom checker diagnostic and triage accuracy for HIV and hepatitis C
Berry, A. C., Cash, B. D., Wang, B., Mulekar, M. S., Haneghan, A. B. V., Yuquimpo, K., Swaney, A., Marshall, M. C., & Green, W. K. (2019) · Epidemiology and Infection · doi:10.1017/s0950268819000268 · Cited by 28 · see citing works
Backman, A.-S., Lagerlund, M., Svensson, T., Blomqvist, P., & Adami, J. (2012) · Emergency Medicine Journal · doi:10.1136/emermed-2011-200464 · Cited by 14 · see citing works
Dall, T. M., West, T., Chakrabarti, R., et al. (2015) · doi:10.13140/rg.2.2.13111.57764 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 344 · see citing works
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 189 · see citing works
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128 · see citing works
Advancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 112 · see citing works
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99 · see citing works
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74 · see citing works
Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review
Sasco, E. R., El-Osta, A. A. R., Alaa, A., Webber, I., Karki, M., Asmar, M. L. E., Purohit, K., Painter, A., & Hayhoe, B. W. J. (2023) · Journal of Medical Internet Research · doi:10.2196/43803 · Cited by 76 · see citing works
Sarbay, İ., Berikol, G. B., & Özturan, İ. U. (2023) · Turkish Journal of Emergency Medicine · doi:10.4103/tjem.tjem_79_23 · Cited by 64 · see citing works
Dialogue System for Early Mental Illness Detection: Toward a Digital Twin Solution
Abilkaiyrkyzy, A., Laamarti, F., Hamdi, M., & Saddik, A. E. (2024) · IEEE Access · doi:10.1109/access.2023.3348783 · Cited by 71 · see citing works
Pairon, A., Philips, H., & Verhoeven, V. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2022.1040926 · Cited by 48 · see citing works
Bushuven, S., Bentele, M., Bentele, S., Gerber, B., Bansbach, J., Ganter, J., Trifunovic-Koenig, M., & Ranisch, R. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-02019-x · Cited by 47 · see citing works
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 54 · see citing works
Cotte, F., Mueller, T., Gilbert, S., Blümke, B., Multmeier, J., Hirsch, M. C., Wicks, P. J., Wolanski, J., Tutschkow, D., Schade‐Brittinger, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/32340 · Cited by 41 · see citing works
Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial
Kopka, M., Schmieding, M. L., Rieger, T., Roesler, E., Balzer, F., & Feufel, M. A. (2022) · JMIR Human Factors · doi:10.2196/35219 · Cited by 37 · see citing works
El-Osta, A. A. R., Webber, I., Alaa, A., Bagkeris, E., Mian, S., Sharabiani, M. T. A., & Majeed, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053566 · Cited by 30 · see citing works
Hennemann, S., Kuhn, S., Witthöft, M., & Jungmann, S. M. (2022) · JMIR Mental Health · doi:10.2196/32832 · Cited by 31 · see citing works
“Most recently, a study by Ceney et al [33] yielded comparable average performance rates (51%, range 22.2%-84”
BackgroundResultInfluentialChallenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review
Ilicki, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279636 · Cited by 33 · see citing works
Sun, S. H., Huynh, K. N., Cortes, G., Hill, R., Tran, J., Yeh, L., Ngo, A. L., Houshyar, R., Yaghmai, V., & Tran, M. D. (2024) · Radiology · doi:10.1148/radiol.232346 · Cited by 32 · see citing works
Fraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/38364 · Cited by 32 · see citing works
Radionova, N., Ög, E., Wetzel, A.-J., Rieger, M. A., & Preiser, C. (2023) · Journal of Medical Internet Research · doi:10.2196/39219 · Cited by 31 · see citing works
Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · Journal of Medical Internet Research · doi:10.2196/37408 · Cited by 30 · see citing works
Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Ben‐Shabat, N., Sharvit, G., Meimis, B., Joya, D. B., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, ., & Amital, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104897 · Cited by 28 · see citing works
Telepsychiatry in an Era of Digital Mental Health Startups
Achtyes, E. D., Glenn, T., Monteith, S., Geddes, J. R., Whybrow, P. C., Martini, J., & Bauer, M. (2023) · Current Psychiatry Reports · doi:10.1007/s11920-023-01425-9 · Cited by 28 · see citing works
Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases
Chen, A., Chen, D. O., & Tian, L. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad245 · Cited by 25 · see citing works
Prevalence and course of pregnancy symptoms using self-reported pregnancy app symptom tracker data
Nissen, M., Campo, N. B., Flaucher, M., Jaeger, K. M., Titzmann, A., Blunck, D., Fasching, P. A., Engelhardt, V., Eskofier, B. M., & Leutheuser, H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00935-3 · Cited by 25 · see citing works
Knitza, J., Taşçılar, K., Fuchs, F., Mohn, J., Kühn, S., Bohr, D., Muehlensiepen, F., Bergmann, C., Labinsky, H., Morf, H., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/55542 · Cited by 24 · see citing works
Faqar-Uz-Zaman, S. F., Anantharajah, L., Baumartz, P., Sobotta, P., Filmann, N., Zmuc, D., Wagner, M., Detemble, C., Sliwinski, S., Marschall, U., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005614 · Cited by 25 · see citing works
Extracting symptoms from free-text responses using ChatGPT among COVID-19 cases in Hong Kong
Wei, W. I., Leung, C. L. K., Tang, A., McNeil, E. B., Wong, S. Y. S., & Kwok, K. O. (2023) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2023.11.002 · Cited by 24 · see citing works
Kopka, M., Feufel, M. A., Berner, E. S., & Schmieding, M. L. (2023) · Digital Health · doi:10.1177/20552076231194929 · Cited by 23 · see citing works
The Triage Capability of Laypersons: Retrospective Exploratory Analysis
Kopka, M., Feufel, M. A., Balzer, F., & Schmieding, M. L. (2022) · JMIR Formative Research · doi:10.2196/38977 · Cited by 22 · see citing works
Triage Errors in Primary and Pre–Primary Care
Nguyen, H., Meczner, A., Burslam-Dawe, K., & Hayhoe, B. W. J. (2022) · Journal of Medical Internet Research · doi:10.2196/37209 · Cited by 21 · see citing works
Latent classes associated with the intention to use a symptom checker for self-triage
Aboueid, S., Meyer, S. B., Wallace, J. R., & Chaurasia, A. K. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259547 · Cited by 16 · see citing works
Gilbert, S., Fenech, M. E., Upadhyay, S., Wicks, P. J., & Novorol, C. (2021) · Australian Journal of Primary Health · doi:10.1071/py21032 · Cited by 17 · see citing works
Wetzel, A.-J., Koch, R., Preiser, C., Mueller, R., Klemmt, M., Ranisch, R., Ehni, H.-J., Wiesing, U., Rieger, M. A., Henking, T., et al. (2022) · JMIR Research Protocols · doi:10.2196/34026 · Cited by 15 · see citing works
Guideline for software life cycle in health informatics
Hauschild, A.-C., Martin, R., Holst, S. C., Wienbeck, J., & Heider, D. (2022) · iScience · doi:10.1016/j.isci.2022.105534 · Cited by 19 · see citing works
Zaman, N. I. D., Hau, Y. W., Leong, M. C., & Al‐Ashwal, R. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10528-x · Cited by 16 · see citing works
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2024) · JMIR AI · doi:10.2196/46875 · Cited by 15 · see citing works
Schmieding, M. L., Kopka, M., Bolanaki, M., Napierala, H., Altendorf, M. B., Kuschick, D., Piper, S. K., Scatturin, L., Schmidt, K. F. R., Schorr, C., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64028 · Cited by 14 · see citing works
Ferhi, L. A., Amar, M. B., Choubani, F., & Bouallègue, R. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1397388 · Cited by 14 · see citing works
Naved, B. A., Ravishankar, S., Colbert, G. E., Johnston, A., Slott, Q. M., & Luo, Y. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01779-9 · Cited by 13 · see citing works
Kopka, M., & Feufel, M. A. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00096-7 · Cited by 13 · see citing works
Leaders’ support for using online symptom checkers in Finnish primary care: Survey study
Laukka, E., Kujala, S., Gluschkoff, K., Kanste, O. I., Hörhammer, I., & Heponiemi, T. (2021) · Health Informatics Journal · doi:10.1177/14604582211052259 · Cited by 12 · see citing works
Diseasomics: Actionable machine interpretable disease knowledge at the point-of-care
Talukder, A. K., Schriml, L. M., Ghosh, A., Biswas, R., Chakrabarti, P., & Haas, R. E. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000128 · Cited by 12 · see citing works
Wetzel, A.-J., Koch, R., Koch, N. N., Klemmt, M., Müller, R., Preiser, C., Rieger, M. A., Rösel, I., Ranisch, R., Ehni, H., et al. (2024) · Digital Health · doi:10.1177/20552076241231555 · Cited by 11 · see citing works
Kopka, M., Napierala, H., Privoznik, M., Sapunova, D., Zhang, S., & Feufel, M. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-83844-z · Cited by 14 · see citing works
Chatbots in Pharmacy: A Boon or a Bane for Patient Care and Pharmacy Practice?
Ramadhani, C. (2023) · Sciences of Pharmacy · doi:10.58920/sciphar02030001 · Cited by 14 · see citing works
Symptoms are known by their companies: towards association guided disease diagnosis assistant
Tiwari, A., Saha, T., Saha, S., Bhattacharyya, P., Begum, S., Dhar, M., & Tiwari, S. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-05032-y · Cited by 11 · see citing works
Faqar-Uz-Zaman, S. F., Filmann, N., Mahkovic, D., Wagner, M., Detemble, C., Kippke, U., Marschall, U., Anantharajah, L., Baumartz, P., Sobotta, P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-041396 · Cited by 10 · see citing works
Technology-supported self-triage decision making
Kopka, M., Wang, S. M., Kunz, S., Schmid, C., & Feufel, M. A. (2025) · npj Health Systems · doi:10.1038/s44401-024-00008-x · Cited by 11 · see citing works
Vuillaume, L. A., Turpinier, J., Cipolat, L., Arnaud-Dépil-Duval, Dumontier, T., Peschanski, N., Kieffer, Y., Barbat, B., Riquier, T., Dinot, V., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0277568 · Cited by 9 · see citing works
Schmude, M., Salim, N., Azadzoy, H., Bane, M., Millen, E., O’Donnell, L., Bode, P., Türk, E., Vaidya, R., & Gilbert, S. (2022) · JMIR Research Protocols · doi:10.2196/34298 · Cited by 9 · see citing works
de Arriba-Pérez, F., & García-Méndez, S. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-024-09508-2 · Cited by 8 · see citing works
Reinders, P., Augustin, M., & Otten, M. (2024) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.15454 · Cited by 7 · see citing works
Liu, V., Kaila, M., & Koskela, T. (2024) · JMIR Human Factors · doi:10.2196/55099 · Cited by 7 · see citing works
Empowering Medical Diagnosis: A Machine Learning Approach for Symptom-Based Health Checker
Ferhi, L. A., Amar, M. B., Choubani, F., & Bouallègue, R. (2024) · Mobile Networks and Applications · doi:10.1007/s11036-024-02369-x · Cited by 7 · see citing works
Toward Symptom Assessment Guided Symptom Investigation and Disease Diagnosis
Tiwari, A., Raj, R., Saha, S., Bhattacharyya, P., Tiwari, S., & Dhar, M. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3236897 · Cited by 7 · see citing works
Validation of an Artificial Intelligence-Powered Virtual Assistant for Emergency Triage in Neurology
Alessandro, L., Crema, S., Castiglione, J. I., Dossi, D., Eberbach, F., Köhler, A., Laffue, A., Marone, A., Nagel, V., Rueda, J. M. P., et al. (2025) · The Neurologist · doi:10.1097/nrl.0000000000000594 · Cited by 6 · see citing works
Gilbert, A., Diep, A. N., Boufraioua, M., Pétré, B., Donneau, A., & Ghuysen, A. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08571-5 · Cited by 6 · see citing works
International dimensions of clinical decision support systems
Magrabi, F., Cresswell, K. M., & Fraser, H. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00013-9 · Cited by 6 · see citing works
Liu, V., Koskela, T., & Kaila, M. (2023) · JMIR Research Protocols · doi:10.2196/41423 · Cited by 6 · see citing works
Benoit, J. R. A., Hartling, L., & Scott, S. D. (2023) · PEC Innovation · doi:10.1016/j.pecinn.2023.100152 · Cited by 5 · see citing works
Koch, R., Steffen, M.-T., Wetzel, A.-J., Preiser, C., Klemmt, M., Ehni, H., Mueller, R., & Joos, S. (2025) · JMIR Formative Research · doi:10.2196/60647 · Cited by 5 · see citing works
Avey: An Accurate AI Algorithm for Self-DiagnosisPreprint
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2022) · medRxiv · doi:10.1101/2022.03.08.22272076 · Cited by 5 · see citing works
Evaluating digital triage symptom checker with historical triage-related adverse events
Ilicki, J., Edman, S., Stalfors, J., & Molin, C. J. (2025) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2025.2563517 · Cited by 4 · see citing works
Wallace, W. J., Chan, C., Chidambaram, S., Hanna, L., Acharya, A., Daniels, E., Normahani, P., Matin, R., Markar, S. R., Sounderajah, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000558 · Cited by 4 · see citing works
Machine Learning in Clinical Text Classification: Specialty Identification and COVID-19 Risk
Tan, C. Y.-X., Talreja, K., Shivam, A., Nag, A., Miranda, M., Raghav, S., Rajaram, A., Lutz, M., Kumar, S., & Kumar, A. (2022) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc55196.2022.9952512 · Cited by 4 · see citing works
Sendtner, G. W., Muecke, M., Grigull, L., Bender, T. T. A., Behning, C., & Schäfer, V. S. (2025) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-025-04112-5 · Cited by 3 · see citing works
A Comprehensive Examination of User Experience in AI-Based Symptom Checker Chatbots
Ferreira, M. C., Veloso, M., & Tavares, J. M. R. S. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-59376-5_8 · Cited by 3 · see citing works
Analysing weighted disease symptoms to recommend an expert through an integrated healthcare model
Mohapatra, S., & Roy, A. (2024) · Discover Public Health · doi:10.1186/s12982-024-00261-3 · Cited by 3 · see citing works
Pankow, A., Meißner-Bendzko, N., Kaufeld, J. K., Fouquette, L., Cotte, F., Gilbert, S., Türk, E., Das, A. M., Terkamp, C., Burmester, G.-R., et al. (2025) · JMIR AI · doi:10.2196/55001 · Cited by 3 · see citing works
Sellin, J., Pantel, J. T., Börsch, N., Conrad, R., & Mücke, M. (2024) · Der Schmerz · doi:10.1007/s00482-023-00777-8 · Cited by 3 · see citing works
Kopka, M., Napierala, H., Privoznik, M., Sapunova, D., Zhang, S., & Feufel, M. A. (2024) · medRxiv · doi:10.1101/2024.04.02.24305193 · Cited by 3 · see citing works
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2024) · medRxiv · doi:10.1101/2024.09.13.24313657 · Cited by 3 · see citing works
Gehlen, T., Joost, T., Solbrig, P., Stahnke, K., Zahn, R., Jahn, M., Amini, D. A., & Back, D. A. (2025) · Diagnostics · doi:10.3390/diagnostics15020221 · Cited by 2 · see citing works
Pecina, J. L., Curry, E. N., Miller, N. E., Thompson, M. C., & North, F. (2025) · Health Services Insights · doi:10.1177/11786329251385022 · Cited by 2 · see citing works
Kopka, M., & Feufel, M. A. (2026) · JMIR Biomedical Engineering · doi:10.2196/88053 · Cited by 2 · see citing works
Patient-targeted mobile applications in healthcare
Karia, J., Mohamed, R., & Petrushkin, H. (2023) · British Journal of Hospital Medicine · doi:10.12968/hmed.2023.0158 · Cited by 2 · see citing works
Avnat, E., Samin, M., Joya, D. B., Schneider, E., Yanko, E., Eshel, D., Ovadia, S., Lev, Y., & Souroujon, D. (2024) · Journal of Medical Internet Research · doi:10.2196/49570 · Cited by 2 · see citing works
Congruency of Symptom Choice Between Patients Performing Online Self-Triage and Nurse Phone Triage
Curry, E. N., Miller, N. E., North, F., Thompson, M. C., & Pecina, J. L. (2025) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2024.0589 · Cited by 2 · see citing works
Health Sync: An Integrated Healthcare Management Platform
Sathya, A., Tarunes, K., Sriram, G. J., Naresh, K. N., Narendran, M. R., & S, K. V. V. (2025) · International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV) · doi:10.1109/icicv64824.2025.11085743 · Cited by 1 · see citing works
Evaluating the use of red flags by online symptom checkers
Sutaria, S., Devakumar, D., Mallinson, P. A. C., Kinra, S., Malak, T., & Meczner, A. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13353-w · Cited by 1 · see citing works
Kopka, M., & Feufel, M. A. (2026) · JMIR Human Factors · doi:10.2196/76168 · Cited by 1 · see citing works
Xu, Y., Prentice, C., Rueda, S. T., Meczner, A., Multmeier, J., Wickham, A. P., Kelly, L., Klepchukova, A., Stsefanovich, H., & Zhaunova, L. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02332-4 · Cited by 3 · see citing works
AI applications in veterinary digital health: a systematic survey
Polapragada, S. K. S. S. (2026) · Frontiers in Veterinary Science · doi:10.3389/fvets.2026.1853395 · Cited by 1 · see citing works
Vuillaume, L. A., Carval, G., Grajoszex, M., Ouamara, N., & Salvestrini, J. (2022) · Annales françaises de médecine d’urgence · doi:10.3166/afmu-2022-0382 · Cited by 1 · see citing works
Eden, D. J., Hermann, A. D., Sombrotto, L. B., Wilner, P. J., & Chen, J. A. (2024) · Harvard Review of Psychiatry · doi:10.1097/hrp.0000000000000401 · Cited by 1 · see citing works
Accuracy of Dental Symptom Checker Web Application in the Singapore Military Population
Lim, S. N., Woon, X. R., Goh, C. E., Ng, J. C., Ang, S. Y., Lim, T. J., & Allen, P. (2024) · International Dental Journal · doi:10.1016/j.identj.2024.07.006 · Cited by 1 · see citing works
North, F., Buss, R. J., Nelson, E. M., Thompson, M. C., Pecina, J. L., & Crum, B. A. (2024) · Health Services Research and Managerial Epidemiology · doi:10.1177/23333928241271933 · Cited by 1 · see citing works
Comprehensive Mobile Solution for Personalized Healthcare Services using Enhanced AI
Kareem, M. S., Manikandan, B., Selvaraj, K., Anju, A., Narendran, M. R., & S, K. V. V. (2025) · International Conference on Data Science and Business Systems (ICDSBS) · doi:10.1109/icdsbs63635.2025.11031900 · Cited by 1 · see citing works
Wesinger, A., Riedl, E., Kittler, H., & Tschandl, P. (2022) · Dermatology Practical & Conceptual · doi:10.5826/dpc.1203a117 · Cited by 1 · see citing works
Study of the User Behaviour Caused by Automatic Recommendation Systems Call to Action
Kopanitsa, G., & Kovalchuk, S. V. (2022) · Studies in health technology and informatics · doi:10.3233/shti220966 · Cited by 1 · see citing works
Painter, A., Hayhoe, B. W. J., Sasco, E. R., & El-Osta, A. A. R. (2022) · doi:10.2196/preprints.42481 · Cited by 1 · see citing works
Said, N., Schumacher, L., & Huff, M. (2023) · doi:10.31234/osf.io/m5q3p · Cited by 1 · see citing works
Painter, A., & Hayhoe, B. W. J. (2024) · doi:10.2196/preprints.63423 · Cited by 1 · see citing works
El-Osta, A. A. R., Kopka, M., Altalib, S., Neves, A. L., Faisal, A. A., POTTS, H. W. W., & Spinelli, G. (2026) · Research Square · doi:10.21203/rs.3.rs-9004371/v1 · Cited by 1 · see citing works
Painter, A., Hayhoe, B. W. J., & El-Osta, A. A. R. (2022) · doi:10.2196/preprints.37408
Fraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · doi:10.2196/preprints.38364
Vuillaume, L. A., Turpinier, J., Cipolat, L., Depil-Duval, A., Dumontier, T., Peschanski, N., Kieffer, Y., Barbat, B., Riquier, T., Dinot, V., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1580704/v1
Assessing Data Gathering Quality of Chatbot Based Symptom Checkers - a Clinical Vignettes StudyPreprint
Ben‐Shabat, N., Sharvit, G., Meimis, B., Joya, D. B., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, ., & Amital, H. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4128799
Sampietro-Colom, L., Fernández-Barceló, C., Abbas, I., Valdasquin, B., Rabasseda, N., García‐Lorenzo, B., Sánchez, M., Sans-Corrales, M., García, N., & Granados, A. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192113873
- Authors
- Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T.
- Affiliations
- University of Exeter (GB)
- University of Leicester (GB)
- Corresponding author
- Ceney, Adam
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-07-15
- Volume
- 16
- Issue
- 7
- Pages
- e0254088
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC8282353
- OpenAlex
- W3182482959
- Cited by
- 125
- References
- 18
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 25 · 2023: 29 · 2022: 33 · 2021: 5
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health and Care Research (AI_AWARD02290)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- online symptom checkers, diagnostic accuracy, triage, health care utilization, clinical vignettes, guideline adherence, external validation, emergency care
- MeSH terms
- Humans, Patient Acceptance of Health Care, Quality of Health Care, Self Care, Triage, Symptom Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
