Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study
Open accessPublished version of a preprint
Abstract
Background: Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients. Objective: The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews. Methods: We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated “unsafe” or “too cautious.” Results: Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%). Conclusions: ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI is rapidly reshaping healthcare and the competencies expected of graduating medical students.
“Artificial intelligence (AI) is rapidly reshaping healthcare and the competencies expected of graduating medical students”
From What the AI era doctor should know: a scoping review of proposed artificial intelligence competencies for medical education (2026) · cites this paper as [1]
The Introduction (page unavailable) describes medical tasks ChatGPT can perform, and the Discussion’s Conclusions (page unavailable) describe its potential for diagnosis and triage. These passages support AI’s relevance to healthcare, but they do not establish that it is already rapidly reshaping healthcare or changing the competencies expected of graduating medical students. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAI has enabled novel digital solutions in diagnostics, imaging, and clinical decision support and patient management.
“AI has enabled novel digital solutions in diagnostics1–4, imaging5,6, and clinical decision support and patient management7–9.”
From What the AI era doctor should know: a scoping review of proposed artificial intelligence competencies for medical education (2026) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · see citing works
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120 · see citing works
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · see citing works
DXplain. An evolving diagnostic decision-support system
Barnett, G. O. (1987) · JAMA · doi:10.1001/jama.258.1.67 · Cited by 391 · see citing works
Large language model AI chatbots require approval as medical devices
Gilbert, S., Harvey, H., Melvin, T., Vollebregt, E., & Wicks, P. (2023) · Nature Medicine · doi:10.1038/s41591-023-02412-6 · Cited by 207 · see citing works
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199 · see citing works
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187 · see citing works
Gilbert, S., Mehl, A., Baluch, A., Cawley, C., Challiner, J., Fraser, H., Millen, E., Montazeri, M., Multmeier, J., Pick, F., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040269 · Cited by 183 · see citing works
Self-citationSafety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182 · see citing works
Self-citationComparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · see citing works
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · see citing works
Hill, M. G., Sim, M., & Mills, B. (2020) · Medical Journal of Australia · doi:10.5694/mja2.50600 · Cited by 114 · see citing works
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99 · see citing works
Harskamp, R. E., & Clercq, L. D. (2024) · Acta cardiologica. Supplementum · doi:10.1080/00015385.2024.2303528 · Cited by 92 · see citing works
Harskamp, R. E., & De Clercq, L. (2023) · doi:10.1101/2023.03.25.23285475 · Cited by 92 · see citing works
Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis
Millenson, M. L., Baldwin, J. L., Zipperer, L., & Singh, H. (2018) · Diagnosis · doi:10.1515/dx-2018-0009 · Cited by 91 · see citing works
What learning algorithm is in-context learning? Investigations with linear models
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2211.15661 · Cited by 86 · see citing works
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74 · see citing works
Cotte, F., Mueller, T., Gilbert, S., Blümke, B., Multmeier, J., Hirsch, M. C., Wicks, P. J., Wolanski, J., Tutschkow, D., Schade‐Brittinger, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/32340 · Cited by 41 · see citing works
Levine, D. M., & Mehrotra, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3287 · Cited by 35 · see citing works
Isabel, a Clinical Decision Support System
Vardell, E., & Moore, M. E. (2011) · Medical Reference Services Quarterly · doi:10.1080/02763869.2011.562800 · Cited by 34 · see citing works
Triage accuracy of online symptom checkers for Accident and Emergency Department patients
Yu, S. W. Y., Ma, A., Tsang, V. H. M., Chung, L. S. W., Leung, S. C., & Leung, L. (2019) · Hong Kong Journal of Emergency Medicine · doi:10.1177/1024907919842486 · Cited by 33 · see citing works
Fraser, H., Cohan, G., Koehler, C., Anderson, J. S., Lawrence, A., Patena, J. V., Bacher, I., & Ranney, M. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/38364 · Cited by 32 · see citing works
Self-citationOpenAI — explain why some countries are excluded from ChatGPT
Wang, S. H. (2023) · Nature · doi:10.1038/d41586-023-00553-9 · Cited by 27 · see citing works
Performance of a new symptom checker in patient triage: Canadian cohort study
Chan, F., Lai, S., Pieterman, M., Richardson, L., Singh, A., Peters, J., Toy, A., Piccininni, C. R., Rouault, T., Wong, K., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260696 · Cited by 23 · see citing works
Montazeri, M., Multmeier, J., Novorol, C., Upadhyay, S., Wicks, P. J., & Gilbert, S. (2021) · JMIR Formative Research · doi:10.2196/26402 · Cited by 19 · see citing works
Identifying Emergency Department Symptom-Based Diagnoses with the Unified Medical Language System
Slovis, B. H., McCarthy, D. M., Nord, G., Doty, A. M. B., Piserchia, K., & Rising, K. L. (2019) · Western Journal of Emergency Medicine · doi:10.5811//westjem.2019.8.44230 · Cited by 12 · see citing works
Cancer diagnosis in UK primary care
Hamilton, W. T. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.63 · Cited by 8 · see citing works
AI in Medicine: Some Pertinent History
Shortliffe, E. H., & Shah, N. H. (2022) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-09108-7_2 · Cited by 6 · see citing works
Zhou, Z. (2021) · Springer eBooks · doi:10.1007/978-981-15-1967-3_3 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662 · see citing works
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Li, W., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01029-4 · Cited by 400 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203 · see citing works
Yun, H. S., & Bickmore, T. (2025) · Journal of Medical Internet Research · doi:10.2196/68560 · Cited by 151 · see citing works
Hoppe, J. M., Auer, M. K., Strüven, A. K., Maßberg, S., & Stremmel, C. (2024) · Journal of Medical Internet Research · doi:10.2196/56110 · Cited by 95 · see citing works
Sallam, M., Barakat, M. M., & Sallam, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/54704 · Cited by 88 · see citing works
Shan, G., Chen, X., Wang, C., Liu, L., Gu, Y., Jiang, H., & Shi, T. (2025) · JMIR Medical Informatics · doi:10.2196/64963 · Cited by 71 · see citing works
Arslan, B., Nuhoğlu, Ç., Satıcı, M. O., & Altınbilek, E. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.12.024 · Cited by 64 · see citing works
Wang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025) · Journal of Medical Internet Research · doi:10.2196/64486 · Cited by 51 · see citing works
Kim, J. H., Kim, S. K., Choi, J., & Lee, Y. (2024) · Digital Health · doi:10.1177/20552076241227132 · Cited by 54 · see citing works
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738 · see citing works
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 54 · see citing works
Colakca, C., Ergın, M., Özensoy, H. S., Şener, A., Gürü, S., & Özhasenekler, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-73229-7 · Cited by 44 · see citing works
Frosolini, A., Catarzi, L., Benedetti, S., Latini, L., Chisci, G., Franz, L., Gennaro, P., & Gabriele, G. (2024) · Diagnostics · doi:10.3390/diagnostics14080839 · Cited by 41 · see citing works
Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery
Vaira, L. A., Lechien, J. R., Abbate, V., Gabriele, G., Frosolini, A., Vito, A. D., Maniaci, A., Mayo‐Yáñez, M., Boscolo‐Rizzo, P., Saibene, A. M., et al. (2025) · OTO Open · doi:10.1002/oto2.70075 · Cited by 36 · see citing works
Artificial intelligence, tasks, skills, and wages: Worker-level evidence from Germany
Engberg, E., Koch, M. R., Lodefalk, M., & Schroeder, S. (2025) · Research Policy · doi:10.1016/j.respol.2025.105285 · Cited by 40 · see citing works
Ho, C., Tian, T., Ayers, A. T., Aaron, R. E., Phillips, V., Wolf, R. M., Mathioudakis, N. N., Dai, T., & Klonoff, D. C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02757-z · Cited by 31 · see citing works
Dedicated AI Expert System vs Generative AI With Large Language Model for Clinical Diagnoses
Feldman, M. J., Hoffer, E. P., Conley, J. J., Chang, J. P., Chung, J. A., Jernigan, M. C., Lester, W. T., Strasser, Z. H., & Chueh, H. C. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.12994 · Cited by 31 · see citing works
Yazaki, M., Maki, S., Furuya, T., Inoue, K., Nagai, K., Nagashima, Y., Maruyama, J., Toki, Y., Kitagawa, K., Iwata, S., et al. (2025) · Prehospital Emergency Care · doi:10.1080/10903127.2024.2374400 · Cited by 29 · see citing works
Silverman, A. L., Shung, D. L., Stidham, R. W., Kochhar, G. S., & Iacucci, M. (2024) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2024.05.048 · Cited by 30 · see citing works
Infante, A., Gaudino, S., Orsini, F., del Ciello, A., Gullì, C., Merlino, B., Natale, L., Iezzi, R., & Sala, E. (2023) · Clinical Radiology · doi:10.1016/j.crad.2023.11.011 · Cited by 28 · see citing works
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., & Forte, A. J. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13102832 · Cited by 27 · see citing works
Kareemi, H., Yadav, K., Price, C. M., Bobrovitz, N., Meehan, A., Li, H., Goel, G., Masood, S., Grant, L., Ben‐Yakov, M., et al. (2025) · Academic Emergency Medicine · doi:10.1111/acem.15099 · Cited by 28 · see citing works
Du, X.-S., Zhou, Z., Wang, Y.-F., Chuang, Y.-W., Li, Y.-M., Yang, R., Zhang, W., Wang, X., Chen, X.-Y., Guan, H., et al. (2024) · medRxiv · doi:10.1101/2024.08.11.24311828 · Cited by 36 · see citing works
Clinical applications of large language models in medicine and surgery: A scoping review
Liang, E. N., Pei, S., Staibano, P., & van der Woerd, B. D. (2025) · Journal of International Medical Research · doi:10.1177/03000605251347556 · Cited by 26 · see citing works
Tong, L., Zhang, C., Liu, R., Yang, J. N., & Sun, Z. (2024) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-024-04996-2 · Cited by 24 · see citing works
Xue, Z., Zhang, Y., Gan, W., Wang, H., She, G., & Zheng, X. (2024) · Journal of Medical Internet Research · doi:10.2196/50882 · Cited by 22 · see citing works
Mixed methods assessment of the influence of demographics on medical advice of ChatGPT
Andreadis, K., Newman, D. R., Twan, C., Shunk, A., Mann, D. M., & Stevens, E. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae086 · Cited by 20 · see citing works
Virtual Assistants: A Review of the Next Frontier in AI Interaction
Todericiu, I. A. (2025) · Acta Universitatis Sapientiae Informatica · doi:10.1007/s44427-025-00002-7 · Cited by 18 · see citing works
Generative AI Chatbots Across Domains: A Systematic Review
Aldhafeeri, L., Aljumah, F., Thabyan, F., Alabbad, M., Alshahrani, S., Alanazi, F., & Al-Nafjan, A. (2025) · Applied Sciences · doi:10.3390/app152011220 · Cited by 17 · see citing works
Harada, Y., Sakamoto, T., Sugimoto, S., & Shimizu, T. (2024) · JMIR Formative Research · doi:10.2196/53985 · Cited by 17 · see citing works
Akbasli, I. T., Bırbılen, A. Z., & Tekşam, Ö. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02871-6 · Cited by 17 · see citing works
Ćirković, A., & Katz, T. R. (2023) · JMIR Formative Research · doi:10.2196/51798 · Cited by 16 · see citing works
Sheikh, M. S., Barreto, E. F., Miao, J., Thongprayoon, C., Gregoire, J. R., Dreesman, B. J., Erickson, S. B., Craici, I. M., & Cheungpasitporn, W. (2024) · Digital Health · doi:10.1177/20552076241248082 · Cited by 15 · see citing works
Schmieding, M. L., Kopka, M., Bolanaki, M., Napierala, H., Altendorf, M. B., Kuschick, D., Piper, S. K., Scatturin, L., Schmidt, K. F. R., Schorr, C., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64028 · Cited by 14 · see citing works
ChatGPT provides acceptable responses to patient questions regarding common shoulder pathology
Ghilzai, U. M., Fiedler, B., Ghali, A. N., Singh, A., Cass, B., Young, A. A., & Ahmed, A. S. (2024) · Shoulder & Elbow · doi:10.1177/17585732241283971 · Cited by 13 · see citing works
Reis, F., & Lenz, C. (2024) · Cureus · doi:10.7759/cureus.53899 · Cited by 12 · see citing works
Performance of GPT-3.5 and GPT-4 on the Korean Pharmacist Licensing Examination: Comparison Study
Jin, H. K., & Kim, E. Y. (2024) · JMIR Medical Education · doi:10.2196/57451 · Cited by 11 · see citing works
Spatial intelligence and contextual relevance in AI-driven health information retrieval
Yiannakoulias, N. (2024) · Applied Geography · doi:10.1016/j.apgeog.2024.103392 · Cited by 11 · see citing works
Applications of Generative Artificial Intelligence in Electronic Medical Records: A Scoping Review
Morjaria, L., Gandhi, B., Haider, N., Mellon, M., & Sibbald, M. (2025) · Information · doi:10.3390/info16040284 · Cited by 11 · see citing works
Hunt, V. M., Sprehe, L. K., de Lomba, W. C., Gameiro, R. R., Woite, N. L., Wade, C. G., Chen, J. H., Rougas, S. C., Fraser, H. S. F., Sullivan, A. M., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02761-9 · Cited by 10 · see citing works
Self-citationYun, H. S., & Bickmore, T. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720239 · Cited by 10 · see citing works
Diagnosis and Triage Performance of Contemporary Large Language Models on Short Clinical Vignettes
Xu, L., Zhao, W., & Huang, X. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02284-y · Cited by 14 · see citing works
Salehin, I., Sajib, M. T. A., Badhon, N. H., Rifat, M. S. H., Amin, N., & Moon, N. N. (2025) · Engineering Reports · doi:10.1002/eng2.70365 · Cited by 9 · see citing works
The Utility and Limitations of Artificial Intelligence-Powered Chatbots in Healthcare
Hayat, J., Lari, M., Alherz, M., & Lari, A. (2024) · Cureus · doi:10.7759/cureus.73127 · Cited by 11 · see citing works
Brochu, B. M., Mirsky, N. A., & Thaller, S. R. (2024) · Artificial Intelligence Surgery · doi:10.20517/ais.2024.61 · Cited by 9 · see citing works
Contrasting rule and machine learning based digital self triage systems in the USA
Naved, B. A., & Luo, Y. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01367-3 · Cited by 9 · see citing works
Role of artificial intelligence in advancing immunology
Alanazi, H. H. (2025) · Immunologic Research · doi:10.1007/s12026-025-09632-7 · Cited by 9 · see citing works
Comparing the accuracy of large language models and prompt engineering in diagnosing realworld cases
Yao, G., Zhang, W. D., Zhu, Y., Wong, U.-K., Zhang, Y., Cui, Y., Shen, G., Li, Z., & Gao, H. Y. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106026 · Cited by 10 · see citing works
Artificial intelligence‐enabled digital biomedical engineering
Song, P., Tang, X., Lv, X., Reis, R. L., Chen, X., Bai, L., & Su, J. (2025) · BMEMat · doi:10.1002/bmm2.70049 · Cited by 9 · see citing works
Chen, M., Wu, Y., Ma, J., Jia, X., Gao, C., Zhao, F., & Qiao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02409-8 · Cited by 13 · see citing works
ChatGPT4’s diagnostic accuracy in inpatient neurology: A retrospective cohort study
Cano-Besquet, S., Rice-Canetto, T. E., Abou‐El‐Hassan, H., Alarcon, S., Zimmerman, J., Issagholian, L., Salomon, N., Rojas, I., Dhahbi, J. M., & Neeki, M. M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e40964 · Cited by 8 · see citing works
Giuffrè, M., You, K., Pang, Z., Kresevic, S., Chung, S., Chen, R., Ko, Y., Chan, C., Saarinen, T., Ajčević, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01589-z · Cited by 8 · see citing works
A scalable framework for evaluating health language models
Mallinar, N., Heydari, A. A., Liu, X., Faranesh, A. Z., Winslow, B. D., Hammerquist, N., Graef, B., Speed, C., Malhotra, M., Patel, S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02492-x · Cited by 14 · see citing works
Evaluating Large Language Models for Burning Mouth Syndrome Diagnosis
Suga, T., Uehara, O., Abiko, Y., & Toyofuku, A. (2025) · Journal of Pain Research · doi:10.2147/jpr.s509845 · Cited by 7 · see citing works
Using conversant artificial intelligence to improve diagnostic reasoning: ready for prime time?
Scott, I. A., Miller, T., & Crock, C. (2024) · The Medical Journal of Australia · doi:10.5694/mja2.52401 · Cited by 6 · see citing works
Jindal, A., Brandão‐de‐Resende, C., Neo, Y. N., Melo, M., & Day, A. C. (2024) · Eye · doi:10.1038/s41433-024-03070-9 · Cited by 6 · see citing works
Meyer, N. S., & Meyer, J. W. (2025) · Cureus · doi:10.7759/cureus.81802 · Cited by 6 · see citing works
Reis, F., Agha-Mir-Salim, L., Hickstein, R., Reis, M., Piper, S. K., Balzer, F., & Boie, S. D. (2026) · Journal of Medical Internet Research · doi:10.2196/84668 · Cited by 8 · see citing works
Evaluating the accuracy of ChatGPT model versions for giving care-seeking advice
Kopka, M., He, L., & Feufel, M. A. (2026) · Communications Medicine · doi:10.1038/s43856-026-01466-0 · Cited by 5 · see citing works
Is ChatGPT a Reliable Tool for Explaining Medical Terms?
Oztermeli, A. D. (2025) · Cureus · doi:10.7759/cureus.77258 · Cited by 5 · see citing works
Wang, T.-H., Jheng, J.-C., Tseng, Y.-T., Chen, L., & Chen, Y. (2025) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000001273 · Cited by 5 · see citing works
Kaboudi, N., Firouzbakht, S., Eftekhar, M. S., Fayazbakhsh, F., Joharivarnoosfaderani, N., Ghaderi, S., Dehdashti, M., Kia, Y. M., Afshari, M., Gharamaleki, M. V., et al. (2024) · medRxiv · doi:10.22037/aaem.v12i1.2384 · Cited by 42 · see citing works
Zhang, J., Zhou, J., Zhou, L., & Ba, Z. (2025) · JMIR Medical Informatics · doi:10.2196/70096 · Cited by 4 · see citing works
Comparison of triage performance among DRP tool, ChatGPT, and outpatient rehabilitation doctors
Zou, Y., Ye, R., Gao, Y., Zhou, J., Li, Y., Chen, W., Zha, F., & Wang, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-99216-0 · Cited by 3 · see citing works
Shostakovych-Koretska, L. R. (2025) · Інфекційні хвороби · doi:10.11603/1681-2727.2025.2.15302 · Cited by 3 · see citing works
Weerasekara, T. B., Chandeepa, C., Amarasuriya, O. S., & Hettiarachchi, C. (2025) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1145/3721201.3725431 · Cited by 3 · see citing works
Gao, S., Yu, M., Zheng, Y., Zhang, M., Yang, Z., & Zhang, J. (2026) · BMC Emergency Medicine · doi:10.1186/s12873-026-01521-y · Cited by 3 · see citing works
ВИКОРИСТАННЯ ШТУЧНОГО ІНТЕЛЕКТУ В КЛІНІЧНІЙ МЕДИЦИНІ Й НАУКОВИХ ДОСЛІДЖЕННЯХ
Shostakovych-Koretska, L. R., & Kopcha, V. S. (2025) · Медична освіта · doi:10.11603/m.2414-5998.2025.1.15382 · Cited by 3 · see citing works
Pankow, A., Meißner-Bendzko, N., Kaufeld, J. K., Fouquette, L., Cotte, F., Gilbert, S., Türk, E., Das, A. M., Terkamp, C., Burmester, G.-R., et al. (2025) · JMIR AI · doi:10.2196/55001 · Cited by 3 · see citing works
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2024) · medRxiv · doi:10.1101/2024.09.13.24313657 · Cited by 3 · see citing works
Tanaka, C., Kinoshita, T., Okada, Y., Satoh, K., Homma, Y., Suzuki, K., Yokobori, S., Oda, J., Otomo, Y., & Tagami, T. (2025) · Acute Medicine & Surgery · doi:10.1002/ams2.70042 · Cited by 2 · see citing works
Advancing medical AI through benchmarking and competition for specialty triage
Ding, C., Bian, M., Yuan, M., Jiang, L., Luo, K., Chen, P., JIANG, Y., & Xu, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02433-8 · Cited by 2 · see citing works
Kopka, M., & Feufel, M. A. (2026) · JMIR Biomedical Engineering · doi:10.2196/88053 · Cited by 2 · see citing works
Ding, W., & Wang, J. (2026) · Journal of Medical Internet Research · doi:10.2196/95072 · Cited by 2 · see citing works
A Conversational Agent for Handling Health Report Inquiries
Carchiolo, V., Malgeri, M., & Sapari, L. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_15 · Cited by 2 · see citing works
Gilardi, N., Ballabio, M., Ravera, F., Ferrando, L., Stabile, M., Bellodi, A., Talerico, G., Cigolini, B., Genova, C., Carbone, F., et al. (2025) · European Journal of Clinical Investigation · doi:10.1111/eci.70113 · Cited by 2 · see citing works
Lin, Y., Wang, Y., Deng, Y., Xu, D., & Wang, M. (2024) · doi:10.2196/preprints.60412 · Cited by 1 · see citing works
Langmann, E., Henking, T., Joos, S., Klemmt, M., Mueller, R., Preiser, C., Ranisch, R., Koch, R., Rieger, M. A., Wetzel, A.-J., et al. (2025) · Ethik in der Medizin · doi:10.1007/s00481-025-00851-y · Cited by 3 · see citing works
Enhancing Elderly Patients' Decision-Making and Experience in Hospitals through Virtual Agents
Liu, M., & Su, Y. (2024) · International Symposium of Chinese CHI (Chinese CHI) · doi:10.1145/3758871.3758930 · Cited by 1 · see citing works
Wong, A., Roberts, M., Pantangco, M. J. A. P. V., Arnold, A., Philp, A., Šlapeta, J. R., & Livingstone, S. (2025) · Veterinary Record · doi:10.1002/vetr.6126 · Cited by 1 · see citing works
Personalized healthcare information delivery using artificial intelligence
Rana, H., Gohel, S., Patel, D., Sharma, M., Gandhi, N., & Suryawanshi, M. V. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-43823-3.00018-2 · Cited by 1 · see citing works
Xu, Y., Prentice, C., Rueda, S. T., Meczner, A., Multmeier, J., Wickham, A. P., Kelly, L., Klepchukova, A., Stsefanovich, H., & Zhaunova, L. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02332-4 · Cited by 1 · see citing works
AI-Powered Symptom Assessment and User Experience: A Case Study of Simtomi and Simtomi-Care
Lee, J., Lee, C. H., Lee, H., Kim, S., Lee, B. H., Shin, M., Shin, H., & Park, J. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536388 · Cited by 1 · see citing works
Cui, L., Wang, M., Xu, Y., Wei, W., & Zhao, X. (2026) · BMC Emergency Medicine · doi:10.1186/s12873-026-01639-z · Cited by 1 · see citing works
Isemann, C. R., Burresi, S., Innocenti, S., & Righi, L. (2026) · International Emergency Nursing · doi:10.1016/j.ienj.2026.101868 · Cited by 1 · see citing works
Hosseini‐Monfared, P., Amiri, S., Mirahmadi, A., Shahbazi, A., Alamian, A., Azizi, M., & Kazemi, S.-M. (2025) · Archives of Academic Emergency Medicine · doi:10.22037/aaemj.v13i1.2580 · Cited by 1 · see citing works
ChatGPT Is Still Not Good Enough at Giving Care-Seeking Advice, or Is It?Preprint
Kopka, M., He, L., & Feufel, M. A. (2025) · medRxiv · doi:10.1101/2025.05.13.25327519 · Cited by 1 · see citing works
Pankow, A., Meißner-Bendzko, N., Kaufeld, J. K., Fouquette, L., Cotte, F., Gilbert, S., Türk, E., Das, A. M., Terkamp, C., Burmester, G.-R., et al. (2023) · doi:10.2196/preprints.55001
Mayerhoffer, H. (2024) · Croatian Nursing Journal · doi:10.24141/2/8/1/3
Artificial Intelligence, Tasks, Skills, and Wages: Worker-Level Evidence from GermanyPreprint
Koch, M. R., Engberg, E., Lodefalk, M., & Schroeder, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4883794
Wang, L., Li, J., Zhuang, B., Huang, S., Wang, C., Li, W., Fang, M., & Gong, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4888108
Chatting together: Using AI chatbots to improve diagnostic excellence
Wu, A. W. (2024) · Journal of Patient Safety and Risk Management · doi:10.1177/25160435241292868
Samadpour, H., Kalhori, S. R. N., Tahmasebi, M., Rakhshi, M., & Lotfi, F. (2024) · Research Square · doi:10.21203/rs.3.rs-5429142/v1
Best practice for supply chain in LLM-assisted medical applications
Zhao, S., & Wang, J. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731748
Morris, R. J. T. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-28969-9.00012-1
Conventional Versus Transepithelial Photorefractive Keratectomy: A Review of Clinical Outcomes
Aalam, W. A. (2025) · The Open Ophthalmology Journal · doi:10.2174/0118743641442674251009095047
Machine Learning Driven Symptoms Diagnosis System
Doughan, Z. F., Hammoud, R., Ghalayini, M.-D., Darweesh, D., Chehade, O. E. H., & Itani, S. (2025) · International Conference on Advances in Biomedical Engineering (ICABME) · doi:10.1109/icabme66883.2025.11211623
EdgeCare: Privacy-Preserving Medical Advising System on Mobile Devices
Weerasekara, T. B., Chandeepa, C., Amarasooriya, O., & Hettiarachchi, C. (2025) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon67903.2025.11217132
- Authors
- Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R.
- Affiliations
- Brown University (US)
- Yale University (US)
- Maine Medical Center (US)
- Corresponding author
- Fraser, Hamish
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2023-10-03
- Volume
- 11
- Pages
- e49995
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC10582809
- OpenAlex
- W4386172820
- Cited by
- 128
- References
- 31
- Citations by year
- 2026: 36 · 2025: 51 · 2024: 38 · 2023: 3
- FWCI
- 5.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Brown University
- Agency for Healthcare Research and Quality (R01 HS029513)
- National Heart, Lung, and Blood Institute (K23HL140081)
- National Institute of General Medical Sciences (P20GM139664)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- symptom checkers, ChatGPT, large language models, emergency department triage, diagnostic accuracy, WebMD, physician diagnosis, self-reported symptoms, misdiagnosis, diagnostic decision support, emergency department visits, patient-generated health data, emergency department overcrowding
- MeSH terms
- Emergency Service, Hospital, Health Personnel, Humans, Physicians, Triage, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
