The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow
Open accessAbstract
IMPORTANCE: Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated. OBJECTIVE: To evaluate ChatGPT's capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. DESIGN: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. SETTING: ChatGPT, a publicly available LLM. PARTICIPANTS: Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation. EXPOSURES: MSD Clinical Manual vignettes. MAIN OUTCOMES AND MEASURES: We measured the proportion of correct responses to the questions posed within the clinical vignettes tested. RESULTS: ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions. CONCLUSIONS AND RELEVANCE: ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, the use of AI chatbots in nursing education faces multifaceted challenges, including technological limitations, ethical considerations, and social acceptance (59).”
From Digital affordances of AI chatbots in nursing education: a systematic review of learning gains and gaps in the evidence (2026) · cites this paper as [59]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Unequal Treatment—Confronting Racial and Ethnic Disparities in Healthcare
Osundeko, O. (2004) · PubMed Central · Cited by 797
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 565
Artificial intelligence and algorithmic bias: implications for health systems
Panch, T., Mattie, H., & Atun, R. (2019) · Journal of Global Health · doi:10.7189/jogh.09.020318 · Cited by 533
Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023) · JAMA · doi:10.1001/jama.2023.1344 · Cited by 420
Research works printed but not linked: 1
- Case studies. Merck Manuals Professional Edition. Accessed February 1, 2023. https://www.merckmanuals.com/professional/pages-with-widgets/case-studies?mode=list
Cited by
Sorted by citations
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
“• Clinical decision-making: Rao et al [6] • Cancer screening: Rao et al [7] compared ChatGPT responses with the American College of Radiology appropriateness criteria for breast pain and breast cancer screening.”
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
“Recently, there has been great interest in using the nascent but powerful chatbot for clinical decision support [16-20].”
Self-citationHaltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396
“Present-day LLMs, such as ChatGPT, are considered to have a promising accuracy in clinical decision-making, 11,12 diagnosis, 13 symptom-assessment, and triage-advise.”
BackgroundLarge language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 430
“On the basis of the current body of literature, this technology has many potential applications, from administrative tasks to gathering and enhancing medical knowledge ( Table ).”
BackgroundChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Yu, P., Xu, H., Hu, X., & Deng, C. (2023) · Healthcare · doi:10.3390/healthcare11202776 · Cited by 296
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187
“…medical questions (e.g. one study showed ChatGPT outright refused to provide a diagnosis in such situations despite established standards of care [33], racial and cultural biases inherent in training datasets that can cause patient harm, LLMs’ inabilities to account for the complexities of body…”
BackgroundTemsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179
“Early research highlighted both potentially promising and concerning healthcare applications for ChatGPT [3,4].”
BackgroundEvaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 164
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121186 · Cited by 171
Rau, A., Rau, S., Zöller, D., Fink, A., Tran, H. D., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 159
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Assessing the capabilities of ChatGPT to improve additive manufacturing troubleshooting
Badini, S., Regondi, S., Frontoni, E., & Pugliese, R. (2023) · Advanced Industrial and Engineering Polymer Research · doi:10.1016/j.aiepr.2023.03.003 · Cited by 153
The Role of Large Language Models in Transforming Emergency Medicine: Scoping Review
Preiksaitis, C., Ashenburg, N., Bunney, G., Chu, A., Kabeer, R., Riley, F., Ribeira, R., & Rose, C. (2024) · JMIR Medical Informatics · doi:10.2196/53787 · Cited by 134
“A model’s ability to identify subtle patterns in data could offer additional diagnostic support by recommending or interpreting laboratory and imaging studies [30,51,52,61].”
BackgroundMethodInfluentialAlkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132
“Our findings are consistent with other studies that show an inconsistent and fragmented approach to teaching AI and digital technology [27].”
ResultVaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Bolzoni, A. R., Committeri, U., Crimi, S., et al. (2023) · Otolaryngology · doi:10.1002/ohn.489 · Cited by 123
Rokhshad, R., Zhang, P., Mohammad‐Rahimi, H., Pitchika, V., Entezari, N., & Schwendicke, F. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.104938 · Cited by 120
Chiesa-Estomba, C. M., Lechien, J. R., Vaira, L. A., Brunet, A., Cammaroto, G., Mayo‐Yáñez, M., Barrueco, Á. S., & Saga-Gutiérrez, C. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08104-8 · Cited by 106
“As highlighted by Rao et al., these limitations become evident in the frequent instances where Chat-GPT provides futile recommendations or fails to provide a diagnosis despite having all the necessary information [7].”
Hoppe, J. M., Auer, M. K., Strüven, A. K., Maßberg, S., & Stremmel, C. (2024) · Journal of Medical Internet Research · doi:10.2196/56110 · Cited by 100
“The rate of correctly identifying a diagnosis among the top 5 suggested differential diagnoses was slightly over 80% for both physicians and ChatGPT [7-9].”
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83
Benefits, limits, and risks of ChatGPT in medicine
Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Wang, M., Hacking, S. M., Glicksberg, B. S., Strauß, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1518049 · Cited by 88
“When tested against the Merck Sharp & Dohme Clinical Manual vignettes, ChatGPT demonstrated 71.7% overall accuracy in clinical decision-making across 36 cases (Rao et al., 2023).”
InfluentialSpennemann, D. H. R. (2023) · Knowledge · doi:10.3390/knowledge3030032 · Cited by 93
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71
“physicians when dealing with case vignettes of conditions with variable severity 6,7 .”
BackgroundChatGPT and Physicians’ Malpractice Risk
Mello, M. M., & Guha, N. (2023) · JAMA Health Forum · doi:10.1001/jamahealthforum.2023.1938 · Cited by 80
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J. P., Breuckmann, F., & Sellmann, T. (2023) · Journal of Medical Internet Research · doi:10.2196/49368 · Cited by 80
“Preliminary evidence shows that ChatGPT has promising applications across the clinical workflow, especially regarding clinical decision-making, narrowing the final diagnosis, and clinical management [5].”
The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice
Shoja, M. M., Van de Ridder, J. M. M., & Rajput, V. (2023) · Cureus · doi:10.7759/cureus.40883 · Cited by 69
A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research
Bagde, H. S., Dhopte, A., Alam, M. K., & Basri, R. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23050 · Cited by 74
“Rao et al. [35] assessed ChatGPT ’ s accuracy in answering clinical reasoning questions and reported an accuracy range of 43 % – 81 % across 20 simulations.”
BackgroundMendel, T., Singh, N., Mann, D. M., Wiesenfeld, B., & Nov, O. (2025) · Journal of Medical Internet Research · doi:10.2196/64290 · Cited by 64
“Numerous studies have explored doctors’ evaluation of LLMs as an information resource and diagnostic aid [13-18] and some research has begun to consider laypeople’s attitudes toward and use of these tools, such as concerning ethical considerations, whether they are differentiable from doctor’s…”
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 68
The Intersection of ChatGPT, Clinical Medicine, and Medical Education
Wong, R. S.-Y., Ming, L. C., & Ali, R. A. R. (2023) · JMIR Medical Education · doi:10.2196/47274 · Cited by 62
“For instance, the use of ChatGPT in making diagnoses and patient management using clinical vignettes may enhance student learning experience and increase accessibility to learning resources [26].”
MethodTanaka, O. M., Gasparello, G. G., Hartmann, G. C., Casagrande, F. A., & Pithon, M. M. (2023) · Dental Press Journal of Orthodontics · doi:10.1590/2177-6709.28.5.e2323183.oar · Cited by 54
Generative Artificial Intelligence, Human Agency and the Future of Cultural Heritage
Spennemann, D. H. R. (2024) · Heritage · doi:10.3390/heritage7070170 · Cited by 88
Almansour, M., & Alfhaid, F. M. (2024) · Medicine · doi:10.1097/md.0000000000038955 · Cited by 68
“[27] Nonetheless, it seems that the initial differential diagnosis generation is a difficult domain, since these algorithms frequently exhibit subpar performance.”
BackgroundRusse, M. F., Fink, A. M., Ngo, H., Tran, H. D., Bamberg, F., Reisert, M., & Rau, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-41512-8 · Cited by 53
On the limitations of large language models in clinical diagnosisPreprint
Reese, J., Daniš, D., Caufield, J. H., Groza, T., Casiraghi, E., Valentini, G., Mungall, C., & Robinson, P. N. (2023) · medRxiv · doi:10.1101/2023.07.13.23292613 · Cited by 47
Cirone, K. D., Akrout, M., Abid, L., & Oakley, A. M. (2024) · JMIR Dermatology · doi:10.2196/55508 · Cited by 43
Pagano, S., Holzapfel, S., Kappenschneider, T., Meyer, M., Maderbacher, G., Grifka, J. K., & Holzapfel, D. E. (2023) · Journal of Orthopaedics and Traumatology · doi:10.1186/s10195-023-00740-4 · Cited by 44
“However, reference to the work of Rao et al. [16] is enlightening.”
MethodRao, A. S., Kim, J., Lie, W., Pang, M., Fuh, L., Dreyer, K. J., & Succi, M. D. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02058-y · Cited by 43
“It has shown impressive accuracy in a number of specialized applications, including in medicine, displaying remarkable accuracy in tasks such as the choice of appropriate imaging modalities in emergent settings to diagnosis and clinical management in clinical vignettes [8–10].”
Self-citationThe impact of Chat Generative Pre-trained Transformer (ChatGPT) on medical education
Heng, J. J. Y., Teo, D. B. S., & Tan, L. F. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad058 · Cited by 41
Al-Sharif, E. M., Penteado, R. C., Jalbout, N. D. E., Topilow, N. J., Shoji, M. K., Kikkawa, D. O., Liu, C. Y., & Korn, B. S. (2024) · Ophthalmic Plastic and Reconstructive Surgery · doi:10.1097/iop.0000000000002567 · Cited by 40
ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., et al. (2024) · European Spine Journal · doi:10.1007/s00586-024-08198-6 · Cited by 37
“Chat-GPT is distinct from non-generative AI models in its ability to generate novel content, making it a valuable resource for clinicians exploring its potential to synthesize scientific literature for clinical decision-making [7–9].”
Self-citationLiu, F., Chang, X., Zhu, Q., Huang, Y., Li, Y., & Wang, H. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06232-1 · Cited by 35
“LLMs have attracted more and more attention in medical education and their transformative implications make their application seemingly inevitable [87–89].”
BackgroundHamed, E. S., Eid, A. M., & Alberry, M. (2023) · Cureus · doi:10.7759/cureus.38784 · Cited by 38
Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care
Koranteng, E., Rao, A. S., Flores, E. J., Lev, M. H., Landman, A., Dreyer, K. J., & Succi, M. D. (2023) · JMIR Medical Education · doi:10.2196/51199 · Cited by 39
Self-citationRisk stratification of thyroid nodules: Assessing the suitability of ChatGPT for text-based analysis
Sievert, M., Conrad, O., Mueller, S. K., Rupp, R., Balk, M., Richter, D., Mantsopoulos, K., Iro, H., & Koch, M. (2023) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2023.104144 · Cited by 33
Mohammadi, S. S., & Nguyen, Q. D. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100495 · Cited by 35
Karakas, C., Brock, D. C., & Lakhotia, A. (2023) · Pediatric Neurology · doi:10.1016/j.pediatrneurol.2023.08.035 · Cited by 28
Şişman, A. Ç., & Acar, A. H. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05732-w · Cited by 28
“It is still uncertain whether these chatbots can consistently offer reliable information to healthcare professionals and assist them in making informed clinical decisions [10, 16, 17].”
A Future of Smarter Digital Health Empowered by Generative Pretrained Transformer
Miao, H., Li, C., & Wang, J. (2023) · Journal of Medical Internet Research · doi:10.2196/49963 · Cited by 26
“Further examples of ChatGPT’s applications in health care analytics and decision-making include the work of Hirosawa et al [30] for differential diagnosis list generation and that of Rao et al [31,32] for clinical decision-making support including cancer screening.”
ChatGPT in Occupational Medicine: A Comparative Study with Human Experts
Padovan, M., Cosci, B., Petillo, A., Nerli, G., Porciatti, F., Scarinci, S., Carlucci, F., Dell’Amico, L., Meliani, N., Necciari, G., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11010057 · Cited by 28
“ChatGPT could also be of valuable assistance to physicians in their clinical decision-making [21] by generating accurate lists of differential diagnoses [22] and providing insights for cancer screening decisions [23].”
BackgroundDiagnosis of malignancy in oropharyngeal confocal laser endomicroscopy using GPT 4.0 with vision
Sievert, M., Aubreville, M., Mueller, S. K., Eckstein, M., Breininger, K., Iro, H., & Goncalves, M. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08476-5 · Cited by 25
“Standardized diagnosis is also possible by designing chat-bots to adhere to established medical guidelines, reducing human error and variability in interpretations and potentially enhancing diagnostic outcomes [23, 24].”
Racial, ethnic, and sex bias in large language model opioid recommendations for pain management
Young, C. C., Enichen, E. J., Rao, A. S., & Succi, M. D. (2024) · Pain · doi:10.1097/j.pain.0000000000003388 · Cited by 26
Self-citationWill ChatGPT be Able to Replace a Spine Surgeon in the Clinical Setting?
Chalhoub, R., Mouawad, A., Aoun, M., Daher, M., El-Sett, P., Kreichati, G. E., Kharrat, K. E., & Sebaaly, A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.02.101 · Cited by 24
Spennemann, D. H. R. (2025) · Publications · doi:10.3390/publications13010012 · Cited by 23
Exhibiting the Heritage of COVID-19—A Conversation with ChatGPT
Spennemann, D. H. R. (2023) · Heritage · doi:10.3390/heritage6080302 · Cited by 26
Bias Sensitivity in Diagnostic Decision-Making: Comparing ChatGPT with Residents
Schmidt, H. G., Rotgans, J. I., & Mamede, S. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-024-09177-9 · Cited by 22
Aljamaan, F., Malki, K. H., Alhasan, K. A., Jamal, A. A., Altamimi, I., Khayat, A., Alhaboob, A. A. N., Abdulmajeed, N., Alshahrani, F. S., Saad, K., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e28962 · Cited by 25
“AI chatbots, such as ChatGPT, have the potential to revolutionize the way healthcare is delivered [7 – 9].”
BackgroundZhou, Y., Moon, C. N., Szatkowski, J., Moore, D., & Stevens, J. M. (2023) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-023-03742-4 · Cited by 18
“In another study, Rao et al. demonstrated the ability of ChatGPT to accurately generate differential diagnoses, suggest appropriate diagnostic tests, and reasonably deduce final diagnoses using medical vignettes published in the Merck Sharpe and Dohme (MSD) clinical manual [9].”
MethodAlmekkawi, A. K., Caruso, J. P., Anand, S., Hawkins, A., Rauf, R., Al-Shaikhli, M., Aoun, S. G., & Bagley, C. A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.11.114 · Cited by 18
Makhoul, M., Melkane, A. E., Khoury, P. E., Hadi, C. E., & Matar, N. E. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08509-z · Cited by 18
“Recent advances in artificial intelligence (AI) have given rise to powerful linguistic models such as Chat-Based Generative Pre-Trained Transformer (ChatGPT) [1, 2].”
AI in radiography education: Evaluating multiple-choice questions difficulty and discrimination
Emekli, E., & Karahan, B. N. (2025) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2025.101896 · Cited by 20
Sawamura, S., Bito, T., Ando, T., Masuda, K., Kameyama, S., & Ishida, H. (2024) · Journal of Physical Therapy Science · doi:10.1589/jpts.36.234 · Cited by 20
“Therefore, it can be used as a support tool but cannot replace professionals’ specialized knowledge and experience for use in the medical field 5, 37, 46) .”
MethodPotential of ChatGPT in facilitating research in radiation oncology?
Gückenberger, M., Andratschke, N. H., Ahmadsei, M., Christ, S. M., Heusel, A. E., Kamal, S., Kroese, T. E., Looman, E. L., Reichl, S., Badra, E. V., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.109894 · Cited by 19
Two artificial intelligence models underperform on examinations in a veterinary curriculum
Coleman, M. C., & Moore, J. N. (2024) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.23.12.0666 · Cited by 16
Young, C. C., Enichen, E. J., Rao, A. S., Hilker, S., Butler, A. M., Laird, J., & Succi, M. D. (2024) · medRxiv · doi:10.1101/2024.08.06.24311544 · Cited by 16
Self-citationSpennemann, D. H. R. (2023) · Preprints.org · doi:10.20944/preprints202308.1271.v1 · Cited by 14
Potential Benefits and Perils of Incorporating ChatGPT to the Movement Disorders Clinic
Deik, A. (2023) · Journal of Movement Disorders · doi:10.14802/jmd.23072 · Cited by 13
Harnessing the open access version of ChatGPT for enhanced clinical opinions
Tenner, Z. M., Cottone, M., & Chavez, M. R. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000355 · Cited by 13
Liang, Z., Li, J., Tang, Y.-H., Zhang, Y., Chen, C.-Y., Li, S., Wang, X., Xu, X., Zhuang, Z., He, S., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-69735-3 · Cited by 12
Abhari, S., Afshari, Y., Fatehi, F., Salmani, H., Garavand, A., Chumachenko, D., Zakerabasali, S., & Morita, P. P. (2024) · Annals of Medicine and Surgery · doi:10.1097/ms9.0000000000002716 · Cited by 13
“Firstly, there was a signi fi cant emphasis on comparing ChatGPT ’ s performance with expert opinion and physicians ’ decision -making [37,38,40,49] .”
Foundation Models: From Current Developments, Challenges, and Risks to Future Opportunities
Hussain, A., Ali, S., Farwa, U. E., Mozumder, M. A. I., & Kim, H.-C. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936649 · Cited by 16
“…particular, it is important to emphasize the great potential of FMs since they are capable of offering rather high-quality data, which may greatly contribute to the overall elimination of problems related to data deficit, abundance, and privacy in the sphere of medicine and healthcare [3, 16, 17].”
Rao, A. S., Mu, A., Enichen, E. J., Gupta, D., Hall, N., Koranteng, E., Marks, W., Senter-Zapata, M. J., Whitehead, D. C., White, B. A., et al. (2025) · Annals of Biomedical Engineering · doi:10.1007/s10439-025-03701-6 · Cited by 13
“Already, LLM-based tools have achieved impressive performance in the diagnosis and management of various conditions and demonstrated utility in improving the efficiency of clinical workflows [8, 12, 15–19, 28 –30 ].”
Self-citationHow Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?
Tunçer, G., & Güçlü, K. G. (2024) · Infectious Diseases and Clinical Microbiology · doi:10.36519/idcm.2024.286 · Cited by 10
Battle of the Bots: Solving Clinical Cases in Osteoarticular Infections With Large Language Models
Borgonovo, F., Matsuo, T., Petri, F., Amin Alavi, S. M., Mazudie Ndjonko, L. C., Gori, A., & Berbari, E. F. (2025) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2025.100230 · Cited by 10
The Utility and Limitations of Artificial Intelligence-Powered Chatbots in Healthcare
Hayat, J., Lari, M., Alherz, M., & Lari, A. (2024) · Cureus · doi:10.7759/cureus.73127 · Cited by 11
Cheng, H. Y. (2024) · JMIR Formative Research · doi:10.2196/63494 · Cited by 9
ChatGPT and other artificial intelligence chatbots and biomedical writing
Gurha, P., Ishaq, N., & Marian, A. J. (2023) · The Journal of Cardiovascular Aging · doi:10.20517/jca.2023.13 · Cited by 9
Evaluation of a novel large language model (LLM)-powered chatbot for oral boards scenarios
Silvestri, C. J., Roshal, J. A., Shah, M. M., Widmann, W. D., Townsend, C. M., Brian, R., L’Huillier, J. C., Navarro, S. M., Lund, S. B., & Sathe, T. S. (2024) · Global Surgical Education - Journal of the Association for Surgical Education · doi:10.1007/s44186-024-00303-z · Cited by 15
“These models have been employed to generate ABS In-Training Examination (ABSITE) questions [22], create diagnostic and patient management vignettes [23], and simulate virtual patients for practicing history taking skills [24].”
Optimizing ChatGPT’s Interpretation and Reporting of Delirium Assessment Outcomes: Exploratory Study
Choi, Y. K., Lin, S., Fick, D. M., Shulman, R. W., Lee, S., Shrestha, P., & Santoso, K. P. (2024) · JMIR Formative Research · doi:10.2196/51383 · Cited by 9
“A study by Rao et al [6] delved deeper into ChatGPT’s potential examining its capacity to support comprehensive iterative clinical reasoning through sequential prompts, demonstrating ChatGPT’s ability to incorporate information from earlier parts of a conversation into downstream responses.”
BackgroundMethodAlShenaiber, A., Datta, S., Mosa, A. J., Binhammer, P., & Ing, E. B. (2024) · Journal of Hand Surgery Global Online · doi:10.1016/j.jhsg.2024.07.011 · Cited by 8
ChatGPT in medical literature – a concise review and SWOT analysisPreprint
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J. P., Breuckmann, F., & Sellmann, T. (2023) · medRxiv · doi:10.1101/2023.05.06.23289608 · Cited by 8
Spennemann, D. H. R. (2023) · Publications · doi:10.3390/publications11030045 · Cited by 17
Assessing ChatGPT's use of person-first language in healthcare conversations
Hackl, E. V. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00099-9 · Cited by 10
Chatbots vs andrologists: Testing 25 clinical cases
Perrot, O., Schirmann, A., Vidart, A., Guillot‐Tantay, C., Izard, V., Lebrét, T., Boillot, B., Mesnard, B., Lebacle, C. M., & Madec, F.-X. (2024) · The French Journal of Urology · doi:10.1016/j.fjurol.2024.102636 · Cited by 7
Assessing the research landscape and clinical utility of large language models: A scoping reviewPreprint
Park, Y., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. T. (2023) · Research Square · doi:10.21203/rs.3.rs-3472000/v1 · Cited by 8
Herbert, D. N., Westendorf, J., Farmer, M., & Reeder, B. (2025) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.24-00319 · Cited by 6
Rao, A. S., Prasad, S., Lee, R. S., Farrell, S., McKinley, S. K., & Succi, M. D. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100241 · Cited by 6
Self-citationSengupta, P., Dutta, S., Chakravarthi, S., Jegasothy, R., Jeganathan, R., & Pichumani, A. (2023) · Gynecology and Obstetrics Clinical Medicine · doi:10.1016/j.gocm.2023.09.002 · Cited by 6
Rao, A. S., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.08.026 · Cited by 6
Self-citationValidation of the QAMAI tool to assess the quality of health information provided by AIPreprint
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Boscolo‐Rizzo, P., Califano, G., Cammaroto, G., et al. (2024) · medRxiv · doi:10.1101/2024.01.25.24301774 · Cited by 5
“Similarly, medical professionals must not abdicate their clinical judgment and decision-making responsibilities to AI, despite its capacity to process and analyze vast amounts of data rapidly [17].”
BackgroundMethodKaboudi, N., Firouzbakht, S., Eftekhar, M. S., Fayazbakhsh, F., Joharivarnoosfaderani, N., Ghaderi, S., Dehdashti, M., Kia, Y. M., Afshari, M., Gharamaleki, M. V., et al. (2024) · medRxiv · doi:10.22037/aaem.v12i1.2384 · Cited by 44
“Rao et al. (25) also reported overall accuracy for ChatGPT at 71.7%, with the highest performance in final diagnosis and the lowest in initial differential diagnosis.”
BackgroundInfluentialChatGPT and Higher Education: A Pathway to Unprecedented Progress
Almahasees, Z. M., Awabdeh, A. A., & Mahmoud, S. (2024) · Journal of Higher Education Theory and Practice · doi:10.33423/jhetp.v24i5.7004 · Cited by 10
The Genie Is Out of the Bottle
Morales-Ramirez, P., Mishek, H., & Dasgupta, A. (2023) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000005446 · Cited by 4
Jiang, Y., Zhu, J., Lin, Y.-Y., Su, Z., Zhang, L.-B., Dong, Z., Song, Q., Fan, P., & Li, Z. (2025) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-025-06451-2 · Cited by 2
Precisión de ChatGPT en el diagnóstico de entidades clínicas en el ámbito de la medicina interna
Andrade-Castellanos, C. A., la Paz, M. T. T.-D., & Farfán-Flores, P. E. (2023) · Gaceta Médica de México · doi:10.24875/gmm.23000297 · Cited by 3
Metadata
- Authors
- Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Succi, Marc D.
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-26
- PMC
- PMC9980239
- OpenAlex
- W4322208207
- Cited by
- 134
- References
- 24
- Citations by year
- 2026: 8 · 2025: 20 · 2024: 55 · 2023: 50
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health
- National Institute of General Medical Sciences (T32GM144273)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- ChatGPT, large language models, clinical decision support, differential diagnosis, clinical vignettes, emergency medicine, clinical reasoning, diagnostic accuracy, clinical management, Emergency Severity Index
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
