Assessing the utility of ChatGPT throughout the entire clinical workflow
Open accessAbstract
IMPORTANCE: Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated. OBJECTIVE: To evaluate ChatGPT's capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. DESIGN: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. SETTING: ChatGPT, a publicly available LLM. PARTICIPANTS: Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation. EXPOSURES: MSD Clinical Manual vignettes. MAIN OUTCOMES AND MEASURES: We measured the proportion of correct responses to the questions posed within the clinical vignettes tested. RESULTS: ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions. CONCLUSIONS AND RELEVANCE: ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The use of AI chatbots in nursing education faces multifaceted challenges, including technological limitations, ethical considerations, and social acceptance.
“Overall, the use of AI chatbots in nursing education faces multifaceted challenges, including technological limitations, ethical considerations, and social acceptance (59).”
From Digital affordances of AI chatbots in nursing education: a systematic review of learning gains and gaps in the evidence (2026) · cites this paper as [59]
The Discussion (page unavailable in our copy) describes ChatGPT’s technical limitations, including hallucination and errors in clinical judgment, and raises bias and regulatory considerations. The paper examines clinical vignettes, not nursing education, and the passages do not address social acceptance in that setting. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
Unequal Treatment—Confronting Racial and Ethnic Disparities in Healthcare
Osundeko, O. (2004) · PubMed Central · Cited by 797 · see citing works
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · see citing works
Artificial intelligence and algorithmic bias: implications for health systems
Panch, T., Mattie, H., & Atun, R. (2019) · Journal of Global Health · doi:10.7189/jogh.09.020318 · Cited by 533 · see citing works
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · see citing works
Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023) · JAMA · doi:10.1001/jama.2023.1344 · Cited by 420 · see citing works
The potential of artificial intelligence to improve patient safety: a scoping review
Bates, D. W., Levine, D. M., Syrowatka, A., Kuznetsova, M., Craig, K. J. T., Rui, A., Jackson, G. P., & Rhee, K. B. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00423-6 · Cited by 351 · see citing works
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257 · see citing works
Self-citationEitel, D. R., Rudkin, S., Malvehy, M. A., Killeen, J. P., & Pines, J. M. (2008) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2008.03.038 · Cited by 204 · see citing works
GPT Takes the Bar ExamPreprint
Bommarito, M., & Katz, D. (2022) · Social Science Research Network · doi:10.48550/arxiv.2212.14402 · Cited by 195 · see citing works
Bommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181 · see citing works
Hurtful words: quantifying biases in clinical contextual word embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384448 · Cited by 122 · see citing works
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · see citing works
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, J. A. M., & Katz, D. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322372 · Cited by 68 · see citing works
Kim, D., Chung, J. W., Choi, J., Succi, M. D., Conklin, J., Longo, M. G. F., Ackman, J. B., Little, B. P., Petranović, M., Kalra, M. K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-29437-8 · Cited by 40 · see citing works
Self-citationChonde, D. B., Pourvaziri, A., Williams, J. M., McGowan, J., Moskos, M. M., Alvarez, C., Narayan, A. K., Daye, D., Flores, E. J., & Succi, M. D. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.01.013 · Cited by 38 · see citing works
Self-citationLi, M., Arun, N., Aggarwal, M., Gupta, S., Singh, P., Little, B. P., Mendoza, D. P., Corradi, G. C. A., Takahashi, M. S., Ferraciolli, S. F., et al. (2022) · Medicine · doi:10.1097/md.0000000000029587 · Cited by 26 · see citing works
Self-citationWitowski, J. S., Choi, J., Jeon, S., Kim, D., Chung, J. W., Conklin, J., Longo, M. G. F., Succi, M. D., & Do, S. (2021) · Blockchain in Healthcare Today · doi:10.30953/bhty.v4.176 · Cited by 21 · see citing works
Self-citationReducing Bias in Healthcare Artificial Intelligence
Byrne, M. D. (2021) · Journal of PeriAnesthesia Nursing · doi:10.1016/j.jopan.2021.03.009 · Cited by 20 · see citing works
Chung, J. W., Kim, D., Choi, J., Yune, S., Song, K. D., Kim, S., Chua, M., Succi, M. D., Conklin, J., Longo, M. G. F., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-24721-5 · Cited by 19 · see citing works
Self-citationGPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, D. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.04408 · Cited by 15 · see citing works
O’Shea, A., Li, M., Mercaldo, N. D., Balthazar, P., Som, A., Yeung, T., Succi, M. D., Little, B. P., Kalpathy–Cramer, J., & Lee, S. I. (2022) · BJR|Open · doi:10.1259/bjro.20210062 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Case studies. Merck Manuals Professional Edition. Accessed February 1, 2023. https://www.merckmanuals.com/professional/pages-with-widgets/case- studies?mode=list
Cited by
Sorted by citations
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505 · see citing works
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412 · see citing works
Self-citationHaltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396 · see citing works
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Yu, P., Xu, H., Hu, X., & Deng, C. (2023) · Healthcare · doi:10.3390/healthcare11202776 · Cited by 267 · see citing works
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234 · see citing works
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · see citing works
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179 · see citing works
Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 156 · see citing works
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121186 · Cited by 158 · see citing works
Rau, A., Rau, S., Zöller, D., Fink, A., Tran, H. D., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 152 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Assessing the capabilities of ChatGPT to improve additive manufacturing troubleshooting
Badini, S., Regondi, S., Frontoni, E., & Pugliese, R. (2023) · Advanced Industrial and Engineering Polymer Research · doi:10.1016/j.aiepr.2023.03.003 · Cited by 137 · see citing works
The Role of Large Language Models in Transforming Emergency Medicine: Scoping Review
Preiksaitis, C., Ashenburg, N., Bunney, G., Chu, A., Kabeer, R., Riley, F., Ribeira, R., & Rose, C. (2024) · JMIR Medical Informatics · doi:10.2196/53787 · Cited by 134 · see citing works
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132 · see citing works
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Bolzoni, A. R., Committeri, U., Crimi, S., et al. (2023) · Otolaryngology · doi:10.1002/ohn.489 · Cited by 123 · see citing works
Rokhshad, R., Zhang, P., Mohammad-Rahimi, H., Pitchika, V., Entezari, N., & Schwendicke, F. (2024) · E -journal of dentistry · doi:10.1016/j.jdent.2024.104938 · Cited by 109 · see citing works
Chiesa-Estomba, C. M., Lechien, J. R., Vaira, L. A., Brunet, A., Cammaroto, G., Mayo‐Yáñez, M., Barrueco, Á. S., & Saga-Gutiérrez, C. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08104-8 · Cited by 106 · see citing works
Hoppe, J. M., Auer, M. K., Strüven, A. K., Maßberg, S., & Stremmel, C. (2024) · Journal of Medical Internet Research · doi:10.2196/56110 · Cited by 100 · see citing works
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83 · see citing works
Benefits, limits, and risks of ChatGPT in medicine
Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Wang, M., Hacking, S. M., Glicksberg, B. S., Strauß, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1518049 · Cited by 85 · see citing works
Spennemann, D. H. R. (2023) · Knowledge · doi:10.3390/knowledge3030032 · Cited by 93 · see citing works
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71 · see citing works
ChatGPT and Physicians’ Malpractice Risk
Mello, M. M., & Guha, N. (2023) · JAMA Health Forum · doi:10.1001/jamahealthforum.2023.1938 · Cited by 80 · see citing works
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70 · see citing works
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J. P., Breuckmann, F., & Sellmann, T. (2023) · Journal of Medical Internet Research · doi:10.2196/49368 · Cited by 78 · see citing works
The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice
Shoja, M. M., Van de Ridder, J. M. M., & Rajput, V. (2023) · Cureus · doi:10.7759/cureus.40883 · Cited by 69 · see citing works
A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research
Bagde, H. S., Dhopte, A., Alam, M. K., & Basri, R. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23050 · Cited by 71 · see citing works
Mendel, T., Singh, N., Mann, D. M., Wiesenfeld, B., & Nov, O. (2025) · Journal of Medical Internet Research · doi:10.2196/64290 · Cited by 64 · see citing works
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 62 · see citing works
The Intersection of ChatGPT, Clinical Medicine, and Medical Education
Wong, R. S.-Y., Ming, L. C., & Ali, R. A. R. (2023) · JMIR Medical Education · doi:10.2196/47274 · Cited by 60 · see citing works
Tanaka, O. M., Gasparello, G. G., Hartmann, G. C., Casagrande, F. A., & Pithon, M. M. (2023) · Dental Press Journal of Orthodontics · doi:10.1590/2177-6709.28.5.e2323183.oar · Cited by 54 · see citing works
Generative Artificial Intelligence, Human Agency and the Future of Cultural Heritage
Spennemann, D. H. R. (2024) · Heritage · doi:10.3390/heritage7070170 · Cited by 77 · see citing works
Almansour, M., & Alfhaid, F. M. (2024) · Medicine · doi:10.1097/md.0000000000038955 · Cited by 68 · see citing works
Russe, M. F., Fink, A. M., Ngo, H., Tran, H. D., Bamberg, F., Reisert, M., & Rau, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-41512-8 · Cited by 51 · see citing works
On the limitations of large language models in clinical diagnosisPreprint
Reese, J., Daniš, D., Caufield, J. H., Groza, T., Casiraghi, E., Valentini, G., Mungall, C., & Robinson, P. N. (2023) · medRxiv · doi:10.1101/2023.07.13.23292613 · Cited by 47 · see citing works
Cirone, K. D., Akrout, M., Abid, L., & Oakley, A. M. (2024) · JMIR Dermatology · doi:10.2196/55508 · Cited by 43 · see citing works
Pagano, S., Holzapfel, S., Kappenschneider, T., Meyer, M., Maderbacher, G., Grifka, J. K., & Holzapfel, D. E. (2023) · Journal of Orthopaedics and Traumatology · doi:10.1186/s10195-023-00740-4 · Cited by 42 · see citing works
Rao, A. S., Kim, J., Lie, W., Pang, M., Fuh, L., Dreyer, K. J., & Succi, M. D. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02058-y · Cited by 43 · see citing works
Self-citationThe impact of Chat Generative Pre-trained Transformer (ChatGPT) on medical education
Heng, J. J. Y., Teo, D. B. S., & Tan, L. F. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad058 · Cited by 41 · see citing works
Al-Sharif, E. M., Penteado, R. C., Jalbout, N. D. E., Topilow, N. J., Shoji, M. K., Kikkawa, D. O., Liu, C. Y., & Korn, B. S. (2024) · Ophthalmic Plastic and Reconstructive Surgery · doi:10.1097/iop.0000000000002567 · Cited by 40 · see citing works
ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., et al. (2024) · European Spine Journal · doi:10.1007/s00586-024-08198-6 · Cited by 37 · see citing works
Self-citationLiu, F., Chang, X., Zhu, Q., Huang, Y., Li, Y., & Wang, H. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06232-1 · Cited by 35 · see citing works
Hamed, E. S., Eid, A. M., & Alberry, M. (2023) · Cureus · doi:10.7759/cureus.38784 · Cited by 36 · see citing works
Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care
Koranteng, E., Rao, A. S., Flores, E. J., Lev, M. H., Landman, A., Dreyer, K. J., & Succi, M. D. (2023) · JMIR Medical Education · doi:10.2196/51199 · Cited by 39 · see citing works
Self-citationRisk stratification of thyroid nodules: Assessing the suitability of ChatGPT for text-based analysis
Sievert, M., Conrad, O., Mueller, S. K., Rupp, R., Balk, M., Richter, D., Mantsopoulos, K., Iro, H., & Koch, M. (2023) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2023.104144 · Cited by 33 · see citing works
Mohammadi, S. S., & Nguyen, Q. D. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100495 · Cited by 34 · see citing works
Karakas, C., Brock, D. C., & Lakhotia, A. (2023) · Pediatric Neurology · doi:10.1016/j.pediatrneurol.2023.08.035 · Cited by 28 · see citing works
Şişman, A. Ç., & Acar, A. H. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05732-w · Cited by 27 · see citing works
A Future of Smarter Digital Health Empowered by Generative Pretrained Transformer
Miao, H., Li, C., & Wang, J. (2023) · Journal of Medical Internet Research · doi:10.2196/49963 · Cited by 26 · see citing works
ChatGPT in Occupational Medicine: A Comparative Study with Human Experts
Padovan, M., Cosci, B., Petillo, A., Nerli, G., Porciatti, F., Scarinci, S., Carlucci, F., Dell’Amico, L., Meliani, N., Necciari, G., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11010057 · Cited by 26 · see citing works
Diagnosis of malignancy in oropharyngeal confocal laser endomicroscopy using GPT 4.0 with vision
Sievert, M., Aubreville, M., Mueller, S. K., Eckstein, M., Breininger, K., Iro, H., & Goncalves, M. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08476-5 · Cited by 25 · see citing works
Racial, ethnic, and sex bias in large language model opioid recommendations for pain management
Young, C. C., Enichen, E. J., Rao, A. S., & Succi, M. D. (2024) · Pain · doi:10.1097/j.pain.0000000000003388 · Cited by 24 · see citing works
Self-citationWill ChatGPT be Able to Replace a Spine Surgeon in the Clinical Setting?
Chalhoub, R., Mouawad, A., Aoun, M., Daher, M., El-Sett, P., Kreichati, G. E., Kharrat, K. E., & Sebaaly, A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.02.101 · Cited by 24 · see citing works
Spennemann, D. H. R. (2025) · Publications · doi:10.3390/publications13010012 · Cited by 23 · see citing works
Exhibiting the Heritage of COVID-19—A Conversation with ChatGPT
Spennemann, D. H. R. (2023) · Heritage · doi:10.3390/heritage6080302 · Cited by 25 · see citing works
Bias Sensitivity in Diagnostic Decision-Making: Comparing ChatGPT with Residents
Schmidt, H. G., Rotgans, J. I., & Mamede, S. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-024-09177-9 · Cited by 22 · see citing works
Aljamaan, F., Malki, K. H., Alhasan, K. A., Jamal, A. A., Altamimi, I., Khayat, A., Alhaboob, A. A. N., Abdulmajeed, N., Alshahrani, F. S., Saad, K., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e28962 · Cited by 25 · see citing works
Zhou, Y., Moon, C. N., Szatkowski, J., Moore, D., & Stevens, J. M. (2023) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-023-03742-4 · Cited by 18 · see citing works
Almekkawi, A. K., Caruso, J. P., Anand, S., Hawkins, A., Rauf, R., Al-Shaikhli, M., Aoun, S. G., & Bagley, C. A. (2024) · World Neurosurgery · doi:10.1016/j.wneu.2024.11.114 · Cited by 18 · see citing works
Makhoul, M., Melkane, A. E., Khoury, P. E., Hadi, C. E., & Matar, N. E. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08509-z · Cited by 18 · see citing works
AI in radiography education: Evaluating multiple-choice questions difficulty and discrimination
Emekli, E., & Karahan, B. N. (2025) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2025.101896 · Cited by 17 · see citing works
Sawamura, S., Bito, T., Ando, T., Masuda, K., Kameyama, S., & Ishida, H. (2024) · Journal of Physical Therapy Science · doi:10.1589/jpts.36.234 · Cited by 19 · see citing works
Potential of ChatGPT in facilitating research in radiation oncology?
Gückenberger, M., Andratschke, N. H., Ahmadsei, M., Christ, S. M., Heusel, A. E., Kamal, S., Kroese, T. E., Looman, E. L., Reichl, S., Badra, E. V., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.109894 · Cited by 16 · see citing works
Two artificial intelligence models underperform on examinations in a veterinary curriculum
Coleman, M. C., & Moore, J. N. (2024) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.23.12.0666 · Cited by 16 · see citing works
Young, C. C., Enichen, E. J., Rao, A. S., Hilker, S., Butler, A. M., Laird, J., & Succi, M. D. (2024) · medRxiv · doi:10.1101/2024.08.06.24311544 · Cited by 16 · see citing works
Self-citationSpennemann, D. H. R. (2023) · Preprints.org · doi:10.20944/preprints202308.1271.v1 · Cited by 13 · see citing works
Potential Benefits and Perils of Incorporating ChatGPT to the Movement Disorders Clinic
Deik, A. (2023) · Journal of Movement Disorders · doi:10.14802/jmd.23072 · Cited by 13 · see citing works
Harnessing the open access version of ChatGPT for enhanced clinical opinions
Tenner, Z. M., Cottone, M., & Chavez, M. R. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000355 · Cited by 12 · see citing works
Liang, Z., Li, J., Tang, Y.-H., Zhang, Y., Chen, C.-Y., Li, S., Wang, X., Xu, X., Zhuang, Z., He, S., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-69735-3 · Cited by 12 · see citing works
Abhari, S., Afshari, Y., Fatehi, F., Salmani, H., Garavand, A., Chumachenko, D., Zakerabasali, S., & Morita, P. P. (2024) · Annals of Medicine and Surgery · doi:10.1097/ms9.0000000000002716 · Cited by 13 · see citing works
Foundation Models: From Current Developments, Challenges, and Risks to Future Opportunities
Hussain, A., Ali, S., Farwa, U. E., Mozumder, M. A. I., & Kim, H.-C. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936649 · Cited by 14 · see citing works
Rao, A. S., Mu, A., Enichen, E. J., Gupta, D., Hall, N., Koranteng, E., Marks, W., Senter-Zapata, M. J., Whitehead, D. C., White, B. A., et al. (2025) · Annals of Biomedical Engineering · doi:10.1007/s10439-025-03701-6 · Cited by 12 · see citing works
Self-citationHow Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?
Tunçer, G., & Güçlü, K. G. (2024) · Infectious Diseases and Clinical Microbiology · doi:10.36519/idcm.2024.286 · Cited by 10 · see citing works
Battle of the Bots: Solving Clinical Cases in Osteoarticular Infections With Large Language Models
Borgonovo, F., Matsuo, T., Petri, F., Amin Alavi, S. M., Mazudie Ndjonko, L. C., Gori, A., & Berbari, E. F. (2025) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2025.100230 · Cited by 10 · see citing works
The Utility and Limitations of Artificial Intelligence-Powered Chatbots in Healthcare
Hayat, J., Lari, M., Alherz, M., & Lari, A. (2024) · Cureus · doi:10.7759/cureus.73127 · Cited by 11 · see citing works
Cheng, H. Y. (2024) · JMIR Formative Research · doi:10.2196/63494 · Cited by 9 · see citing works
ChatGPT and other artificial intelligence chatbots and biomedical writing
Gurha, P., Ishaq, N., & Marian, A. J. (2023) · The Journal of Cardiovascular Aging · doi:10.20517/jca.2023.13 · Cited by 9 · see citing works
Evaluation of a novel large language model (LLM)-powered chatbot for oral boards scenarios
Silvestri, C. J., Roshal, J. A., Shah, M., Widmann, W. D., Townsend, C. M., Brian, R., L’Huillier, J. C., Navarro, S. M., Lund, S. B., & Sathe, T. S. (2024) · Global Surgical Education - Journal of the Association for Surgical Education · doi:10.1007/s44186-024-00303-z · Cited by 13 · see citing works
Optimizing ChatGPT’s Interpretation and Reporting of Delirium Assessment Outcomes: Exploratory Study
Choi, Y. K., Lin, S., Fick, D. M., Shulman, R. W., Lee, S., Shrestha, P., & Santoso, K. P. (2024) · JMIR Formative Research · doi:10.2196/51383 · Cited by 8 · see citing works
AlShenaiber, A., Datta, S., Mosa, A. J., Binhammer, P., & Ing, E. B. (2024) · Journal of Hand Surgery Global Online · doi:10.1016/j.jhsg.2024.07.011 · Cited by 8 · see citing works
ChatGPT in medical literature – a concise review and SWOT analysisPreprint
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J. P., Breuckmann, F., & Sellmann, T. (2023) · medRxiv · doi:10.1101/2023.05.06.23289608 · Cited by 8 · see citing works
Spennemann, D. H. R. (2023) · Publications · doi:10.3390/publications11030045 · Cited by 17 · see citing works
Assessing ChatGPT's use of person-first language in healthcare conversations
Hackl, E. V. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00099-9 · Cited by 7 · see citing works
Chatbots vs andrologists: Testing 25 clinical cases
Perrot, O., Schirmann, A., Vidart, A., Guillot‐Tantay, C., Izard, V., Lebrét, T., Boillot, B., Mesnard, B., Lebacle, C. M., & Madec, F.-X. (2024) · The French Journal of Urology · doi:10.1016/j.fjurol.2024.102636 · Cited by 7 · see citing works
Assessing the research landscape and clinical utility of large language models: A scoping reviewPreprint
Park, Y., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. T. (2023) · Research Square · doi:10.21203/rs.3.rs-3472000/v1 · Cited by 8 · see citing works
Herbert, D. N., Westendorf, J., Farmer, M., & Reeder, B. (2025) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.24-00319 · Cited by 6 · see citing works
Rao, A. S., Prasad, S., Lee, R. S., Farrell, S., McKinley, S. K., & Succi, M. D. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100241 · Cited by 6 · see citing works
Self-citationSengupta, P., Dutta, S., Chakravarthi, S., Jegasothy, R., Jeganathan, R., & Pichumani, A. (2023) · Gynecology and Obstetrics Clinical Medicine · doi:10.1016/j.gocm.2023.09.002 · Cited by 6 · see citing works
Rao, A. S., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.08.026 · Cited by 6 · see citing works
Self-citationValidation of the QAMAI tool to assess the quality of health information provided by AIPreprint
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Boscolo‐Rizzo, P., Califano, G., Cammaroto, G., et al. (2024) · medRxiv · doi:10.1101/2024.01.25.24301774 · Cited by 5 · see citing works
Kaboudi, N., Firouzbakht, S., Eftekhar, M. S., Fayazbakhsh, F., Joharivarnoosfaderani, N., Ghaderi, S., Dehdashti, M., Kia, Y. M., Afshari, M., Gharamaleki, M. V., et al. (2024) · medRxiv · doi:10.22037/aaem.v12i1.2384 · Cited by 42 · see citing works
ChatGPT and Higher Education: A Pathway to Unprecedented Progress
Almahasees, Z. M., Awabdeh, A. A., & Mahmoud, S. (2024) · Journal of Higher Education Theory and Practice · doi:10.33423/jhetp.v24i5.7004 · Cited by 10 · see citing works
The Genie Is Out of the Bottle
Morales-Ramirez, P., Mishek, H., & Dasgupta, A. (2023) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000005446 · Cited by 4 · see citing works
Jiang, Y., Zhu, J., Lin, Y.-Y., Su, Z., Zhang, L.-B., Dong, Z., Song, Q., Fan, P., & Li, Z. (2025) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-025-06451-2 · Cited by 2 · see citing works
Precisión de ChatGPT en el diagnóstico de entidades clínicas en el ámbito de la medicina interna
Andrade-Castellanos, C. A., la Paz, M. T. T.-D., & Farfán-Flores, P. E. (2023) · Gaceta Médica de México · doi:10.24875/gmm.23000297 · Cited by 3 · see citing works
- Authors
- Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Succi, Marc D.
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-26
- PMC
- PMC9980239
- OpenAlex
- W4322208207
- Cited by
- 134
- References
- 24
- Citations by year
- 2026: 8 · 2025: 20 · 2024: 55 · 2023: 50
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- medRxiv (acceptedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health
- National Institute of General Medical Sciences (T32GM144273)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- ChatGPT, large language models, clinical decision support, differential diagnosis, clinical vignettes, emergency medicine, clinical reasoning, diagnostic accuracy, clinical management, Emergency Severity Index
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
