Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study
Open accessPublished version of a preprint
Abstract
Background: Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. Objective: This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. Methods: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks. Results: ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types. Conclusions: ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT can achieve high accuracy in classifying basic parameters such as pH and oxygenation status.
“Studies have demonstrated that ChatGPT can achieve high accuracy in classifying basic parameters such as pH and oxygenation status8.”
From Structural error asymmetry and harm-weighted analysis of ChatGPT versus ICU Physicians in acid–base interpretation: a prospective observational study (2026) · cites this paper as [8]
The Results (Overall Performance; page unavailable) report 71.8% accuracy across clinical vignettes, supporting a general claim about ChatGPT’s clinical-task accuracy. The paper does not report accuracy for classifying pH or oxygenation status specifically (searched: “pH,” “oxygenation,” and their combination). (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe only analysis that tested an LLM throughout the diagnostic clinical workflow used curated lists of possible answers and examined only 36 hypothetical clinical vignettes.
“the only analysis that tested an LLM throughout the diagnostic clinical workflow used curated lists of possible answers and examined only 36 hypothetical clinical vignettes28.”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [28]
The Methods (Study Design; Data Sources and Measurement, page not provided) describe testing ChatGPT across successive stages of the clinical workflow using all 36 MSD Manual vignettes. The vignette questions included select-all-that-apply choices, supporting the description of answer lists. The Introduction calls this study the first to use earlier conversational context in downstream responses, but does not establish that it was the only analysis testing an LLM throughout the workflow. (LLM, conf=0.82)
Experimental — accuracy not yet measuredHallucinations put the whole AI chatbot data output under significant question, especially in implementing and applying AI aid into clinical practice.
“As such, hallucinations put the whole AI chatbot data output under significant question, especially in implementing and applying AI aid into clinical practice [44,45].”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [44]
The Discussion (page unavailable) identifies hallucinations as a limitation of ChatGPT and says even small clinical errors can cause harm, so these risks must inform its clinical use and regulation. It does not say hallucinations call all AI chatbot output into question; the paper also reports substantial accuracy for ChatGPT on the clinical vignettes it tested. (LLM, conf=0.85)
Experimental — accuracy not yet measuredWell-founded feedback after completing a task is essential for an effective learning process.
“Well-founded feedback after completing a task is essential for an effective learning process [ 13 ].”
From Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial (2024) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“These limitations are inherent to the AI model itself and can be broadly divided into several categories, including misalignment and hallucination [28,29].”
Capps, L. (2003) · JAMA · doi:10.1001/jama.290.18.2487-b · Cited by 4,558 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · see citing works
Artificial intelligence and algorithmic bias: implications for health systems
Panch, T., Mattie, H., & Atun, R. (2019) · Journal of Global Health · doi:10.7189/jogh.09.020318 · Cited by 533 · see citing works
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · see citing works
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 481 · see citing works
Self-citationFlanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023) · JAMA · doi:10.1001/jama.2023.1344 · Cited by 420 · see citing works
The potential of artificial intelligence to improve patient safety: a scoping review
Bates, D. W., Levine, D. M., Syrowatka, A., Kuznetsova, M., Craig, K. J. T., Rui, A., Jackson, G. P., & Rhee, K. B. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00423-6 · Cited by 329 · see citing works
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257 · see citing works
Self-citationEitel, D. R., Rudkin, S., Malvehy, M. A., Killeen, J. P., & Pines, J. M. (2008) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2008.03.038 · Cited by 204 · see citing works
Bommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181 · see citing works
Assessing the utility of ChatGPT throughout the entire clinical workflow
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D. (2023) · doi:10.1101/2023.02.21.23285886 · Cited by 134 · see citing works
Self-citationHurtful words: quantifying biases in clinical contextual word embeddings
Zhang, H., Lu, A. X., Abdalla, M., McDermott, M., & Ghassemi, M. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384448 · Cited by 122 · see citing works
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · see citing works
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, J. A. M., & Katz, D. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322372 · Cited by 68 · see citing works
Chonde, D. B., Pourvaziri, A., Williams, J. M., McGowan, J., Moskos, M. M., Alvarez, C., Narayan, A. K., Daye, D., Flores, E. J., & Succi, M. D. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.01.013 · Cited by 40 · see citing works
Self-citationKim, D., Chung, J. W., Choi, J., Succi, M. D., Conklin, J., Longo, M. G. F., Ackman, J. B., Little, B. P., Petranović, M., Kalra, M. K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-29437-8 · Cited by 40 · see citing works
Self-citationLi, M., Arun, N., Aggarwal, M., Gupta, S., Singh, P., Little, B. P., Mendoza, D. P., Corradi, G. C. A., Takahashi, M. S., Ferraciolli, S. F., et al. (2022) · Medicine · doi:10.1097/md.0000000000029587 · Cited by 26 · see citing works
Self-citationWitowski, J. S., Choi, J., Jeon, S., Kim, D., Chung, J. W., Conklin, J., Longo, M. G. F., Succi, M. D., & Do, S. (2021) · Blockchain in Healthcare Today · doi:10.30953/bhty.v4.176 · Cited by 21 · see citing works
Self-citationReducing Bias in Healthcare Artificial Intelligence
Byrne, M. D. (2021) · Journal of PeriAnesthesia Nursing · doi:10.1016/j.jopan.2021.03.009 · Cited by 20 · see citing works
Chung, J. W., Kim, D., Choi, J., Yune, S., Song, K. D., Kim, S., Chua, M., Succi, M. D., Conklin, J., Longo, M. G. F., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-24721-5 · Cited by 19 · see citing works
Self-citationO’Shea, A., Li, M., Mercaldo, N. D., Balthazar, P., Som, A., Yeung, T., Succi, M. D., Little, B. P., Kalpathy–Cramer, J., & Lee, S. I. (2022) · BJR|Open · doi:10.1259/bjro.20210062 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [25]Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care Smedley BD Stith AY Nelson AR Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care 2003 Washington, DC National Academies Press (US) 25032386
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
“However, the only analysis that tested an LLM throughout the diagnostic clinical workflow used curated lists of possible answers and examined only 36 hypothetical clinical vignettes 28 .”
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234 · see citing works
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · see citing works
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179 · see citing works
Aljamaan, F., Temsah, M.-H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024) · JMIR Medical Informatics · doi:10.2196/54345 · Cited by 166 · see citing works
“As such, hallucinations put the whole AI chatbot data output under significant question, especially in implementing and applying AI aid into clinical practice [44,45].”
MethodHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
The Role of Large Language Models in Transforming Emergency Medicine: Scoping Review
Preiksaitis, C., Ashenburg, N., Bunney, G., Chu, A., Kabeer, R., Riley, F., Ribeira, R., & Rose, C. (2024) · JMIR Medical Informatics · doi:10.2196/53787 · Cited by 134 · see citing works
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132 · see citing works
Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06399-7 · Cited by 137 · see citing works
The application of Chat Generative Pre-trained Transformer in nursing education
Liu, J., Liu, F., Fang, J., & Liu, S. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2023.102064 · Cited by 103 · see citing works
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71 · see citing works
Hurt, R. T., Stephenson, C. R., Gilman, E. A., Aakre, C. A., Croghan, I. T., Mundi, M. S., Ghosh, K., & Edakkanambeth Varayil, J. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251332215 · Cited by 78 · see citing works
The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice
Shoja, M. M., Van de Ridder, J. M. M., & Rajput, V. (2023) · Cureus · doi:10.7759/cureus.40883 · Cited by 69 · see citing works
Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology
Luo, M.-J., Pang, J., Bi, S., Lai, Y., Zhao, J., Shang, Y., Cui, T., Yang, Y., Lin, Z., Zhao, L., et al. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2513 · Cited by 70 · see citing works
Reasoning with large language models for medical question answering
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae131 · Cited by 66 · see citing works
Potential and pitfalls of romantic Artificial Intelligence (AI) companions: A systematic review
Ho, J. Q. H., Hu, M., Chen, T. X., & Hartanto, A. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100715 · Cited by 64 · see citing works
ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students
Zawiah, M., Al-Ashwal, F., Gharaibeh, L., Abu Farha, R., Alzoubi, K., Abu Hammour, K., Qasim, Q. A., & Abrah, F. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s439223 · Cited by 51 · see citing works
A bibliometric analysis of the advance of artificial intelligence in medicine
Lin, M., Lin, L., Lin, L., Lin, Z., & Yan, X. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1504428 · Cited by 36 · see citing works
A foundational architecture for AI agents in healthcare
Liu, F., Niu, Y., Zhang, Q., Wang, K., Dong, Z., Wong, I. N., Cheng, L., Li, T., Duan, L., Li, K., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102374 · Cited by 41 · see citing works
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28 · see citing works
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Reading Turchioe, M., Kisselev, S., Van Bulck, L., & Bakken, S. (2024) · Applied Clinical Informatics · doi:10.1055/a-2373-3151 · Cited by 20 · see citing works
Quantum leap in medical mentorship: exploring ChatGPT’s transition from textbooks to terabytes
Chokkakula, S., Chong, S., Yang, B., Jiang, H., Yu, J., Han, R., Attitalla, I. H., Yin, C., & Zhang, S. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1517981 · Cited by 12 · see citing works
Nasef, H., Patel, H., Amin, Q., Baum, S., Ratnasekera, A., Ang, D., Havron, W. S., Nakayama, D., & Elkbuli, A. (2025) · The American Surgeon™ · doi:10.1177/00031348241256075 · Cited by 4 · see citing works
Gulen, D., Gözden, H. E., Ekin, S., & Ceylan, I. (2026) · Scientific Reports · doi:10.1038/s41598-026-44576-4 · Cited by 1 · see citing works
“Studies have demonstrated that ChatGPT can achieve high accuracy in classifying basic parameters such as pH and oxygenation status 8 .”
- Authors
- Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D.
- Affiliations
- Harvard University (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Succi, Marc D
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2023-08-22
- Volume
- 25
- Pages
- e48659
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10481210
- OpenAlex
- W4386045865
- Cited by
- 412
- References
- 26
- Citations by year
- 2026: 95 · 2025: 131 · 2024: 139 · 2023: 45 · 2022: 1
- FWCI
- 16.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health
- National Institute of General Medical Sciences (T32GM144273)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- ChatGPT, large language models, clinical decision support, AI chatbots, differential diagnosis, clinical vignettes, diagnostic accuracy, clinical management, model hallucination, clinical reasoning
- MeSH terms
- Clinical Decision-Making, User-Centered Design, Artificial Intelligence, Humans, Organizations, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
