Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Abstract
Chatbots are computer programs with which one can have a conversation. In this article, the authors describe how the GPT-4 chatbot, which has been given a general education, could affect the practice of medicine.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs offer a promising avenue to overcome the limitations of BERT-based tools due to superior contextual semantic understanding.
“Large language models (LLMs), with their superior contextual semantic understanding, offer a promising avenue to overcome these limitations [8, 9].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [8]
Previous research has documented LLM hallucination in medical information, legal reasoning, and general factual queries.
“Previous research has documented LLM hallucination in various contexts, including medical information [8], legal reasoning [9], and general factual queries [10].”
From Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025 (2026) · cites this paper as [8]
Prior work claims that language models approach human performance, even for complex tasks.
“This is in contrast to prior work that claims LMs approach human performance, even for complex tasks (Ding et al., 2022; Nori et al., 2023; Lee et al., 2023) even though the task of writing biographies is fairly easy.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Lee et al., 2023
Prior work claims LMs approach human performance, even for complex tasks.
“This is in contrast to prior work that claims LMs approach human performance, even for complex tasks (Ding et al., 2022; Nori et al., 2023; Lee et al., 2023) even though the task of writing biographies is fairly easy.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Lee et al., 2023
Studies evaluating GPT-4 and Claude on medical board examinations report accuracy rates of 80-90%.
“Studies evaluating GPT-4 and Claude on medical board examinations report accuracy rates of 80-90% [3,6]”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
Large Language Models demonstrate remarkable capabilities in medical knowledge synthesis, diagnostic reasoning, and clinical documentation.
“Large Language Models (LLMs) demonstrating remarkable capabilities in medical knowledge synthesis, diagnostic reasoning, and clinical documentation [1-3].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs.
“Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
Adapting AI models to specific tasks and datasets without losing robustness is a significant hurdle, often resulting in significant variability in accuracy across different settings.
“While these models offer powerful foundational capabilities, adapting them to specific tasks and datasets without losing robustness is a significant hurdle, often resulting in significant variability in accuracy across different settings.7,9,10”
Generative AI models are prone to hallucinations and can produce factually incorrect yet plausible outputs that may mislead users.
“Despite their ability to create novel and customized content, these models are prone to hallucinations7 and can produce factually incorrect yet plausible outputs that may mislead users.”
If generative AI in health care is used for diagnostics, personalized medicine, or clinical assistance, AI hallucination can be extremely dangerous and may even harm patients' lives.
“If generative AI in health care is used for diagnostics, personalized medicine, or clinical assistance, AI hallucination can be extremely dangerous and may even harm patients’ lives [83].”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [83]
Clinical deployment of LLMs carries substantial limitations, including generation of inaccurate or fabricated outputs (hallucinations), variable output quality, and absence of transparent reasoning pathways.
“However, their clinical deployment carries substantial limitations, including the generation of inaccurate or fabricated outputs (commonly termed “hallucinations”), variable output quality, and the absence of transparent reasoning pathways [4].”
Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.
“Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.9,17,156”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [9]
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J. P., & Lim, C. C. T. (2017) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,521 · see citing works
A Review of Approaches for Predicting Drug–Drug Interactions Based on Machine Learning
Han, K., Cao, P., Wang, Y., Xie, F., Ma, J., Yu, M., Wang, J., Xu, Y., Zhang, Y., & Wan, J. (2022) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.814858 · Cited by 178 · see citing works
Beaulieu‐Jones, B. K., Yuan, W., Brat, G. A., Beam, A. L., Weber, G. M., Ruffin, M., & Kohane, I. S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00426-3 · Cited by 157 · see citing works
Comparison of biomedical relationship extraction methods and models for knowledge graph creation
Milosevic, N., & Thielemann, W. (2022) · Journal of Web Semantics · doi:10.1016/j.websem.2022.100756 · Cited by 40 · see citing works
The US Medical Licensing Examination
Cancarevic, I. (2021) · Springer eBooks · doi:10.1007/978-3-030-62249-7_23 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567 · see citing works
Artificial Intelligence and Machine Learning in Clinical Medicine, 2023
Haug, C. J., & Drazen, J. M. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2302038 · Cited by 1,215 · see citing works
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794 · see citing works
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784 · see citing works
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708 · see citing works
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S. B., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 616 · see citing works
Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510 · see citing works
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443 · see citing works
Creation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Li, W., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01029-4 · Cited by 400 · see citing works
Haltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396 · see citing works
Stade, E. C., Stirman, S. W., Ungar, L. H., Boland, C. L., Schwartz, H. A., Yaden, D. B., Sedoc, J., DeRubeis, R. J., Willer, R., & Eichstaedt, J. C. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00056-z · Cited by 391 · see citing works
Large Language Models in Medicine: The Potentials and Pitfalls
Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J., & Daneshjou, R. (2024) · Annals of Internal Medicine · doi:10.7326/m23-2772 · Cited by 394 · see citing works
Emerging opportunities and challenges for the future of reservoir computing
Yan, M., Huang, C., Bienstman, P., Peter Tiňo, B., Lin, W. J., & Sun, J. (2024) · Nature Communications · doi:10.1038/s41467-024-45187-1 · Cited by 381 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Generative AI in medical practice: in-depth exploration of privacy and security challenges
Chen, Y., & Esmaeilzadeh, P. (2024) · Journal of Medical Internet Research · doi:10.2196/53008 · Cited by 353 · see citing works
“If generative AI in health care is used for diagnostics, personalized medicine, or clinical assistance, AI hallucination can be extremely dangerous and may even harm patients’lives [83].”
Autonomous travel decision-making: An early glimpse into ChatGPT and generative AI
Wong, I. A., Lian, Q. L., & Sun, D. (2023) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2023.06.022 · Cited by 361 · see citing works
Bhayana, R. (2024) · Radiology · doi:10.1148/radiol.232756 · Cited by 340 · see citing works
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 316 · see citing works
Artificial Intelligence in Medicine
Beam, A. L., Drazen, J. M., Kohane, I. S., Leong, T.-Y., Manrai, A. K., & Rubin, E. J. (2023) · New England Journal of Medicine · doi:10.1056/nejme2206291 · Cited by 303 · see citing works
Zaretsky, J., Kim, J. M., Baskharoun, S., Zhao, Y., Austrian, J., Aphinyanaphongs, Y., Gupta, R., Blecker, S. B., & Feldman, J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.0357 · Cited by 303 · see citing works
Artificial intelligence in clinical medicine: catalyzing a sustainable global healthcare paradigm
Krishnan, G., Singh, S., Pathania, M., Gosavi, S., Abhishek, S., Parchani, A., & Dhar, M. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1227091 · Cited by 326 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
Artificial intelligence-powered electronic skin
Xu, C., Solomon, S. A., & Gao, W. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00760-z · Cited by 259 · see citing works
A Turing Test of Whether AI Chatbots Are Behaviorally Similar to Humans
Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2313925121 · Cited by 254 · see citing works
A Systematic Review of Generative AI for Teaching and Learning Practice
Ogunleye, B., Zakariyyah, K. I., Ajao, O., Olayinka, O., & Sharma, H. (2024) · Education Sciences · doi:10.3390/educsci14060636 · Cited by 251 · see citing works
Bernstein, I. A., Zhang, Y., Govil, D., Majid, I., Chang, R. T., Sun, Y., Shue, A., Chou, J. C., Schehlein, E. M., Christopher, K. L., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.30320 · Cited by 250 · see citing works
The Role of Large Language Models in Medical Education: Applications and Implications
Safranek, C. W., Sidamon-Eristoff, A. E., Gilson, A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/50945 · Cited by 233 · see citing works
Kang, Y., & Kim, J. (2024) · Nature Communications · doi:10.1038/s41467-024-48998-4 · Cited by 268 · see citing works
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
“Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.”
BackgroundCascella, M., Semeraro, F., Montomoli, J., Bellini, V., Piazza, O., & Bignami, E. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02045-3 · Cited by 219 · see citing works
Large language model AI chatbots require approval as medical devices
Gilbert, S., Harvey, H., Melvin, T., Vollebregt, E., & Wicks, P. (2023) · Nature Medicine · doi:10.1038/s41591-023-02412-6 · Cited by 207 · see citing works
Garg, R. K., Urs, V. L., Agrawal, A. A., Chaudhary, S. K., Paliwal, V. K., & Kar, S. K. (2023) · Health Promotion Perspectives · doi:10.34172/hpp.2023.22 · Cited by 233 · see citing works
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201 · see citing works
Use of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200 · see citing works
Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
Nov, O., Singh, N., & Mann, D. (2023) · JMIR Medical Education · doi:10.2196/46939 · Cited by 198 · see citing works
Fountzilas, E., Pearce, T. E., Baysal, M. A., Chakraborty, A., & Tsimberidou, A. M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01471-y · Cited by 219 · see citing works
The clinicians’ guide to large language models: a general perspective with a focus on hallucinations
Roustan, D., & Bastardot, F. (2025) · Interactive Journal of Medical Research · doi:10.2196/59823 · Cited by 191 · see citing works
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · see citing works
ChatGPT: The cognitive effects on learning and memory
Bai, L., Liu, X., & Su, J. (2023) · Brain‐X · doi:10.1002/brx2.30 · Cited by 182 · see citing works
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179 · see citing works
Zhou, B., Li, X., Liu, T., Xu, K., Liu, W., & Bao, J. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102333 · Cited by 181 · see citing works
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189 · see citing works
Miao, J., Thongprayoon, C., Suppadungsuk, S., Valencia, O. A. G., & Cheungpasitporn, W. (2024) · Medicina · doi:10.3390/medicina60030445 · Cited by 176 · see citing works
Chatbot vs medical student performance on free-response clinical reasoning examinations.
Strong, E., DiGiammarino, A., Weng, Y., Kumar, A., Hosamani, P., Hom, J., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2909 · Cited by 169 · see citing works
Shaping the future of AI in healthcare through ethics and governance
Bouderhem, R. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02894-w · Cited by 195 · see citing works
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189 · see citing works
Where Medical Statistics Meets Artificial Intelligence
Hunter, D. J., & Holmes, C. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2212850 · Cited by 165 · see citing works
Large language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161 · see citing works
Freyer, O., Wiest, I. C., Kather, J. N., & Gilbert, S. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00124-9 · Cited by 162 · see citing works
Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 156 · see citing works
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/48392 · Cited by 156 · see citing works
Chakraborty, C., Pal, S., Bhattacharya, M., Dash, S., & Lee, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1237704 · Cited by 170 · see citing works
Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152 · see citing works
Investigating Students’ Perceptions towards Artificial Intelligence in Medical Education
Buabbas, A. J., Miskin, B., Alnaqi, A. A., Ayed, A. K., Shehab, A. A., Syed-Abdul, S., & Uddin, M. (2023) · Healthcare · doi:10.3390/healthcare11091298 · Cited by 152 · see citing works
AI-Generated Draft Replies Integrated Into Health Records and Physicians’ Electronic Communication
Tai-Seale, M., Baxter, S. L., Vaida, F., Walker, A., Sitapati, A. M., Osborne, C., Diaz, J., Desai, N., Webb, S., Polston, G., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.6565 · Cited by 158 · see citing works
Medical Artificial Intelligence and Human Values
Yu, K.-H., Healey, E., Leong, T.-Y., Kohane, I. S., & Manrai, A. K. (2024) · New England Journal of Medicine · doi:10.1056/nejmra2214183 · Cited by 149 · see citing works
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
Anh-Hoang, D., Tran, V., & Nguyen, L.-M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1622292 · Cited by 146 · see citing works
Evaluating large language models as agents in the clinic
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01083-y · Cited by 143 · see citing works
Waqas, A., Bui, M. M., Glassy, E. F., Naqa, I. E., Borkowski, P., Borkowski, A., & Rasool, G. (2023) · Laboratory Investigation · doi:10.1016/j.labinv.2023.100255 · Cited by 149 · see citing works
Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 169 · see citing works
Akpan, I. J., Kobara, Y. M., Owolabi, J., Akpan, A. A., & Offodile, O. F. (2024) · International Transactions in Operational Research · doi:10.1111/itor.13522 · Cited by 171 · see citing works
Large Language Models as Evolutionary Optimizers
Liu, S., Chen, C., Qu, X., Tang, K., & Ong, Y.-S. (2024) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec60901.2024.10611913 · Cited by 138 · see citing works
Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying
Soroush, A., Glicksberg, B. S., Zimlichman, E., Barash, Y., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E. (2024) · NEJM AI · doi:10.1056/aidbp2300040 · Cited by 137 · see citing works
Choi, H. S., Song, J. Y., Shin, K. H., Chang, J. H., & Jang, B. (2023) · Radiation Oncology Journal · doi:10.3857/roj.2023.00633 · Cited by 133 · see citing works
Menz, B. D., Kuderer, N. M., Bacchi, S., Modi, N. D., Chin-Yee, B., Hu, T., Rickard, C., Haseloff, M., Vitry, A., McKinnon, R. A., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078538 · Cited by 132 · see citing works
Woodman, R. J., & Mangoni, A. A. (2023) · Aging Clinical and Experimental Research · doi:10.1007/s40520-023-02552-2 · Cited by 142 · see citing works
Black Box Warning: Large Language Models and the Future of Infectious Diseases Consultation
Schwartz, I. S., Link, K. E., Daneshjou, R., & Cortés-Penfield, N. (2024) · Clinical Infectious Diseases · doi:10.1093/cid/ciad633 · Cited by 131 · see citing works
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130 · see citing works
Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine
Zhang, K., Meng, X.-B., Yan, X.-Y., Ji, J.-M., Liu, J.-Q., Xu, H., Zhang, H., Liu, D., Wang, J.-J., Wang, X.-L., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59069 · Cited by 153 · see citing works
Influence of believed AI involvement on the perception of digital medical advice
Reis, M., Reis, F., & Kunde, W. (2024) · Nature Medicine · doi:10.1038/s41591-024-03180-7 · Cited by 130 · see citing works
Cadamuro, J., Cabitza, F., Debeljak, Ž., Bruyne, S. D., Frans, G., Pérez, S. M., Özdemir, H., Tolios, A., Carobene, A., & Padoan, A. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2023-0355 · Cited by 120 · see citing works
Williams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119 · see citing works
Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study
Levkovich, I., & Elyoseph, Z. (2023) · JMIR Mental Health · doi:10.2196/51232 · Cited by 119 · see citing works
Lin, J. C. H., Younessi, D. N., Kurapati, S. S., Tang, O. Y., & Scott, I. U. (2023) · Eye · doi:10.1038/s41433-023-02564-2 · Cited by 118 · see citing works
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 130 · see citing works
Eppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117 · see citing works
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
Large language models streamline automated machine learning for clinical studies
Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J. N., Truhn, D., & Nebelung, S. (2024) · Nature Communications · doi:10.1038/s41467-024-45879-8 · Cited by 114 · see citing works
Le Guellec, B., Lefèvre, A., Geay, C., Shorten, L., Bruge, C., Hacein-Bey, L., Amouyel, P., Pruvo, J.-P., Kuchcinski, G., & Hamroun, A. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230364 · Cited by 115 · see citing works
Applications of large language models in cancer care: current evidence and future perspectives
Iannantuono, G. M., Bracken-Clarke, D., Floudas, C. S., Roselli, M., Gulley, J. L., & Karzai, F. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1268915 · Cited by 115 · see citing works
Addressing 6 challenges in generative AI for digital health: A scoping review
Templin, T., Perez, M. W., Sylvia, S., Leek, J. T., & Sinnott-Armstrong, N. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000503 · Cited by 115 · see citing works
A Nationwide Network of Health AI Assurance Laboratories
Shah, N. H., Halamka, J. D., Saria, S., Pencina, M., Tazbaz, T., Tripathi, M., Callahan, A., Hildahl, H., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2023.26930 · Cited by 114 · see citing works
A comprehensive review of large language models: issues and solutions in learning environments
Shahzad, T., Mazhar, T., Tariq, M., Ahmad, W., Ouahada, K., & Hamam, H. (2025) · Discover Sustainability · doi:10.1007/s43621-025-00815-8 · Cited by 129 · see citing works
A toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113 · see citing works
Artificial Intelligence in Clinical Diagnosis: Opportunities, Challenges, and Hype
Kulkarni, P. A., & Singh, H. (2023) · JAMA · doi:10.1001/jama.2023.11440 · Cited by 112 · see citing works
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., Qureshi, F., & Cheungpasitporn, W. (2023) · Clinics and Practice · doi:10.3390/clinpract14010008 · Cited by 111 · see citing works
Potential of large language models in health care: Delphi study
Denecke, K., May, R., LLMHealthGroup, & Rivera Romero, O. (2024) · Journal of Medical Internet Research · doi:10.2196/52399 · Cited by 111 · see citing works
Large Language Model–Based Responses to Patients’ In-Basket Messages
Small, W. R., Wiesenfeld, B. M., Brandfield-Harvey, B., Jonassen, Z., Mandal, S., Stevens, E. R., Major, V. J., Lostraglio, E., Szerencsy, A. C., Jones, S., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.22399 · Cited by 108 · see citing works
Artificial intelligence: revolutionizing cardiology with large language models
Boonstra, M., Weissenbacher, D., Moore, J. H., Gonzalez‐Hernandez, G., & Asselbergs, F. W. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad838 · Cited by 107 · see citing works
ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis
Bicknell, B. T., Butler, D., Whalen, S., Ricks, J., Dixon, C. J., Clark, A. B., Spaedy, O., Skelton, A., Edupuganti, N., Dzubinski, L., et al. (2024) · JMIR Medical Education · doi:10.2196/63430 · Cited by 107 · see citing works
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 118 · see citing works
- Authors
- Lee, P., Bubeck, S., & Petro, J.
- Affiliations
- Microsoft (United States) (US)
- From Microsoft Research, Redmond, WA (P.L., S.B.); and Nuance Communications, Burlington, MA (J.P.).
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 2023-03-30
- Volume
- 388
- Issue
- 13
- Pages
- 1233-1239
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- OpenAlex
- W4361289889
- Cited by
- 1,738
- References
- 6
- Citations by year
- 2026: 405 · 2025: 419 · 2024: 542 · 2023: 370 · 2022: 1 · 2013: 1
- FWCI
- 69.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- New England Journal of Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- GPT-4, large language models, AI chatbots, medical practice, clinical decision support, artificial intelligence in healthcare
- MeSH terms
- Artificial Intelligence, Humans, Medicine, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
