The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Abstract
Chatbots are computer programs with which one can have a conversation. In this article, the authors describe how the GPT-4 chatbot, which has been given a general education, could affect the practice of medicine.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Large language models (LLMs), with their superior contextual semantic understanding, offer a promising avenue to overcome these limitations [8, 9].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [8]
In the citing paper
“Previous research has documented LLM hallucination in various contexts, including medical information [8], legal reasoning [9], and general factual queries [10].”
From Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025 (2026) · cites this paper as [8]
In the citing paper
“This is in contrast to prior work that claims LMs approach human performance, even for complex tasks (Ding et al., 2022; Nori et al., 2023; Lee et al., 2023) even though the task of writing biographies is fairly easy.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Lee et al., 2023
In the citing paper
“This is in contrast to prior work that claims LMs approach human performance, even for complex tasks (Ding et al., 2022; Nori et al., 2023; Lee et al., 2023) even though the task of writing biographies is fairly easy.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Lee et al., 2023
In the citing paper
“Studies evaluating GPT-4 and Claude on medical board examinations report accuracy rates of 80-90% [3,6]”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
In the citing paper
“Large Language Models (LLMs) demonstrating remarkable capabilities in medical knowledge synthesis, diagnostic reasoning, and clinical documentation [1-3].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
In the citing paper
“Most published medical AI benchmarks employ enterprise GPUs (NVIDIA A100, H100) or cloud API access, reporting neither hardware specifications nor inference costs [2,3,7].”
From Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware (2026) · cites this paper as [3]
In the citing paper
“While these models offer powerful foundational capabilities, adapting them to specific tasks and datasets without losing robustness is a significant hurdle, often resulting in significant variability in accuracy across different settings.7,9,10”
In the citing paper
“Despite their ability to create novel and customized content, these models are prone to hallucinations7 and can produce factually incorrect yet plausible outputs that may mislead users.”
In the citing paper
“If generative AI in health care is used for diagnostics, personalized medicine, or clinical assistance, AI hallucination can be extremely dangerous and may even harm patients’ lives [83].”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [83]
In the citing paper
“However, their clinical deployment carries substantial limitations, including the generation of inaccurate or fabricated outputs (commonly termed “hallucinations”), variable output quality, and the absence of transparent reasoning pathways [4].”
In the citing paper
“Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.9,17,156”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [9]
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J. P., & Lim, C. C. T. (2017) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,574
A Review of Approaches for Predicting Drug–Drug Interactions Based on Machine Learning
Han, K., Cao, P., Wang, Y., Xie, F., Ma, J., Yu, M., Wang, J., Xu, Y., Zhang, Y., & Wan, J. (2022) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.814858 · Cited by 180
Beaulieu‐Jones, B. K., Yuan, W., Brat, G. A., Beam, A. L., Weber, G. M., Ruffin, M., & Kohane, I. S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00426-3 · Cited by 168
Comparison of biomedical relationship extraction methods and models for knowledge graph creation
Milošević, N., & Thielemann, W. (2022) · Journal of Web Semantics · doi:10.1016/j.websem.2022.100756 · Cited by 49
The US Medical Licensing Examination
Cancarevic, I. (2021) · Springer eBooks · doi:10.1007/978-3-030-62249-7_23 · Cited by 2
Research works printed but not linked: 1
- Corbelle JG Bugarín-Diz A Alonso-Moral J Taboada J. Dealing with hallucination and omission in neural Natural Language Generation: a use case on meteorology. In: Proceedings and Abstracts of the 15th International Conference on Natural Language Generation July 18–22 2022. Waterville ME: Arria 2022.
Cited by
Sorted by citations
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Zhai, C., Wibowo, S., & Li, L. D. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00316-7 · Cited by 1,567
Artificial Intelligence and Machine Learning in Clinical Medicine, 2023
Haug, C. J., & Drazen, J. M. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2302038 · Cited by 1,215
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
Metadata
- Authors
- Lee, P., Bubeck, S., & Petro, J.
- Affiliations
- Microsoft (United States) (US)
- From Microsoft Research, Redmond, WA (P.L., S.B.); and Nuance Communications, Burlington, MA (J.P.).
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 2023-03-30
- Volume
- 388
- Issue
- 13
- Pages
- 1233-1239
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- OpenAlex
- W4361289889
- Semantic Scholar
- 72b74bcff8fd76eff6789111a7ce5d0d6c5ac4db
- Cited by
- 1,738
- References
- 6
- Citations by year
- 2026: 405 · 2025: 419 · 2024: 542 · 2023: 370
- FWCI
- 69.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- New England Journal of Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- GPT-4, large language models, AI chatbots, medical practice, clinical decision support, artificial intelligence in healthcare
- MeSH terms
- Artificial Intelligence, Humans, Medicine, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
