Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ Knowledge
Abstract
This Medical News article discusses new research on artificial intelligence systems such as ChatGPT and Med-PaLM.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Strategies used in the medical field have been shown to induce hallucination.
“strategies used in the medical field have been shown to induce hallucination (Umapathi et al., 2023;Harris, 2023;Ji et al., 2023)”
From MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024) · cites this paper as Harris, 2023
Failures in medical question answering originate from a lack of domain knowledge rather than from reasoning rationale.
“Such failures in medical question-answering originate from a lack of domain knowledge (Harris, 2023;Kung et al., 2023;Tian et al., 2024) instead of reasoning rationale.”
From MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024) · cites this paper as Harris, 2023
Cited by
Sorted by citations
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84 · see citing works
Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions
Han, T., Adams, L. C., Bressem, K. K., Busch, F., Nebelung, S., & Truhn, D. (2024) · JAMA · doi:10.1001/jama.2023.27861 · Cited by 83 · see citing works
A bibliometric analysis of the rise of ChatGPT in medical research
Barrington, N. M., Gupta, N., Musmar, B., Doyle, D., Panico, N., Godbole, N., Reardon, T., & D’Amico, R. S. (2023) · Medical Sciences · doi:10.3390/medsci11030061 · Cited by 69 · see citing works
Inherent bias in large language models: a random sampling analysis.
Ayoub, N. F., Balakrishnan, K., Ayoub, M. S., Barrett, T. F., David, A. P., & Gray, S. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.03.003 · Cited by 47 · see citing works
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41 · see citing works
- Authors
- Harris, E.
- Corresponding author
- Harris, Emily
- ORCID
- Harris, Emily
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 2023-09-05
- Volume
- 330
- Issue
- 9
- Pages
- 792
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W4385620111
- Cited by
- 70
- References
- 0
- Citations by year
- 2026: 14 · 2025: 19 · 2024: 28 · 2023: 9
- FWCI
- 2.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute on Aging (P30AG059307)
- National Institute on Drug Abuse (UG1DA015815)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, ChatGPT, medical question answering, clinical knowledge, AI accuracy, clinical decision support, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
