The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying
Open accessAbstract
BACKGROUND Large language models (LLMs) have attracted significant interest for automated clinical coding. However, early data show that LLMs are highly error-prone when mapping medical codes. We sought to quantify and benchmark LLM medical code querying errors across several available LLMs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Even state-of-the-art proprietary LLMs perpetuate historic biases41, cite inappropriate medical articles42 and fail to perform information-driven administrative tasks like medical coding43.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [43]
In the citing paper
“studies examining specific tasks, such as ICD (International Classification of Diseases) coding9, have revealed significant performance deficiencies”
From Large Language Models lack essential metacognition for reliable medical reasoning (2025) · cites this paper as [9]
In the citing paper
“Yet, while these models may recall medical facts, it remains challenging for the models to process information logically and generate responses that demonstrate sound reasoning5.”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [5]
In the citing paper
“LLMs have demonstrated poor comprehension of ICD codes4”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [4]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3348 · Cited by 1,952
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Research works printed but not linked: 7
- American Medical Association. CPT® overview and code approval. 2023 (https://www.ama-assn.org/practice-management/cpt/cpt-overview-and-code-approval).
- Open AI. GPT-4 technical report. March 15 2023 (https://arxiv.org/abs/2303.08774). Preprint. · arXiv
- Anil R Borgeaud S Wu Y et al. Gemini: a family of highly capable multimodal models. December 19 2023 (https://arxiv.org/abs/2312.11805). Preprint. · arXiv
- Zhang T Kishore V Wu F Weinberger KQ Artzi Y. BERTscore: evaluating text generation with BERT. 2019 (https://arxiv.org/abs/1904.09675). Preprint. · arXiv
- Nogueira R Jiang Z Lin J. Investigating the limitations of transformers with simple arithmetic tasks. April 12 2021 (https://arxiv.org/abs/2102.13019). Preprint. · arXiv
- Yang K Liu J Wu J et al. If LLM is the wizard then code is the wand: a survey on how code empowers large language models to serve as intelligent agents. January 8 2024 (https://arxiv.org/abs/2401.00812). Preprint. · arXiv
- Lu P Peng B Cheng H et al. Chameleon: plug-and-play compositional reasoning with large language models. October 31 2023 (https://arxiv.org/abs/2304.09842). Preprint. · arXiv
Cited by
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
“Even state-of-the-art proprietary LLMs perpetuate historic biases 41 , cite inappropriate medical articles 42 and fail to perform information-driven administrative tasks like medical coding 43 .”
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 172
“For instance, studies examining speci fi c tasks, such as ICD (International Classi fi cation of Diseases) coding 9 , have revealed signi fi cant performance de fi ciencies, under-scoring the need for more comprehensive evaluation frameworks 10,11 .”
Freyer, O., Wiest, I. C., Kather, J. N., & Gilbert, S. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00124-9 · Cited by 204
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134
Metadata
- Authors
- Soroush, A., Glicksberg, B. S., Zimlichman, E., Barash, Y., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E.
- Affiliations
- Icahn School of Medicine at Mount Sinai (US)
- Tel Aviv University (IL)
- Mount Sinai Health System (US)
- ORCID
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-04-25
- Volume
- 1
- Issue
- 5
- Pages
- AIdbp2300040-AIdbp2300040
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4394943312
- Cited by
- 137
- References
- 14
- Citations by year
- 2026: 45 · 2025: 64 · 2024: 28
- FWCI
- 41.86
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Funders
- AGA Research Foundation
- National Institutes of Health
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, ICD-10-CM, benchmarking, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. https://app.cochat.ai/papers/10.1056/aidbp2300040
- The paper
- Soroush, A., Glicksberg, B. S., Zimlichman, E., Barash, Y., Freeman, R., Charney, A. W., Nadkarni, G. N., & Klang, E. (2024). Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. NEJM AI. https://doi.org/10.1056/aidbp2300040
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.