CoChat
Sign Up
Cited by 7Medical Coding and Health InformationMachine Learning in HealthcareArtificial Intelligence in Healthcare and Education

Assessing retrieval-augmented large language models for medical coding

Klang, Eyal, Tessler, Idit, Apakama, Donald U., Abbott, Ethan, Glicksberg, Benjamin S., Arnold, Monique, Moses, Akini, Sakhuja, Ankit +8 more·2025·NEJM AI
DOI 10.1056/aics2401161
Open this paper in CoChat — free account

Abstract

, Tenth Revision, Clinical Modification. A RAG system enhanced LLM predictions using data from over 1 million emergency department visits. We compared RAG-enhanced codes with provider-assigned codes. A masked review by four physicians and two LLMs determined which codes - LLM- or provider-assigned - were more accurate and specific. RAG-enhanced LLMs demonstrated superior accuracy and specificity compared with provider-assigned codes. Human reviewers favored RAG-enhanced Generative Pretrained Transformer 4 (GPT-4) for accuracy in 447 instances, versus 277 instances for provider-assigned codes (P<0.001). For specificity, RAG-enhanced GPT-4 was preferred in 509 cases, compared with 181 for provider-assigned codes (P<0.001). Smaller open-access models also showed significant improvement with RAG, demonstrating that integrating RAG with LLM medical coding may reduce errors and improve clinical documentation. (Funded by the National Center for Advancing Translational Sciences and the Office of Research Infrastructure Programs of the National Institutes of Health.).

Integrity signals

Registry

Crossref · 10.1056/aics2401161

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In complex medical coding tasks, a RAG-enhanced agentic system demonstrated superior performance to human providers; expert reviewers favored the AI's accuracy in 447 instances compared to 277 for provider-assigned codes (p < 0.001).

    “For instance, in complex medical coding tasks where diagnostic specificity is tied to financial and legal outcomes, a RAG-enhanced agentic system demonstrated superior performance to human providers; expert reviewers favored the AI’s accuracy in 447 instances compared to 277 for provider-assigned codes (p < 0.001) [54].”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [54]

  • The system was preferred for its specificity in 509 cases versus 181 for human providers (p < 0.001).

    “Furthermore, the system was preferred for its specificity in 509 cases versus 181 for human providers (p < 0.001), illustrating that the high token expenditure of agentic loops is economically justified when the cost of diagnostic or administrative error exceeds the marginal cost of increased computing.”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [54]

  • Recent evaluations indicate that structured agentic panels can reduce complex diagnostic expenditures from approximately $7850 to $2397 while significantly increasing specificity over human specialists.

    “Recent evaluations indicate that structured agentic panels can reduce complex diagnostic expenditures from approximately $7850 to $2397 while significantly increasing specificity over human specialists [54,55].”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [54]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Klang, E., Tessler, I., Apakama, D. U., Abbott, E., Glicksberg, B. S., Arnold, M., Moses, A., Sakhuja, A., Soroush, A., Charney, A. W., Reich, D. L., McGreevy, J., Gavin, N., Carr, B., Freeman, R., & Nadkarni, G. N.
Affiliations
  • Mount Sinai Health System (US)
  • Tel Aviv University (IL)
  • Icahn School of Medicine at Mount Sinai (US)
ORCID
Venue
NEJM AI
Publisher
Massachusetts Medical Society
Published
2025-09-25
Volume
2
Issue
10
Pages
AIcs2401161-AIcs2401161
ISSN
2836-9386 (ISSN-L)
DOI
10.1056/aics2401161
OpenAlex
W4414531167
Cited by
6
References
21
Citations by year
2026: 4 · 2025: 2
FWCI
4.72
Type
journal article
Language
en
Open access
closed
Locations
  • NEJM AI (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
retrieval-augmented generation, large language models, medical coding, ICD-10-CM, model evaluation, clinical documentation
MeSH terms
Large Language Models, Generative Artificial Intelligence, Emergency Service, Hospital, Humans, International Classification of Diseases, Clinical Coding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review