CoChat

A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models

Open access
Stuhlmiller, Timothy J., Rabe, AJ, Rapp, Jeff, Awawda, Alaa, Kouser, Hiba, Lui, Kristi, Salamon, Hugh, Chuyka, Donald, Mahoney, William, Furgason, John M., Paul, Madhuri, Scarpa, Frank J., Kesari, Santosh, Newton, Mika, Wong, Kenny K., Kramer, Glenn A., Shapiro, Mark A. · 2025 · medRxiv

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index; 4 of the 7 sources its claims cite were checked for support.

Abstract

An LLM-powered extraction process achieves a significant gain in clinically relevant information over pre-structured data available in the EHR...

Overview

CoChat Summary

TL;DR

This paper presents an LLM-based pipeline for extracting structured medication data from heterogeneous EHR documents, including C-CDA, FHIR, and image files, with mapping to FHIR R4 and OMOP. In validation on 34 patients, the medication extractor achieved ~95% accuracy and F1 across seven fields, and large-scale deployment across 11,115 cancer patients increased medication and oncology-specific data completeness over pre-structured EHR data.

Key findings

  • The medication extraction schema, validated on 34 patients and 5,789 extracted medications, achieved ~95% accuracy and F1-score across 7 data fields; per-field F1 ranged from 0.89 for end_date to 1.00 for medication_name.
  • Deployment across 11,115 patients extracted ~2.6 million medication records from 169,071 documents, increasing total medications by 27% and distinct drug ingredients by 31% over structured C-CDA data.
  • For oncology-specific medications, LLM extraction increased total oncology medication records by 60%, distinct oncology drug ingredients by 64%, and patients with at least one structured oncology medication by 33%; it also extracted 122 unique oncology therapies/medications not found in structured C-CDA tables.
  • LLM extraction improved availability of indication for prescription (61% vs. 31%) and discontinuation reason (17% vs. 0%) compared with pre-structured C-CDA oncology medication data.

Methods

  • Documents in C-CDA, FHIR, and image formats were preprocessed with parsing, OCR, metadata extraction, and chunking; text chunks were embedded using OpenAI text-embedding-3-small, and a random forest classifier was trained on embeddings to classify chunks into 26 predefined information types.
  • Extraction used semantic search and document classification to select relevant documents and schema-specific prompts; LLMs performed named entity recognition and relation extraction, with outputs mapped to FHIR R4 and normalized to OMOP vocabularies.
  • Validation used clinically trained human reviewers with a second QC reviewer and third referee; performance was calculated per data field using accuracy, precision, recall, and F1, and deployment was assessed in 11,115 patients with confirmed cancer diagnoses.

Limitations

  • The paper states that extraction accuracy depends on high-quality input documents, and OCR performance for scanned documents such as faxes may introduce errors or omissions.
  • The paper states that strict validation was necessary to mitigate hallucinations and retain only explicitly stated extractions, and that current validation is heavily biased against the LLM, using two human abstractors and a referee.
  • The paper notes that future work may establish inter-rater reliability between human abstractors to position LLM extraction relative to expert human performance; the initial validation cohort was small at 34 patients, though deployment covered 11,115 patients.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2025.02.25.25322898

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 1 Not valid · 2 Not checked · 39 Own results and interpretations

23 more claims

Create a free account to see all 46 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 14
  • It may state two separate claims as one. 7

Create a free account to see these claims.

Citations

Cites

Show all 8 references

Research works printed but not linked: 1

  1. Lee K , Firat O , Ashish Agarwal A , Fannjiang C , Sussillo D. Hallucinations in Neural Machine Translation. ICLR 2019. Submitted 27 Sept 2018. https://openreview.net/pdf?id=SkxJ-309FQ

Cited by

See all 13

Metadata

Authors
Stuhlmiller, T. J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., Furgason, J. M., Paul, M., Scarpa, F. J., Kesari, S., Newton, M., Wong, K. K., Kramer, G. A., & Shapiro, M. A.
Affiliations
  • xCures, Inc. Oakland, CA
  • NeoGenomics (United States) (US)
  • The Lundquist Institute (US)
Corresponding author
Stuhlmiller, Timothy J.
Venue
medRxiv
Publisher
openRxiv
Published
2025-02-26
OpenAlex
W4408007551
Cited by
13
References
6
Citations by year
2026: 10 · 2025: 3
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion, cc-by-nc)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
electronic health records, large language models, clinical information extraction, named entity recognition, relation extraction, retrieval-augmented generation, FHIR, cancer diagnosis, treatment lines, objective response rate, medication extraction, radiotherapy, surgical procedures, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models. https://app.cochat.ai/papers/10.1101/2025.02.25.25322898
The paper
Stuhlmiller, T. J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., Furgason, J. M., Paul, M., Scarpa, F. J., Kesari, S., Newton, M., Wong, K. K., Kramer, G. A., & Shapiro, M. A. (2025). A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models. medRxiv. https://doi.org/10.1101/2025.02.25.25322898

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.