A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index; 4 of the 7 sources its claims cite were checked for support.
Abstract
An LLM-powered extraction process achieves a significant gain in clinically relevant information over pre-structured data available in the EHR...
Overview
CoChat Summary
TL;DR
This paper presents an LLM-based pipeline for extracting structured medication data from heterogeneous EHR documents, including C-CDA, FHIR, and image files, with mapping to FHIR R4 and OMOP. In validation on 34 patients, the medication extractor achieved ~95% accuracy and F1 across seven fields, and large-scale deployment across 11,115 cancer patients increased medication and oncology-specific data completeness over pre-structured EHR data.
Key findings
- The medication extraction schema, validated on 34 patients and 5,789 extracted medications, achieved ~95% accuracy and F1-score across 7 data fields; per-field F1 ranged from 0.89 for end_date to 1.00 for medication_name.
- Deployment across 11,115 patients extracted ~2.6 million medication records from 169,071 documents, increasing total medications by 27% and distinct drug ingredients by 31% over structured C-CDA data.
- For oncology-specific medications, LLM extraction increased total oncology medication records by 60%, distinct oncology drug ingredients by 64%, and patients with at least one structured oncology medication by 33%; it also extracted 122 unique oncology therapies/medications not found in structured C-CDA tables.
- LLM extraction improved availability of indication for prescription (61% vs. 31%) and discontinuation reason (17% vs. 0%) compared with pre-structured C-CDA oncology medication data.
Methods
- Documents in C-CDA, FHIR, and image formats were preprocessed with parsing, OCR, metadata extraction, and chunking; text chunks were embedded using OpenAI text-embedding-3-small, and a random forest classifier was trained on embeddings to classify chunks into 26 predefined information types.
- Extraction used semantic search and document classification to select relevant documents and schema-specific prompts; LLMs performed named entity recognition and relation extraction, with outputs mapped to FHIR R4 and normalized to OMOP vocabularies.
- Validation used clinically trained human reviewers with a second QC reviewer and third referee; performance was calculated per data field using accuracy, precision, recall, and F1, and deployment was assessed in 11,115 patients with confirmed cancer diagnoses.
Limitations
- The paper states that extraction accuracy depends on high-quality input documents, and OCR performance for scanned documents such as faxes may introduce errors or omissions.
- The paper states that strict validation was necessary to mitigate hallucinations and retain only explicitly stated extractions, and that current validation is heavily biased against the LLM, using two human abstractors and a referee.
- The paper notes that future work may establish inter-rater reliability between human abstractors to position LLM extraction relative to expert human performance; the initial validation cohort was small at 34 patients, though deployment covered 11,115 patients.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 1 Not valid · 2 Not checked · 39 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Notably, those systems fall well short of the 95% accuracy and F1-scores proposed for clinical use following the original I2B2 challenge.2”
§ A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models > Introduction
What the sources say
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
The Abstract proposes approximately 95% accuracy for a clinically acceptable system and reports a best overall F-measure of 85.65%. But the Conclusions describe the acceptable F-score as minimally above 90 and only perhaps above 95, so the paper does not establish 95% as a firm F-score threshold. (LLM, conf=0.84)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It has been observed that LLMs are effective zero-shot or one-shot clinical information extractors.6”
§ A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models > LLM Training and Iteration
What the sources say
- Partly supportedcites [6] sourceExperimental — accuracy not yet measured
Section 4.1 reports that a zero-shot LLM-based method identifies diseases from clinical notes and outperforms rule-based baselines. This supports an example of effective zero-shot clinical extraction, but the cited passages do not establish effectiveness for one-shot extraction or for LLMs generally. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM extraction increased oncology medications by 60%, distinct oncology therapies by 64%, and the number of patients with structured oncology medication data by 33%.”
- Also stated in: § Results · plain
“This yielded a 33% increase in the number of patients with at least one oncology medication, a 60% increase in total oncology medication records, and a 64% increase in distinct oncology drug ingredients.”
§ A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The LLM enhanced data completeness, improving availability of indication for prescription (61% vs. 31%) and discontinuation reason (17% vs. 0%), outperforming pre-structured data in key clinical variables.”
- Also stated in: § Results · plain
“The LLM also extracted enhanced medication variables, outperforming the pre-structured C-CDA oncology medication data for the indication for prescription (61% vs. 31%) and reason for medication discontinuation (17% vs. 0%).”
§ A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deployment across 11,115 patients extracted 2.6 million medication records, increasing total medications by 27% and distinct drug ingredients by 31% over structured data in the EHR.”
- Also stated in: § Results · plain
“Document selection for the LLM-based extraction identified 84,637 additional medical documents for data extraction beyond those containing pre-structured medications (18% more), and generated an increase in 27% more total medications and 31% more distinct drug ingredients than what was available in the C-CDA medication tables.”
§ A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models > Abstract
plainstates a causen = 11,115- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
23 more claims
Create a free account to see all 46 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 7
Citations
Cites
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
Patrick, J., & Li, M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003939 · Cited by 159 · cited 1×
“1,2 However, rules-based systems are difficult to develop and can lack real-world robustness.”
Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review
Neha, F., Bhati, D., & Shukla, D. K. (2025) · AI · doi:10.3390/ai6090226 · Cited by 117
Fraile Navarro, D., Ijaz, K., Rezazadegan, D., Rahimi-Ardabili, H., Dras, M., Coiera, E., & Berkovsky, S. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105122 · Cited by 82
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61 · cited 1×
“8 When measuring LLM performance herein, we are focused on faithfulness, thus only explicitly stated, verifiable extractions are deemed valid.”
Contextualized medication information extraction using Transformer-based deep learning architectures
Chen, A., Yu, Z., Yang, X., Guo, Y., Bian, J., & Wu, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104370 · Cited by 40 · cited 1×
“3 Here, we present a highly scalable solution using general purpose LLMs for accurate clinical text classification and structured data extraction that can be implemented and adapted quickly, achieves targeted performance, enables efficient and effective data retrieval, and integrates with standard…”
Garza, M. Y., Williams, T., Ounpraseuth, S., Hu, Z., Lee, J., Snowden, J., Walden, A. C., Simon, A. E., Devlin, L. A., Young, L. W., et al. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105749 · Cited by 17
Research works printed but not linked: 1
- Lee K , Firat O , Ashish Agarwal A , Fannjiang C , Sussillo D. Hallucinations in Neural Machine Translation. ICLR 2019. Submitted 27 Sept 2018. https://openreview.net/pdf?id=SkxJ-309FQ
Cited by
Young, R., Matthews, A. M., & Poston, B. (2025) · Algorithms · doi:10.3390/a18050296 · Cited by 11
Using Large Language Models for Medical Diagnosis: A Survey and Future Trends
Ashqar, R. I., Ashqar, A., Elehnawy, M., & Ashqar, H. I. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15784-3_30 · Cited by 4
Artificial intelligence language models for medical text analysis: A systematic review
Sorayaie Azar, A., Bagherzadeh Mohasefi, J., Wiil, U. K., Naemi, A., & Ebrahimi, A. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103441 · Cited by 2
Cloud-Based Medical Named Entity Recognition: A FIT4NER-Based Approach
Tamla, P., Freund, F., & Hemmje, M. L. (2025) · Inf. · doi:10.3390/info16050395
“Other common formats include Fast Healthcare Interoperability Resources (FHIR) [24] and Consolidated Clinical Document Architecture (CCDA) [25], which both provide standardized structures for organizing and sharing clinical data.”
Artificial Intelligence for Automated, Highly Accurate, and Scalable Multimodal EHR Data AbstractionPreprint
Margaritis, G., Petridis, P., Bertsimas, D., Bloom, J. P., Hagberg, R. C., Habib, R., Shahian, D. M., & Orfanoudaki, A. (2026) · medRxiv · doi:10.64898/2026.03.16.26348522 · Cited by 1
Metadata
- Authors
- Stuhlmiller, T. J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., Furgason, J. M., Paul, M., Scarpa, F. J., Kesari, S., Newton, M., Wong, K. K., Kramer, G. A., & Shapiro, M. A.
- Affiliations
- xCures, Inc. Oakland, CA
- NeoGenomics (United States) (US)
- The Lundquist Institute (US)
- Corresponding author
- Stuhlmiller, Timothy J.
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2025-02-26
- OpenAlex
- W4408007551
- Cited by
- 13
- References
- 6
- Citations by year
- 2026: 10 · 2025: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- electronic health records, large language models, clinical information extraction, named entity recognition, relation extraction, retrieval-augmented generation, FHIR, cancer diagnosis, treatment lines, objective response rate, medication extraction, radiotherapy, surgical procedures, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models. https://app.cochat.ai/papers/10.1101/2025.02.25.25322898
- The paper
- Stuhlmiller, T. J., Rabe, A., Rapp, J., Awawda, A., Kouser, H., Lui, K., Salamon, H., Chuyka, D., Mahoney, W., Furgason, J. M., Paul, M., Scarpa, F. J., Kesari, S., Newton, M., Wong, K. K., Kramer, G. A., & Shapiro, M. A. (2025). A Scalable Method for Validated Data Extraction from Electronic Health Records with Large Language Models. medRxiv. https://doi.org/10.1101/2025.02.25.25322898
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.