The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Ensuring Reliability of Curated Electronic Health Record–Derived Data: The Validation of Accuracy for Large Language Model–/Machine Learning–Extracted Information and Data (VALID) Framework
Abstract
Large language models (LLMs) are increasingly used to extract clinical data from electronic health records, offering significant improvements in scalability and efficiency for real-world data (RWD) curation in oncology. However, the adoption of LLMs introduces new challenges in ensuring the reliability, accuracy, and fairness of extracted data, which are essential for research, regulatory, and clinical applications. Existing quality assurance frameworks for RWD and artificial intelligence (AI) do not fully address the unique error modes and complexities associated with LLM-extracted data. In this paper, we propose a comprehensive framework for evaluating the quality of clinical data extracted by LLMs. The framework integrates variable-level performance benchmarking against expert human abstraction, verification checks for internal consistency and plausibility, and replication analyses comparing LLM-extracted data to human-abstracted data sets or external standards. This multidimensional approach enables the identification of variables most in need of improvement, systematic detection of latent errors, and confirmation of data set fitness-for-purpose in real-world research. Additionally, the framework supports bias assessment by stratifying across demographic subgroups. By providing a rigorous and transparent method for assessing LLM-extracted RWD, this framework advances industry standards and supports the trustworthy use of AI-powered evidence generation in oncology research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
Good practices for quantitative bias analysis
Lash, T. L., Fox, M. P., MacLehose, R. F., Maldonado, G., McCandless, L. C., & Greenland, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu149 · Cited by 651
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
Funk, M. J., & Landi, S. N. (2014) · Current Epidemiology Reports · doi:10.1007/s40471-014-0027-z · Cited by 216
Adamson, B. J. S., Waskom, M., Blarre, A., Kelly, J., Krismer, K., Nemeth, S., Gippetti, J., Ritten, J. J., Harrison, K., Ho, G., et al. (2023) · Frontiers in Pharmacology · doi:10.3389/fphar.2023.1180962 · Cited by 103
Self-citationPadula, W. V., Kreif, N., Vanness, D. J., Adamson, B. J. S., Rueda, J. D., Felizzi, F., Jonsson, P. F., IJzerman, M. J., Butte, A. J., & Crown, W. H. (2022) · Value in Health · doi:10.1016/j.jval.2022.03.022 · Cited by 94
Self-citationBenedum, C. M., Sondhi, A., Fidyk, E., Cohen, A. B., Nemeth, S., Adamson, B. J. S., Estévez, M., & Bozkurt, S. (2023) · Cancers · doi:10.3390/cancers15061853 · Cited by 36
Self-citationFleurence, R., Kent, S., Adamson, B. J. S., Tcheng, J. E., Balicer, R., Ross, J. S., Haynes, K., Muller, P., Campbell, J., Bouée-Benhamiche, E., et al. (2024) · Value in Health · doi:10.1016/j.jval.2024.01.019 · Cited by 35
Self-citationBurford, K. G., Itzkowitz, N. G., Ortega, A. G., Teitler, J. O., & Rundle, A. G. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25981 · Cited by 33
Research works printed but not linked: 5
- Kalai AT Nachum O Vempala SS et al: Why language models hallucinate. arXiv 10.48550/arXiv.2509.04664 · DOI · Paper page
- Cohen AB, Adamson B, Larch JK, et al: Large language model extraction of PD-L1 biomarker testing details from electronic health records. AI Precis Oncol 2:49-75, 2025
- Agrawal M Adams G Nussbaum N et al: TIFTI: A framework for extracting drug intervals from longitudinal clinic notes. arXiv 10.48550/arXiv.1811.12793 · DOI
- Zhang Q, Krismer K, Lu Y, et al: Concordance of response-based clinical trial and machine learning–generated real-world end points. J Clin Oncol 43, 2025 (suppl 16; abstr e13625)
- Castellanos EH, Wittmershaus BK, Chandwani S: Raising the bar for real-world data in oncology: Approaches to quality across multiple dimensions. JCO Clin Cancer Inform 10.1200/CCI.23.00046 · DOI
Metadata
- Authors
- Estevez, M., Singh, N., Dyson, L., Adamson, B., Krismer, K., Magee, K., Yuan, Q., Hildner, M. W., Fidyk, E., Williams, T., Mbah, O., Khan, F., Seidl-Rathkopf, K., & Cohen, A. B.
- Venue
- JCO Clinical Cancer Informatics
- Published
- 2026-01
- Volume
- 10
- Issue
- 1
- ISSN
- 2473-4276
- Cited by
- 11
- References
- 23
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ensuring Reliability of Curated Electronic Health Record–Derived Data: The Validation of Accuracy for Large Language Model–/Machine Learning–Extracted Information and Data (VALID) Framework. https://app.cochat.ai/papers/10.1200/cci-25-00215
- The paper
- Estevez, M., Singh, N., Dyson, L., Adamson, B., Krismer, K., Magee, K., Yuan, Q., Hildner, M. W., Fidyk, E., Williams, T., Mbah, O., Khan, F., Seidl-Rathkopf, K., & Cohen, A. B. (2026). Ensuring Reliability of Curated Electronic Health Record–Derived Data: The Validation of Accuracy for Large Language Model–/Machine Learning–Extracted Information and Data (VALID) Framework. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci-25-00215
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
