The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Ensuring Reliability of Curated Electronic Health Record–Derived Data: The Validation of Accuracy for Large Language Model–/Machine Learning–Extracted Information and Data (VALID) Framework
Abstract
Large language models (LLMs) are increasingly used to extract clinical data from electronic health records, offering significant improvements in scalability and efficiency for real-world data (RWD) curation in oncology. However, the adoption of LLMs introduces new challenges in ensuring the reliability, accuracy, and fairness of extracted data, which are essential for research, regulatory, and clinical applications. Existing quality assurance frameworks for RWD and artificial intelligence (AI) do not fully address the unique error modes and complexities associated with LLM-extracted data. In this paper, we propose a comprehensive framework for evaluating the quality of clinical data extracted by LLMs. The framework integrates variable-level performance benchmarking against expert human abstraction, verification checks for internal consistency and plausibility, and replication analyses comparing LLM-extracted data to human-abstracted data sets or external standards. This multidimensional approach enables the identification of variables most in need of improvement, systematic detection of latent errors, and confirmation of data set fitness-for-purpose in real-world research. Additionally, the framework supports bias assessment by stratifying across demographic subgroups. By providing a rigorous and transparent method for assessing LLM-extracted RWD, this framework advances industry standards and supports the trustworthy use of AI-powered evidence generation in oncology research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
Good practices for quantitative bias analysis
Lash, T. L., Fox, M. P., MacLehose, R. F., Maldonado, G., McCandless, L. C., & Greenland, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu149 · Cited by 651
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
Funk, M. J., & Landi, S. N. (2014) · Current Epidemiology Reports · doi:10.1007/s40471-014-0027-z · Cited by 216
Adamson, B. J. S., Waskom, M., Blarre, A., Kelly, J., Krismer, K., Nemeth, S., Gippetti, J., Ritten, J. J., Harrison, K., Ho, G., et al. (2023) · Frontiers in Pharmacology · doi:10.3389/fphar.2023.1180962 · Cited by 103
Self-citationPadula, W. V., Kreif, N., Vanness, D. J., Adamson, B. J. S., Rueda, J. D., Felizzi, F., Jonsson, P. F., IJzerman, M. J., Butte, A. J., & Crown, W. H. (2022) · Value in Health · doi:10.1016/j.jval.2022.03.022 · Cited by 94
Self-citationBenedum, C. M., Sondhi, A., Fidyk, E., Cohen, A. B., Nemeth, S., Adamson, B. J. S., Estévez, M., & Bozkurt, S. (2023) · Cancers · doi:10.3390/cancers15061853 · Cited by 36
Self-citationFleurence, R., Kent, S., Adamson, B. J. S., Tcheng, J. E., Balicer, R., Ross, J. S., Haynes, K., Muller, P., Campbell, J., Bouée-Benhamiche, E., et al. (2024) · Value in Health · doi:10.1016/j.jval.2024.01.019 · Cited by 35
Self-citationBurford, K. G., Itzkowitz, N. G., Ortega, A. G., Teitler, J. O., & Rundle, A. G. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25981 · Cited by 33
Estévez, M., Benedum, C. M., Jiang, C., Cohen, A. B., Phadke, S., Sarkar, S., & Bozkurt, S. (2022) · Cancers · doi:10.3390/cancers14133063 · Cited by 32
Self-citationImplementing Accuracy, Completeness, and Traceability for Data Reliability
Riskin, D. J., Monda, K. L., Gagne, J. J., Reynolds, R. F., Garan, A. R., Dreyer, N. A., Muntner, P., & Bradbury, B. D. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.0128 · Cited by 29
A framework for evaluating clinical artificial intelligence systems without ground-truth annotations
Kiyasseh, D., Cohen, A., Jiang, C., & Altieri, N. (2024) · Nature Communications · doi:10.1038/s41467-024-46000-9 · Cited by 26
Self-citationCohen, A. B., Rosic, A., Harrison, K., Richey, M., Nemeth, S., Ambwani, G., Miksad, R. A., Haaland, B. A., & Jiang, C. (2023) · Applied Sciences · doi:10.3390/app13106209 · Cited by 24
Self-citationA hybrid approach to scalable real-world data curation by machine learning and human expertsPreprint
Waskom, M., Tan, K., Wiberg, H., Cohen, A. B., Wittmershaus, B. K., & Shapiro, W. (2023) · medRxiv · doi:10.1101/2023.03.06.23286770 · Cited by 10
Self-citationPPM8 A MACHINE LEARNING MODEL FOR CANCER BIOMARKER IDENTIFICATION IN ELECTRONIC HEALTH RECORDS
Ambwani, G., Cohen, A. B., Estévez, M., Singh, N., Adamson, B. J. S., Nussbaum, N. C., & Birnbaum, B. (2019) · Value in Health · doi:10.1016/j.jval.2019.04.1631 · Cited by 6
Self-citationMagee, K., Yuan, Q., Blarre, A., Cohen, A. B., Dolor, A., Krismer, K., Williams, T., & Zhang, Q. (2025) · Value in Health · doi:10.1016/j.jval.2025.04.1293 · Cited by 4
Self-citationCohen, A. B., Estévez, M., Kelly, J. G., Gippetti, J., Fidyk, E., Brandstadter, J. D., & Fajgenbaum, D. C. (2023) · Blood · doi:10.1182/blood-2023-174924 · Cited by 3
Self-citationCastellanos, E. H., Yuan, Q., Wadé, N. B., Patel, K., Rinaldi, C., Reiss, S., Hankinson, E. A., Cohen, A. B., & Estévez, M. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.2024.42.16_suppl.11149 · Cited by 3
Self-citationFidyk, E., Kalesinskas, L., Krismer, K., Blarre, A., Bouzit, L., Ritten, J. J., Gao, J., Marinescu, A., Kelly, J., Harrison, K., et al. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.2024.42.16_suppl.3610 · Cited by 2
Self-citationFidyk, E., Ward, P., Estévez, M., Krismer, K., Ritten, J. J., Marinescu, A., & Cohen, A. B. (2025) · Journal of Clinical Oncology · doi:10.1200/jco.2025.43.16_suppl.555 · Cited by 2
Self-citationYuan, Q., Dolor, A., Qian, Y., Donnelly, D., Estévez, M., Kuznetsova, Y., Singh, N., & Yerram, P. (2025) · Value in Health · doi:10.1016/j.jval.2025.04.1177 · Cited by 2
Self-citationAbstract B002: Fairness by Design: End-to-End Bias Evaluation for LLM-Generated Data
Estévez, M., Mbah, O., Sheikh, A., Ward, P., Vietorisz, E., Hildner, M. W., Dyson, L., Singh, N., & Cohen, A. B. (2025) · Clinical Cancer Research · doi:10.1158/1557-3265.aimachine-b002 · Cited by 1
Self-citation
Research works printed but not linked: 5
- Kalai AT Nachum O Vempala SS et al: Why language models hallucinate. arXiv 10.48550/arXiv.2509.04664 · DOI · Paper page
- Cohen AB, Adamson B, Larch JK, et al: Large language model extraction of PD-L1 biomarker testing details from electronic health records. AI Precis Oncol 2:49-75, 2025
- Agrawal M Adams G Nussbaum N et al: TIFTI: A framework for extracting drug intervals from longitudinal clinic notes. arXiv 10.48550/arXiv.1811.12793 · DOI
- Zhang Q, Krismer K, Lu Y, et al: Concordance of response-based clinical trial and machine learning–generated real-world end points. J Clin Oncol 43, 2025 (suppl 16; abstr e13625)
- Castellanos EH, Wittmershaus BK, Chandwani S: Raising the bar for real-world data in oncology: Approaches to quality across multiple dimensions. JCO Clin Cancer Inform 10.1200/CCI.23.00046 · DOI
Metadata
- Authors
- Estevez, M., Singh, N., Dyson, L., Adamson, B., Krismer, K., Magee, K., Yuan, Q., Hildner, M. W., Fidyk, E., Williams, T., Mbah, O., Khan, F., Seidl-Rathkopf, K., & Cohen, A. B.
- Venue
- JCO Clinical Cancer Informatics
- Published
- 2026-01
- Volume
- 10
- Issue
- 1
- ISSN
- 2473-4276
- Cited by
- 11
- References
- 23
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
