Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial
Open accessAbstract
Importance: The gold standard for outcome adjudication in clinical trials is medical record review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication of medical records by natural language processing (NLP) may offer a more resource-efficient alternative but this approach has not been validated in a multicenter setting. Objective: To externally validate the Community Care Cohort Project (C3PO) NLP model for heart failure (HF) hospitalization adjudication, which was previously developed and tested within one health care system, compared to gold-standard CEC adjudication in a multicenter clinical trial. Design, Setting, and Participants: This was a retrospective analysis of the Influenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated Heart Failure (INVESTED) trial, which compared 2 influenza vaccines in 5260 participants with cardiovascular disease at 157 sites in the US and Canada between September 2016 and January 2019. Analysis was performed from November 2022 to October 2023. Exposures: Individual sites submitted medical records for each hospitalization. The central INVESTED CEC and the C3PO NLP model independently adjudicated whether the cause of hospitalization was HF using the prepared hospitalization dossier. The C3PO NLP model was fine-tuned (C3PO + INVESTED) and a de novo NLP model was trained using half the INVESTED hospitalizations. Main Outcomes and Measures: Concordance between the C3PO NLP model HF adjudication and the gold-standard INVESTED CEC adjudication was measured by raw agreement, κ, sensitivity, and specificity. The fine-tuned and de novo INVESTED NLP models were evaluated in an internal validation cohort not used for training. Results: Among 4060 hospitalizations in 1973 patients (mean [SD] age, 66.4 [13.2] years; 514 [27.4%] female and 1432 [72.6%] male]), 1074 hospitalizations (26%) were adjudicated as HF by the CEC. There was good agreement between the C3PO NLP and CEC HF adjudications (raw agreement, 87% [95% CI, 86-88]; κ, 0.69 [95% CI, 0.66-0.72]). C3PO NLP model sensitivity was 94% (95% CI, 92-95) and specificity was 84% (95% CI, 83-85). The fine-tuned C3PO and de novo NLP models demonstrated agreement of 93% (95% CI, 92-94) and κ of 0.82 (95% CI, 0.77-0.86) and 0.83 (95% CI, 0.79-0.87), respectively, vs the CEC. CEC reviewer interrater reproducibility was 94% (95% CI, 93-95; κ, 0.85 [95% CI, 0.80-0.89]). Conclusions and Relevance: The C3PO NLP model developed within 1 health care system identified HF events with good agreement relative to the gold-standard CEC in an external multicenter clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility. Further study is needed to determine whether NLP will improve the efficiency of future multicenter clinical trials by identifying clinical events at scale.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Combining ICD codes with pretrained transformer-based models applied to discharge summaries can improve the screening of HF hospitalizations.
“To address this limitation, previous works have demonstrated that combining ICD codes with pretrained transformer-based models applied to discharge sum maries can improve the screening of HF hospitalizations [10].”
From Assessing large language models for acute heart failure classification and information extraction from French clinical notes (2025) · cites this paper as [10]
The Introduction (page unavailable) describes a transformer-based model that adjudicates HF hospitalizations from discharge summaries, and the Results report good agreement with physician adjudication. But the paper discusses ICD codes as an imprecise way to identify HF hospitalizations; it does not report improved screening from combining ICD codes with the NLP model. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Learning--based Text Classification
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad-Khasmakhi, N., Asgari-Chenaghlu, M., & Gao, J. (2021) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,967 · see citing works
2017 Cardiovascular and Stroke Endpoint Definitions for Clinical Trials
Hicks, K. A., Mahaffey, K. W., Mehran, R., Nissen, S. E., Wiviott, S. D., Dunn, B., Solomon, S. D., Marler, J. R., Teerlink, J. R., Farb, A., et al. (2018) · Circulation · doi:10.1161/circulationaha.117.033502 · Cited by 750 · see citing works
Self-citationElectronic health records to facilitate clinical research
Cowie, M. R., Blomster, J. I., Curtis, L. H., Duclaux, S., Ford, I., Fritz, F., Goldman, S., Janmohamed, S., Kreuzer, J., Leenay, M., et al. (2017) · Clinical Research in Cardiology · doi:10.1007/s00392-016-1025-6 · Cited by 651 · see citing works
Deep Learning Based Text Classification: A Comprehensive ReviewPreprint
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Asgari-Chenaghlu, M., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03705 · Cited by 385 · see citing works
(2022) · Cineca Institutional Research Information System (Tor Vergata University) · doi:10.1161/circulationaha · Cited by 291 · see citing works
Reproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273 · see citing works
The Prevention of Hospital Readmissions in Heart Failure
Ziaeian, B., & Fonarow, G. C. (2015) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2015.09.004 · Cited by 261 · see citing works
Mentz, R. J., Anstrom, K. J., Eisenstein, E. L., Sapp, S. K., Greene, S. J., Morgan, S., Testani, J. M., Harrington, A. H., Sachdev, V., Ketema, F., et al. (2023) · JAMA · doi:10.1001/jama.2022.23924 · Cited by 191 · see citing works
Yang, J., Soltan, A. A. S., & Clifton, D. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00614-9 · Cited by 181 · see citing works
Phenotypic Refinement of Heart Failure in a National Biobank Facilitates Genetic Discovery
Aragam, K. G., Chaffin, M. D., Levinson, R. T., McDermott, G. C., Choi, S. H., Shoemaker, M. B., Haas, M. E., Weng, L.-C. C., Lindsay, M. E., Smith, J. G., et al. (2019) · Circulation · doi:10.1161/circulationaha.118.035774 · Cited by 153 · see citing works
Self-citationVardeny, O., Kim, K., Udell, J. A., Joseph, J., Desai, A. S., Farkouh, M. E., Hegde, S. M., Hernandez, A. F., McGeer, A., Talbot, H. K., et al. (2020) · JAMA · doi:10.1001/jama.2020.23649 · Cited by 139 · see citing works
Self-citationTesseract: an open-source optical character recognition engine
Kay, A. (2007) · Linux journal · Cited by 123 · see citing works
Cohort design and natural language processing to reduce bias in electronic health records research
Khurshid, S. S., Reeder, C., Harrington, L. X., Singh, P., Sarma, G., Friedman, S. F., Achille, P. D., Diamant, N., Cunningham, J. W., Turner, A. C., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00590-0 · Cited by 80 · see citing works
Self-citationClinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencesPreprint
Li, Y., Wehbe, R. M., Ahmad, F. S., Wang, H., & Luo, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11838 · Cited by 65 · see citing works
Ambrosy, A. P., Parikh, R., Sung, S. H., Narayanan, A., Masson, R., Lam, P.-Q., Kheder, K., Iwahashi, A., Hardwick, A. B., Fitzpatrick, J. K., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.35152 · Cited by 53 · see citing works
Bielinski, S. J., Pathak, J., Carrell, D., Takahashi, P. Y., Olson, J. E., Larson, N. B., Liu, H., Sohn, S., Wells, Q. S., Denny, J. C., et al. (2015) · Journal of Cardiovascular Translational Research · doi:10.1007/s12265-015-9644-2 · Cited by 51 · see citing works
Vardeny, O., Udell, J. A., Joseph, J., Farkouh, M. E., Hernandez, A. F., McGeer, A., Talbot, H. K., Bhatt, D. L., Cannon, C. P., Goodman, S. G., et al. (2018) · American Heart Journal · doi:10.1016/j.ahj.2018.05.007 · Cited by 46 · see citing works
Self-citationStrom, J. B., Faridi, K. F., Butala, N. M., Zhao, Y., Tamez, H., Valsdottir, L. R., Brennan, J. M., Shen, C., Popma, J. J., Kazi, D. S., et al. (2020) · Circulation · doi:10.1161/circulationaha.120.046159 · Cited by 37 · see citing works
Prediction across healthcare settings: a case study in predicting emergency department disposition
Barak‐Corren, Y., Chaudhari, P. P., Perniciaro, J. L., Waltzman, M. L., Fine, A. M., & Reis, B. Y. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00537-x · Cited by 37 · see citing works
Tyl, B., Lopez Sendon, J., Borer, J. S., Lopez De Sa, E., Lerebours, G., Varin, C., De Montigny, A., Pannaux, M., & Komajda, M. (2020) · Circulation: Heart Failure · doi:10.1161/circheartfailure.119.006720 · Cited by 29 · see citing works
Natural language processing for adjudication of heart failure in the electronic health record
Cunningham, J. W., Singh, P., Reeder, C., Lau, E. S., Khurshid, S., Wang, X., Ellinor, P. T., Lubitz, S. A., Batra, P., & Ho, J. E. (2023) · JACC: Heart Failure · doi:10.1016/j.jchf.2023.02.012 · Cited by 21 · see citing works
Self-citationCarson, P. E., Teerlink, J. R., Michel Komajda, N., Anand, I. S., Anker, S. D., Butler, J., Doehner, W., Ferreira, J. P., Filippatos, G. S., Haass, M., et al. (2023) · JACC Heart Failure · doi:10.1016/j.jchf.2022.11.017 · Cited by 21 · see citing works
Impact of an Expanded Hospital Recognition Program for Heart Failure Quality of Care
Heidenreich, P. A., Zhao, X., Hernandez, A. F., et al. (2014) · Journal of the American Heart Association · doi:10.1161/jaha.114.000950 · Cited by 13 · see citing works
Goto, S., Homilius, M., John, J. E., Truslow, J. G., Werdich, A. A., Blood, A. J., Park, B. H., MacRae, C. A., & Deo, R. C. (2020) · medRxiv · doi:10.1101/2020.11.12.20230706 · Cited by 7 · see citing works
Developing New Methods for Comparing Treatments in Case-Control Studies
Danaei, G., Rasouli, B. S., Chubak, J., et al. (2021) · doi:10.25302/07.2021.me.160936748 · Cited by 1 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [10]10. Rose T, Stevenson M, Whitehead M. The Reuters Corpus volume 1 -from yesterday's news to tomorrow's language resources. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02). European Language Resources Association (ELRA); 2002. http://www.lrec-conf.org/proceedings/ lrec2002/pdf/80.pdf
Cited by
Sorted by citations
Bazoge, A., Wargny, M., Constant dit Beaufils, P., Morin, E., Daille, B., Gourraud, P.-A., & Hadjadj, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110609 · Cited by 6 · see citing works
Artificial Intelligence-Enabled ECG Analysis to Predict Incident Heart Failure
Khurshid, S., Friedman, S. F., Kany, S., D’Souza, V., Roshandelpoor, A., Kosyakovsky, L. B., Chitsazan, M., Cunningham, J. W., Singh, P., Lau, E. S., et al. (2026) · Circulation: Heart Failure · doi:10.1161/circheartfailure.125.013927 · Cited by 2 · see citing works
Self-citation
- Authors
- Cunningham, J. W., Singh, P., Reeder, C., Claggett, B., Marti-Castellote, P. M., Lau, E. S., Khurshid, S., Batra, P., Lubitz, S. A., Maddah, M., Philippakis, A., Desai, A. S., Ellinor, P. T., Vardeny, O., Solomon, S. D., & Ho, J. E.
- Affiliations
- Broad Institute (US)
- Brigham and Women's Hospital (US)
- Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
- University of Minnesota (US)
- Corresponding author
- Solomon, Scott D.
- ORCID
- Venue
- JAMA Cardiology
- Publisher
- American Medical Association (AMA)
- Published
- 2024-02-01
- Volume
- 9
- Issue
- 2
- Pages
- 174
- ISSN
- 2380-6583 (ISSN-L)
- PMC
- PMC10640703
- OpenAlex
- W4388591773
- Cited by
- 38
- References
- 26
- Citations by year
- 2026: 10 · 2025: 10 · 2024: 16 · 2023: 2
- FWCI
- 8.52
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- JAMA Cardiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- American Heart Association (23CDA1052151)
- National Heart, Lung, and Blood Institute (R01HL140224, R01HL134893, K24HL153669, K23HL169839, R01HL160003)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Failure Treatment and Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- heart failure hospitalization, natural language processing, influenza vaccination, cardiovascular disease, external validation, medical records review, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
