Natural language processing to adjudicate heart failure hospitalizations in global clinical trials
Open accessAbstract
BACKGROUND: Medical record review by a physician clinical events committee is the gold standard for identifying cardiovascular outcomes in clinical trials, but is labor-intensive and poorly reproducible. Automated outcome adjudication by artificial intelligence (AI) could enable larger and less expensive clinical trials but has not been validated in global studies. METHODS: We developed a novel model for automated AI-based heart failure adjudication (Heart Failure Natural Language Processing) using hospitalizations from 3 international clinical outcomes trials. This model was tested on potential heart failure hospitalizations from the DELIVER trial (Dapagliflozin Evaluation to Improve the Lives of Patients With Preserved Ejection Fraction Heart Failure), a cardiovascular outcomes trial comparing dapagliflozin with placebo in 6063 patients with heart failure with mildly reduced or preserved ejection fraction. AI-based adjudications were compared with adjudications from a clinical events committee that followed Food and Drug Administration-based criteria. RESULTS: AI-based adjudication agreed with the clinical events committee in 83% of events. A strategy of human review for events that the AI model deemed uncertain (16%) would have achieved 91% agreement with the clinical events committee while reducing the adjudication workload by 84%. The estimated effect of dapagliflozin on heart failure hospitalization was nearly identical with AI-based adjudication (hazard ratio, 0.76 [95% CI, 0.66-0.88]) compared with clinical events committee adjudication (hazard ratio, 0.77 [95% CI, 0.67-0.89]). The AI model extracted symptoms, signs, and treatments of heart failure from each medical record in tabular format and quoted sentences documenting them. CONCLUSIONS: AI-based adjudication of clinical outcomes has the potential to improve the efficiency of global clinical trials while preserving accuracy and interpretability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Marti-Castellote et al. [91] reduced clinical event adjudication effort by 84% through a hybrid AI-human strategy enabled by interpretable evidence quotation.
“Marti-Castellote et al. [91] reduced clinical event adjudication effort by 84% through a hybrid AI-human strategy enabled by interpretable evidence quotation”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [91]
The Abstract (Methods) estimates that human review of the 16% of events the AI deemed uncertain would reduce adjudication workload by 84%; it does not report an achieved reduction. The Discussion says quoted evidence helps human reviewers check the model’s reasoning, but identifies uncertainty assessment—not evidence quotation—as the basis for selecting events for human review. (LLM, conf=0.88)
Experimental — accuracy not yet measuredSome explanations cite specific, retrievable evidence, e.g. RAG-grounded source citations or knowledge-graph paths, enabling independent verification.
“Some explanations cite specific, retrievable evidence, e.g. RAG-grounded source citations or knowledge-graph paths, enabling independent verification . This characterizes the RAG-grounded approaches in our review [63, 87,91,92,94,99].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [91]
The Methods (The GPT4o Component) say the model quotes dossier sentences supporting each criterion, and the Discussion says a human reviewer may double-check its reasoning. That supports the claim about specific evidence enabling review, but the paper does not describe its approach as RAG-grounded or based on knowledge-graph paths. (LLM, conf=0.85)
Experimental — accuracy not yet measuredSystems that automate evidence retrieval and present explanations within EHR interfaces reduce the effort required for verification.
“Systems that automate evidence retrieval and present explanations within EHR interfaces reduce the effort required for verification [91].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [91]
The Discussion (page unavailable) says the model quotes dossier sentences supporting its decisions, allowing reviewers to double-check its reasoning, and estimates that combining it with human review could reduce adjudication effort. It does not describe automated evidence retrieval or explanations presented within an EHR interface, nor does it establish a reduction in verification effort specifically. (LLM, conf=0.86)
Experimental — accuracy not yet measuredHybrid human-AI workflows leverage explanation confidence to triage cases.
“Hybrid human-AI workflows leverage explanation confidence to triage cases: high-confidence predictions with clear explanations may proceed with minimal oversight, while uncertain or poorly-explained cases are routed to human review [91,94].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [91]
The Results section, “HF-NLP Score and Hybrid NLP + Human Adjudication,” evaluates hypothetical strategies that accept more-confident AI adjudications and send uncertain cases for human review. The Discussion says the model’s quoted evidence lets reviewers check its reasoning, but does not propose routing poorly explained cases to human review or using explanation quality as a triage criterion. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dapagliflozin in Heart Failure with Mildly Reduced or Preserved Ejection Fraction
Solomon, S. D., McMurray, J. J. V., Claggett, B., de Boer, R. A., DeMets, D., Hernandez, A. F., Inzucchi, S. E., Kosiborod, M. N., Lam, C. S. P., Martinez, F., et al. (2022) · New England Journal of Medicine · doi:10.1056/nejmoa2206286 · Cited by 2,884 · see citing works
Self-citationTyl, B., Lopez Sendon, J., Borer, J. S., Lopez De Sa, E., Lerebours, G., Varin, C., De Montigny, A., Pannaux, M., & Komajda, M. (2020) · Circulation: Heart Failure · doi:10.1161/circheartfailure.119.006720 · Cited by 29 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- [1]Rory Collins, Louise Bowman, Martin Landray, Richard Peto. The Magic of Randomization versus the Myth of Real-World Evidence. N Engl J Med. 2020; 382(7): 674– 678. doi: 10.1056/NEJMsb1901642 32053307 · DOI
- [2]Moore TJ, Zhang H, Anderson G, Alexander GC. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015–2016. JAMA Intern Med. 2018; 178(11): 1451– 1457. doi: 10.1001/jamainternmed.2018.3931 30264133 PMC6248200 · DOI · Paper page
- [3]Moore Thomas J, Heyward James, Anderson Gerard, Alexander G Caleb. Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015–2017: a cross-sectional study. BMJ Open. 2020; 10(6): e038863. doi: 10.1136/bmjopen-2020-038863 PMC7295430 32532786 · DOI
- [4]Hicks Karen A, Mahaffey Kenneth W, Mehran Roxana, 2017 Cardiovascular and Stroke Endpoint Definitions for Clinical Trials. Circulation. 2018; 137(9): 961– 972. doi: 10.1161/CIRCULATIONAHA.117.033502 29483172 · DOI · Paper page
- [5]Abraham WT, Psotka MA, Fiuzat M, Standardized definitions for evaluation of heart failure therapies: scientific expert panel from the Heart Failure Collaboratory and Academic Research Consortium. Eur J Heart Fail. 2020; 22(12): 2175– 2186. doi: 10.1002/ejhf.2018 33017862 · DOI
- [7]Cunningham Jonathan W, Singh Pulkit, Reeder Christopher, Natural Language Processing for Adjudication of Heart Failure in the Electronic Health Record. JACC Heart Fail. 2023; 11(7): 852– 854. doi: 10.1016/j.jchf.2023.02.012 36939660 PMC10694785 · DOI · Paper page
- [8]Cunningham JW, Singh P, Reeder C, Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial. JAMA Cardiol. 2023; 9(2): 174– 181. doi: 10.1001/jamacardio.2023.4859 PMC10640703 37950744 · DOI · Paper page
- [9]Ambrosy AP, Parikh RV, Sung SH, A Natural Language Processing–Based Approach for Identifying Hospitalizations for Worsening Heart Failure Within an Integrated Health Care Delivery System. JAMA Netw Open. 2021; 4(11):e2135152-e2135152. doi: 10.1001/jamanetworkopen.2021.35152 PMC8609413 34807259 · DOI
- [10]Goto S, Homilius M, John JE, Artificial intelligence-enabled event adjudication: estimating delayed cardiovascular effects of respiratory viruses. medRxiv. Published online January 1, 2020:2020.11.12.20230706. doi: 10.1101/2020.11.12.20230706 · DOI
- [11]Mahaffey KW, Gibson CM, Lopes RD. Innovation in Event Adjudication—Human vs Machine. JAMA Cardiol. 2024; 9(2): 101– 102. doi: 10.1001/jamacardio.2023.4900 37950739 · DOI
- [12]Hernandez AF, Lindsell CJ. The Future of Clinical Trials: Artificial to Augmented to Applied Intelligence. JAMA. 2023; 330(21): 2061– 2063. doi: 10.1001/jama.2023.23822 37950740 · DOI · Paper page
- [13]Vardeny O, Kim K, Udell JA, Effect of High-Dose Trivalent vs Standard-Dose Quadrivalent Influenza Vaccine on Mortality or Cardiopulmonary Hospitalization in Patients With High-risk Cardiovascular Disease: A Randomized Clinical Trial. JAMA. 2021; 325(1): 39– 49. doi: 10.1001/jama.2020.23649 33275134 PMC7718608 · DOI
- [14]Pfeffer Marc A, Claggett Brian, Lewis Eldrin F, Angiotensin Receptor–Neprilysin Inhibition in Acute Myocardial Infarction. N Engl J Med. 2021; 385(20): 1845– 1855. doi: 10.1056/NEJMoa2104508 34758252 · DOI
- [15]Solomon SD, McMurray JJV, Anand IS, Angiotensin–Neprilysin Inhibition in Heart Failure with Preserved Ejection Fraction. N Engl J Med. 2019; 381(17): 1609– 1620. doi: 10.1056/NEJMoa1908655 31475794 · DOI · Paper page
- [17]Kay A. Tesseract: an open-source optical character recognition engine. Linux J. 2007; 2007(159): 2.
- [20]Li Y, Wehbe RM, Ahmad FS, Wang H, Luo Y. Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences. Published online 2022. doi: 10.48550/ARXIV.2201.11838 · DOI
- [21]Open AI, Achiam J, Adler S, GPT-4 Technical Report. Published online 2024. https://arxiv.org/abs/2303.08774 · arXiv
- [22]Felker GM, Butler J, Januzzi JL, Desai AS, McMurray JJV, Solomon SD. Probabilistic Readjudication of Heart Failure Hospitalization Events in the PARAGON-HF Study. Circulation. 2021; 143(23): 2316– 2318. doi: 10.1161/CIRCULATIONAHA.121.054496 34097449 · DOI
- [23]Sharma A, Mahaffey KW, Gibson CM, Clinical events classification (CEC) in clinical trials: Report on the current landscape and future directions — proceedings from the CEC Summit 2018. Am Heart J. 2022; 246: 93– 104. doi: 10.1016/j.ahj.2021.12.012 34973948 · DOI
- [24]Kung TH, Cheatham M, Medenilla A, Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2(2): e0000198. doi: 10.1371/journal.pdig.0000198 36812645 PMC9931230 · DOI · Paper page
- [26]Ozan Unlu, Jiyeon Shin, Mailly Charlotte J, Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening. NEJM AI. 2024; 1(7): AIoa2400181. doi: 10.1056/AIoa2400181 · DOI
Cited by
Sorted by citations
Artificial Intelligence-Enabled ECG Analysis to Predict Incident Heart Failure
Khurshid, S., Friedman, S. F., Kany, S., D’Souza, V., Roshandelpoor, A., Kosyakovsky, L. B., Chitsazan, M., Cunningham, J. W., Singh, P., Lau, E. S., et al. (2026) · Circulation: Heart Failure · doi:10.1161/circheartfailure.125.013927 · Cited by 2 · see citing works
Self-citationExplainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
- Authors
- Marti-Castellote, P. M., Reeder, C., Claggett, B., Singh, P., Lau, E. S., Khurshid, S., Batra, P., Lubitz, S. A., Maddah, M., Vardeny, O., Lewis, E. F., Pfeffer, M., Jhund, P., Desai, A. S., McMurray, J. J. V., Ellinor, P. T., Ho, J. E., Solomon, S. D., & Cunningham, J. W.
- Affiliations
- Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Boston, MA (P.M.M.-C., B.C., M.P., A.S.D., S.D.S., J.W.C.).
- Broad Institute (US)
- Brigham and Women's Hospital (US)
- University of Minnesota (US)
- Stanford Medicine (US)
- British Heart Foundation (GB)
- ORCID
- Marti-Castellote, Pablo M.
- Reeder, Christopher
- Claggett, Brian
- Singh, Pulkit
- Lau, Emily S.
- Khurshid, Shaan
- Batra, Puneet
- Lubitz, Steven A.
- Maddah, Mahnaz
- Vardeny, Orly
- Lewis, Eldrin F.
- Pfeffer, Marc
- Jhund, Pardeep
- Desai, Akshay S.
- McMurray, John J.V.
- Ellinor, Patrick T.
- Ho, Jennifer E.
- Solomon, Scott D.
- Cunningham, Jonathan W.
- Venue
- Circulation: Heart Failure
- Published
- 2025-01
- Volume
- 18
- Issue
- 1
- Pages
- e012514-e012514
- ISSN
- 1941-3289 (ISSN-L)
- 1941-3297
- PMC
- PMC11910182
- OpenAlex
- W4404439584
- Cited by
- 14
- References
- 2
- Citations by year
- 2026: 9 · 2025: 5
- FWCI
- 4.28
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Locations
- Circulation Heart Failure (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- Funders
- AstraZeneca
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Failure Treatment and Management
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- heart failure hospitalization, natural language processing, artificial intelligence, dapagliflozin, heart failure with mildly reduced ejection fraction, medical records review, trial efficiency
- MeSH terms
- Sodium-Glucose Transporter 2 Inhibitors, Artificial Intelligence, Benzhydryl Compounds, Clinical Trials as Topic, Female, Glucosides, Heart Failure, Hospitalization, Humans, Male, Natural Language Processing, Stroke Volume, Reproducibility of Results, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
