Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening: Retrospective Pilot Study
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Background: Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. Objective: This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials. Methods: This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test. Results: In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. Conclusions: The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Beck, J. T., Rammage, M., Jackson, G. P., Preininger, A. M., Dankwa-Mullan, I., Roebuck, M. C., Torres, A., Holtzen, H., Coverdill, S. E., Williamson, M. P., et al. (2020) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00079 · Cited by 127 · cited 2×
“The core NLP and ML technologies designed for patient-centered matching within the system have been described elsewhere and will not be detailed in this manuscript focused on performance evaluation [13-15].”
BackgroundMethodClauser, S. B., Johnson, M. R., O'Brien, D. M., Beveridge, J. M., Fennell, M. L., & Kaluzny, A. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-63 · Cited by 49 · cited 1×
“Patients with cancer treated in multispecialty clinical settings with access to clinical trials may experience better survival and quality of life [1-5].”
Background
Cited by
Artificial intelligence applied to clinical trials: opportunities and challenges
Askin, S., Burkhalter, D., Calado, G., & El Dakrouni, S. (2023) · Health and Technology · doi:10.1007/s12553-023-00738-2 · Cited by 285
Lu, X., Yang, C., Liang, L., Hu, G., Zhong, Z., & Jiang, Z. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae243 · Cited by 77
“Unreliability of results under data constraints 1,2,5,8–11,13,15,16,18,19,21,23,27,28,31, 33,36,41,42,44 Limited generalizability 2,4–8,13–15, 17 , 19”
Chow, R., Midroni, J., Kaur, J., Boldt, G., Liu, G., Eng, L., Liu, F.-F., Haibe-Kains, B., Lock, M., & Raman, S. (2023) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djad013 · Cited by 65
Saady, M., Eissa, M., Yacoub, A. S., Hamed, A. B., & Azzazy, H. M. E.-S. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103066 · Cited by 15
Feasibility of institution‐agnostic, EHR‐integrated regional clinical trial matching
Shriver, S. P., Arafat, W., Potteiger, C., Butler, D. L., Beg, M. S., Hullings, M., Semy, S., Lister, Z., Khosama, L., Armstrong, S., et al. (2024) · Cancer · doi:10.1002/cncr.35022 · Cited by 13
Metadata
- Authors
- Haddad, T., Helgeson, J. M., Pomerleau, K. E., Preininger, A. M., Roebuck, M. C., Dankwa-Mullan, I., Jackson, G. P., & Goetz, M. P.
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2021-03-26
- Volume
- 9
- Issue
- 3
- Pages
- e27767
- ISSN
- 2291-9694 (ISSN-L)
- OpenAlex
- W3135855542
- Cited by
- 79
- References
- 2
- FWCI
- 2.72
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.