Artificial Intelligence Clinical Evidence Engine for Automatic Identification, Prioritization, and Extraction of Relevant Clinical Oncology Research
Open accessAbstract
PURPOSE: We developed a system to automate analysis of the clinical oncology scientific literature from bibliographic databases and match articles to specific patient cohorts to answer specific questions regarding the efficacy of a treatment. The approach attempts to replicate a clinician’s mental processes when reviewing published literature in the context of a patient case. We describe the system and evaluate its performance. METHODS: We developed separate ground truth data sets for each of the tasks described in the paper. The first ground truth was used to measure the natural language processing (NLP) accuracy from approximately 1,300 papers covering approximately 3,100 statements and approximately 25 concepts; performance was evaluated using a standard F1 score. The ground truth for the expert classifier model was generated by dividing papers cited in clinical guidelines into a training set and a test set in an 80:20 ratio, and performance was evaluated for accuracy, sensitivity, and specificity. RESULTS: The NLP models were able to identify individual attributes with a 0.7-0.9 F1 score, depending on the attribute of interest. The expert classifier machine learning model was able to classify the individual records with a 0.93 accuracy (95% CI, 0.9 to 0.96, P < .0001), and sensitivity and specificity of 0.95 and 0.91, respectively. Using a decision boundary of 0.5 for the positive (expert) label, the classifier demonstrated an F1 score of 0.92. CONCLUSION: The system identified and extracted evidence from the oncology literature with a high degree of accuracy, sensitivity, and specificity. This tool enables timely access to the most relevant biomedical literature, providing critical support to evidence-based practice in areas of rapidly evolving science.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The three publications that specifically mentioned titles also used abstracts in their datasets.
“while the three publications that specifically mentioned titles also used abstracts in their datasets. 43 , 73 , 74”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [73]
A model for outcome detection might learn to be biased towards outcomes frequently appearing in a certain domain, such as chemotherapy-related outcomes for cancer literature, or might favour detecting outcomes more frequent in older trial texts if the underlying training data are older or outdated.
“A model for outcome detection, for example, might learn to be biased towards outcomes frequently appearing in a certain domain, such as chemotherapy-related outcomes for cancer literature or it might favour to detect outcomes more frequent in older trial texts if the underlying training data are older or outdated. 73”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
PubMed and beyond: a survey of web tools for searching biomedical literature
Lu, Z. (2011) · Database · doi:10.1093/database/baq036 · Cited by 574 · see citing works
The Precision Medicine Initiative
Ashley, E. A. (2015) · JAMA · doi:10.1001/jama.2015.3595 · Cited by 562 · see citing works
A SIMPLE ALGORITHM FOR IDENTIFYING ABBREVIATION DEFINITIONS IN BIOMEDICAL TEXT
SCHWARTZ, A. S., & Hearst, M. A. (2002) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789812776303_0042 · Cited by 509 · see citing works
How Many Variables Can Humans Process?
Halford, G. S., Baker, R., McCredden, J., & Bain, J. (2005) · Psychological Science · doi:10.1111/j.0956-7976.2005.00782.x · Cited by 452 · see citing works
PubTator central: automated concept annotation for biomedical full text articles
Wei, C.-H., Allot, A., Leaman, R., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz389 · Cited by 400 · see citing works
Biomedical Ontologies in Action: Role in Knowledge Management, Data Integration and Decision Support
Bodenreider, O. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638585 · Cited by 368 · see citing works
Decision Support Systems in Oncology
Walsh, S., de Jong, E. E. C., van Timmeren, J. E., Ibrahim, A., Compter, I., Peerlings, J., Sanduleanu, S., Refaee, T., Keek, S. A., Larue, R. T. H. M., et al. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.18.00001 · Cited by 165 · see citing works
Computational Complexity and Human Decision-Making
Bossaerts, P. L., & Murawski, C. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.09.005 · Cited by 151 · see citing works
Abbreviation definition identification based on automatic precision estimates
Sohn, S., Comeau, D. C., Kim, W. B., & Wilbur, W. J. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-402 · Cited by 120 · see citing works
Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · see citing works
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61 · see citing works
Bao, Y., Deng, Z., Wang, Y., Kim, H., Armengol, V. D., Francisco Acevedo, C., Ouardaoui, N., Wang, C., Parmigiani, G., Barzilay, R., et al. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00042 · Cited by 59 · see citing works
HemOnc.org: A Collaborative Online Knowledge Platform for Oncology Professionals
Warner, J. L., Cowan, A. J., Hall, A. C., & Yang, P. C. (2015) · Journal of Oncology Practice · doi:10.1200/jop.2014.001511 · Cited by 40 · see citing works
Li, Q., Zhai, H., Deléger, L., Lingren, T., Kaiser, M., Stoutenborough, L., & Solti, I. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001487 · Cited by 21 · see citing works
Computerized Approach to Creating a Systematic Ontology of Hematology/Oncology Regimens
Malty, A., Jain, S. K., Yang, P. C., Harvey, K., & Warner, J. L. (2018) · JCO Clinical Cancer Informatics · doi:10.1200/cci.17.00142 · Cited by 19 · see citing works
Quertle and KNALIJ: searching PubMed has never been so easy and effective.
Giglia, E. (2011) · PubMed · Cited by 14 · see citing works
Advances in Website Information Resources to Aid in Clinical Practice
Rioth, M. J., Osterman, T. J., & Warner, J. L. (2015) · American Society of Clinical Oncology Educational Book · doi:10.14694/edbook_am.2015.35.e608 · Cited by 8 · see citing works
Warner, J. L., Anick, P., & Drews, R. E. (2013) · Journal of Oncology Practice · doi:10.1200/jop.2013.000931 · Cited by 8 · see citing works
Zhang, M., Fiol, G. D., Grout, R. W., Jonnalagadda, S., Medlin, R. P., Mishra, R., Weir, C. R., Liu, H., Mostafa, J., & Marcelo Fiszman, B. (2013) · PubMed · Cited by 5 · see citing works
Use of machine learning to identify relevant research publications in clinical oncology.
Suárez, F., Sanders, C., Stevens, R. J., Nielsen, R., Britt, M. W., Preininger, A. M., & Jackson, G. P. (2019) · Journal of Clinical Oncology · doi:10.1200/jco.2019.37.15_suppl.6558 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106 · see citing works
Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems
Yang, Q., Hao, Y., Quan, K., Yang, S., Zhao, Y., Kuleshov, V., & Wang, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581393 · Cited by 75 · see citing works
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 49 · see citing works
Emani, S., Rui, A., Rocha, H. A. L., Rizvi, R. F., Juaçaba, S. F., Jackson, G. P., & Bates, D. W. (2022) · JMIR Cancer · doi:10.2196/31461 · Cited by 39 · see citing works
Self-citationSuwanvecho, S., Suwanrusme, H., Jirakulaporn, T., Issarachai, S. S., Taechakraichana, N., Lungchukiet, P., Decha, W., Boonpakdee, W., Thanakarn, N., Wongrattananon, P., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa334 · Cited by 30 · see citing works
Self-citationTransformers for extracting breast cancer information from Spanish clinical narratives
Pabón, O. S., Montenegro, O., García-Barragán, Á., Torrente, M., Provencio, M., Menasalvas, E., & Robles, V. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102625 · Cited by 31 · see citing works
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27 · see citing works
Stenzl, A., Sternberg, C. N., Ghith, J., Serfass, L., Schijvenaars, B. J. A., & Sboner, A. (2021) · British Journal of Urology · doi:10.1111/bju.15662 · Cited by 24 · see citing works
Ahmad, P. N., Liu, Y., Khan, K., Jiang, T., & Burhan, U. (2023) · Sensors · doi:10.3390/s23239355 · Cited by 22 · see citing works
Zeng, J., Pico, C. X. C., Saridogan, T., Shufean, M. A., Kahle, M., Yang, D., Shaw, K. R. M., & Meric‐Bernstam, F. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00109 · Cited by 10 · see citing works
Deep learning to extract Breast Cancer diagnosis concepts
Pabón, O. S., Torrente, M., García-Barragán, Á., Provencio, M., Menasalvas, E., & Robles, V. (2022) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms55023.2022.00010 · Cited by 10 · see citing works
Windisch, P. Y., Dennstädt, F., Koechli, C., Foerster, R., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00150 · Cited by 5 · see citing works
Windisch, P. Y., Dennstädt, F., Koechli, C., Forster, R. E., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · medRxiv · doi:10.1101/2024.06.17.24309020 · Cited by 1 · see citing works
Rizvi, R. F., Emani, S., Rocha, H. A. L., de Aquino, C. M., Garabedian, P. M., Rui, A., Arruda, C. A. M., Sands-Lincoln, M., Rozenblum, R., Felix, W., et al. (2022) · ACI Open · doi:10.1055/s-0042-1751088
Self-citationSeeram, E., & Kanade, V. A. (2024) · Springer eBooks · doi:10.1007/978-3-031-64049-0_10
Tseng, O. L.-I., Brar, S., Dawes, M. G., Ranchod, H., Lacaille, D. V., Su, V., & Mitton, C. (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70143
Johnson, E., Smith, R. P., & Brown, L. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/j563691456
- Authors
- Saiz, F. S., Sanders, C., Stevens, R., Nielsen, R., Britt, M., Yuravlivker, L., Preininger, A. M., & Jackson, G. P.
- Affiliations
- IBM Watson Health, IBM Corporation, Cambridge, MA
- IBM (United States) (US)
- Venue
- JCO Clinical Cancer Informatics
- Published
- 2021-12
- Volume
- 5
- Issue
- 5
- Pages
- 102-111
- ISSN
- 2473-4276 (ISSN-L)
- PMC
- PMC8140792
- OpenAlex
- W3118721495
- Cited by
- 17
- References
- 21
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 4 · 2022: 5 · 2021: 2
- FWCI
- 1.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- JCO Clinical Cancer Informatics (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- Gilead Sciences
- Vanderbilt University
- Vanderbilt University Medical Center
- CVS Health
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- oncology literature, evidence extraction, natural language processing, clinical decision support, treatment efficacy, clinical guidelines, evidence-based practice, bibliographic databases
- MeSH terms
- Machine Learning, Artificial Intelligence, Humans, Medical Oncology, Natural Language Processing, Sensitivity and Specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
