Requirements and reliability of AI in the medical context
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The digital information age has been a catalyst in creating a renewed interest in Artificial Intelligence (AI) approaches, especially the subclass of computer algorithms that are popularly grouped into Machine Learning (ML). These methods have allowed one to go beyond limited human cognitive ability into understanding the complexity in the high dimensional data. Medical sciences have seen a steady use of these methods but have been slow in adoption to improve patient care. There are some significant impediments that have diluted this effort, which include availability of curated diverse data sets for model building, reliable human-level interpretation of these models, and reliable reproducibility of these methods for routine clinical use. Each of these aspects has several limiting conditions that need to be balanced out, considering the data/model building efforts, clinical implementation, integration cost to translational effort with minimal patient level harm, which may directly impact future clinical adoption. In this review paper, we will assess each aspect of the problem in the context of reliable use of the ML methods in oncology, as a representative study case, with the goal to safeguard utility and improve patient care in medicine in general.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 1×
“Efforts have been originally spearheaded by the Defense Advanced Research Agency (DARPA), some of these methods employ approximations methods such as LIME (Local Interpretable Model-Agnostic Explanations), others have focused on visualization efforts such as saliency or activation maps[38, 71].”
MethodBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Another notable application of AI method is the ability to detect diabetic retinopathy using retinal fundus photographs, which has shown to be useful to scale up in regions with lower medical resources [44, 45].”
BackgroundDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · cited 1×
“Similarly, The Cancer Imaging Archive (TCIA) houses open imaging data about 126 imaging studies (CT, PET, MRI, etc.) datasets available to be public [22–24].”
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · cited 1×
“However, this optimism[32] went through a dormant period, while computer technology went through a steep developmental period, coupled with the development of the information age, this mindset has transformed into possibility of advancement in most and skepticism in a few others [33].”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
The future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015 · cited 1×
“Recently, there has been an extensive effort to learn localized data at multiple sources in a federated fashion that encouraged decentralized learning [80].”
BackgroundThe Future of Digital Health with Federated Learning
Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B., Maier-Hein, K., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.08119 · Cited by 3,495
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · cited 1×
“Challenges for translation of AI systems in health care include logical difficulties in the implementation, barrier to adoption, sociocultural related issues and non-amenable for workflow changes [86].”
BackgroundRNA-seq: An assessment of technical reproducibility and comparison with gene expression arrays
Marioni, J. C., Mason, C. E., Mane, S. M., Stephens, M., & Gilad, Y. (2008) · Genome Research · doi:10.1101/gr.079558.108 · Cited by 2,771
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2837502 · Cited by 2,431 · cited 1×
“Early applications of these methods were in radiological sciences [53, 96, 97] and has made inroads in other oncological -omics datasets[98, 99].”
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · cited 2×
“Early applications of these methods were in radiological sciences [53, 96, 97] and has made inroads in other oncological -omics datasets[98, 99].”
BackgroundArtificial intelligence in cancer imaging: Clinical challenges and applications
Bi, W. L., Hosny, A., Schabath, M. B., Giger, M. L., Birkbak, N. J., Mehrtash, A., Allison, T., Arnaout, O., Abbosh, C., Dunn, I. F., et al. (2019) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21552 · Cited by 1,846
Tu, J. V. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00002-9 · Cited by 1,818 · cited 1×
“These challenges have led to instances of missed expectations, sometimes referred to in the literature as AI winters [18, 19].”
BackgroundArtificial intelligence in digital pathology — new tools for diagnosis and precision oncology
Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V., & Madabhushi, A. (2019) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-019-0252-y · Cited by 1,698 · cited 1×
“Nevertheless, there were some noted successes in the field that had made use of the data repositories, that include radiological sciences, radiation oncology and large patient records [4–8].”
BackgroundThe World’s Technological Capacity to Store, Communicate, and Compute Information
Hilbert, M., & López, P. (2011) · Science · doi:10.1126/science.1200970 · Cited by 1,643 · cited 1×
“It has been widely perceived that data will enable wide ranges of enterprises from finance, marketing, entertainment to medicine to better understand their customer behavior and cater future services that can be directly linked to their economic and well-being of future interests [3].”
BackgroundChecklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1×
“More specific recommendations for radiological application have been proposed, such as the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) [59].”
MethodBenjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · cited 1×
“However, to be a reliable method performance accuracy along with added clinical value are considered, these aspects are well emphasized in the recent Food and Drug Administration’s (FDA) regulations on AI systems [66, 67].”
MethodEthical and legal challenges of artificial intelligence-driven healthcare
Gerke, S., Minssen, T., & Cohen, G. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00012-5 · Cited by 1,170
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · cited 1×
“To date most of the studies that claim to have clinical translation are based on retrospective data, likely to belong to prior technologies and medical research continues to evolve at a rapid phase[55].”
BackgroundSaltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K. R., Zhao, T., Batiste, R., et al. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.03.086 · Cited by 1,027
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979
A primer on deep learning in genomics
Zou, J., Huss, M., Abid, A., Mohammadi, P., Torkamani, A., & Telenti, A. (2019) · Nature Genetics · doi:10.1038/s41588-018-0295-5 · Cited by 891
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Surgery · doi:10.1002/bjs.9736 · Cited by 889
Artificial intelligence in medical practice: the question to the answer?
Miller, D. D., & Brown, E. W. (2018) · The American Journal of Medicine · doi:10.1016/j.amjmed.2017.10.035 · Cited by 774 · cited 1×
“These methods along with ML approaches are here in stay and will support the expert medical professions, specifically more in oncology due to disease complexity [103].”
MethodMinimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 665 · cited 2×
“In recent consensus statements that provide criteria to evaluate multivariate statistical methods have been well documented, below are a list of widely accepted contenders TRIPOD, CLAIM, MI-CLAIM [58–60], which were developed by a large survey of scholars and industrial experts’ opinions.”
BackgroundDeep learning based tissue analysis predicts outcome in colorectal cancer
Bychkov, D., Linder, N., Turkki, R., Nordling, S., Kovanen, P. E., Verrill, C., Walliander, M., Lundin, M., Haglund, C., & Lundin, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-21758-3 · Cited by 644 · cited 1×
“Where-in models trained on common object images (cats, dogs, toilets, etc.) that are from mixed context have been used to pre-trained deep network models and successfully claimed to provide disease risk assessment in oncology [41, 68].”
Improving Computer-Aided Detection UsingConvolutional Neural Networks and Random View Aggregation
Roth, H. R., Lu, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2482920 · Cited by 608 · cited 1×
“Deep networks have been successfully shown to be useful in detecting polyps in colonoscopy images [46].”
BackgroundIn AI We Trust: Ethics, Artificial Intelligence, and Reliability
Ryan, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00228-y · Cited by 600 · cited 1×
“The use of AI in medicine has brought various challenges starting with postulation of the evaluation criteria, sufficient datasets to test varying approaches and establish comparable tools to create standards for clinical translation [83, 84].”
MethodDeep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study
Hosny, A., Parmar, C., Coroller, T. P., Grossmann, P., Zeleznik, R., Kumar, A., Bussink, J., Gillies, R. J., Mak, R. H., & Aerts, H. J. W. L. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002711 · Cited by 579 · cited 1×
“Early applications of these methods were in radiological sciences [53, 96, 97] and has made inroads in other oncological -omics datasets[98, 99].”
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540 · cited 1×
“One such concern has been widely referenced in a recent effort to improve transparency and reliability of these deep network findings [102], which has outlined the need for open disclosure of codebase, datasets used for training and preprocessing steps to achieve desired outcome.”
BackgroundShortliffe, E. H., Davis, R., Axline, S. G., Buchanan, B. G., Green, C. C., & Cohen, S. N. (1975) · Computers and Biomedical Research · doi:10.1016/0010-4809(75)90009-9 · Cited by 493 · cited 1×
“The system would use patient information and lab measurement for diagnosis of potential bacterial infections and recommend treatment options accordingly [30].”
BackgroundEddy, D. M., Hollingworth, W., Caro, J. J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Value in Health · doi:10.1016/j.jval.2012.04.012 · Cited by 466 · cited 1×
“Recent consensus statement describes the expectations of a decision making model in healthcare [69], which states that transparency can be achieved by explicit model definition followed by a model validation.”
BackgroundO'Sullivan, S., Nevejans, N., Allen, C., Blyth, A., Leonard, S., Pagallo, U., Holzinger, K., Holzinger, A., Sajid, M. I., & Ashrafian, H. (2019) · The International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.1968 · Cited by 437 · cited 1×
“There has been renewed debate about governance and interpretability of AI models in medical sciences, these concerns have been exacerbated due to unexplainable models using deep networks [64, 65].”
BackgroundIBM Watson, heal thyself: How IBM overpromised and underdelivered on AI health care
Strickland, E. (2019) · IEEE Spectrum · doi:10.1109/mspec.2019.8678513 · Cited by 402 · cited 1×
“These challenges have led to instances of missed expectations, sometimes referred to in the literature as AI winters [18, 19].”
BackgroundGeis, J. R., Brady, A. P., Wu, C. C., Spencer, J., Ranschaert, E., Jaremko, J. L., Langer, S. G., Borondy Kitts, A., Birch, J., Shields, W. F., et al. (2019) · Radiology · doi:10.1148/radiol.2019191586 · Cited by 388
Giger, M. L., Chan, H., & Boone, J. (2008) · Medical Physics · doi:10.1118/1.3013555 · Cited by 311
How to develop machine learning models for healthcare
Chen, P.-H. C., Liu, Y., & Peng, L. (2019) · Nature Materials · doi:10.1038/s41563-019-0345-0 · Cited by 274 · cited 1×
“It may be necessary that ML methods follow a series of internal and external validation for ML related clinical applications to improve chances of being useful for routine patient care with some extent of prospective (or live) training [56, 57].”
MethodLost in thought—the limits of the human mind and the future of medicine
Obermeyer, Z., & Lee, T. H. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1705348 · Cited by 252
Artificial Intelligence in Medicine
Schwartz, W. B., Patil, R. S., & Szolovits, P. (1987) · New England Journal of Medicine · doi:10.1056/nejm198703123161109 · Cited by 152 · cited 1×
“However, this optimism[32] went through a dormant period, while computer technology went through a steep developmental period, coupled with the development of the information age, this mindset has transformed into possibility of advancement in most and skepticism in a few others [33].”
Kulikowski, C. A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677895 · Cited by 144
Varadarajan, A. V., Bavishi, P., Ruamviboonsuk, P., Chotcomwongse, P., Venugopalan, S., Narayanaswamy, A., Cuadros, J., Kanai, K., Bresnick, G., Tadarati, M., et al. (2020) · Nature Communications · doi:10.1038/s41467-019-13922-8 · Cited by 139
Leslie, D. (2019) · Zenodo · doi:10.5281/zenodo.3240529 · Cited by 126 · cited 1×
“Model reliability in a clinical setting involves accuracy of the system, relevance for clinical translation, ethical fairness in its decision and expert/patient trust or acceptance of these systems [85].”
BackgroundTechnical and clinical overview of deep learning in radiology
Ueda, D., Shimazaki, A., & Miki, Y. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0795-3 · Cited by 100 · cited 1×
“There have been many useful applications in radiology [51, 52], one recent work shows ability to detect malignant nodules in screening CT images [53].”
BackgroundBoulbes, D. R., Costello, T., Baggerly, K., Fan, F., Wang, R., Bhattacharya, R., Ye, X., & Ellis, L. M. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-18-0227 · Cited by 41 · cited 1×
“…of clinical models by an independent research group either by re-implementing the method or using the same code base to achieve (close to) similar results would be a critical need in clinical sciences and would fulfill at the highest level (level 4) as assessed by the TRIPOD statement [73].”
MethodThe DeepMind debacle demands dialogue on data
Shah, H. (2017) · Nature · doi:10.1038/547259a · Cited by 25
Cited by
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
Hurkmans, C., Bibault, J.-E., Brock, K. K., van Elmpt, W., Feng, M., David Fuller, C., Jereczek-Fossa, B. A., Korreman, S., Landry, G., Madesta, F., et al. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110345 · Cited by 110
AI and machine learning ethics, law, diversity, and global impact
Drabiak, K., Kyzer, S., Nemov, V., & El Naqa, I. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220934 · Cited by 84
Self-citation
Metadata
- Authors
- Balagurunathan, Y., Mitchell, R., & El Naqa, I.
- Venue
- Physica Medica
- Publisher
- Elsevier BV
- Published
- 2021-03
- Volume
- 83
- Pages
- 72-78
- ISSN
- 1120-1797 (ISSN-L)
- OpenAlex
- W3137165523
- Cited by
- 91
- References
- 49
- FWCI
- 3.00
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.