CoChat
Sign Up
Cited by 106Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

Explainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review

Open access
de Vries, Bart M., Zwezerijnen, Gerben J. C., Burchell, George L., van Velden, Floris H. P., Menke-van der Houven van Oordt, Catharina Willemien, Boellaard, Ronald·2023·Frontiers in Medicine
DOI 10.3389/fmed.2023.1180773PMC PMC10213317
Open this paper in CoChat — free account

Abstract

Rational: Deep learning (DL) has demonstrated a remarkable performance in diagnostic imaging for various diseases and modalities and therefore has a high potential to be used as a clinical tool. However, current practice shows low deployment of these algorithms in clinical practice, because DL algorithms lack transparency and trust due to their underlying black-box mechanism. For successful employment, explainable artificial intelligence (XAI) could be introduced to close the gap between the medical professionals and the DL algorithms. In this literature review, XAI methods available for magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET) imaging are discussed and future suggestions are made. Methods: PubMed, Embase.com and Clarivate Analytics/Web of Science Core Collection were screened. Articles were considered eligible for inclusion if XAI was used (and well described) to describe the behavior of a DL model used in MR, CT and PET imaging. Results: A total of 75 articles were included of which 54 and 17 articles described post and ad hoc XAI methods, respectively, and 4 articles described both XAI methods. Major variations in performance is seen between the methods. Overall, post hoc XAI lacks the ability to provide class-discriminative and target-specific explanation. Ad hoc XAI seems to tackle this because of its intrinsic ability to explain. However, quality control of the XAI methods is rarely applied and therefore systematic comparison between the methods is difficult. Conclusion: There is currently no clear consensus on how XAI should be deployed in order to close the gap between medical professionals and DL algorithms for clinical implementation. We advocate for systematic technical and clinical quality assessment of XAI methods. Also, to ensure end-to-end unbiased and safe integration of XAI in clinical workflow, (anatomical) data minimization and quality control methods should be included.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fmed.2023.1180773

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Explainable artificial intelligence (XAI) techniques have become indispensable instruments in medical artificial intelligence systems to solve the intrinsic opacity of deep learning algorithms.

    “Particularly for clinical and regulatory adoption, explainable artificial intelligence (XAI) techniques have become indispensable instruments in medical artificial intelligence systems to solve the intrinsic opacity of deep learning algorithms [174].”

    From Biomarker-Guided Imaging and AI-Augmented Diagnosis of Degenerative Joint Disease (2025) · cites this paper as [174]

    The Introduction says deep-learning algorithms lack transparency and that XAI can mitigate this problem by explaining model decisions. But the Discussion says current XAI methods may not suffice for clinical deployment, and the Conclusion says no single method addresses all concerns. The source therefore supports XAI’s potential importance, not the claim that it has become indispensable or solves opacity. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 90 · Cited by 106

Research works printed but not linked: 94

  1. [1]Alongi P Stefano A Comelli A Spataro A Formica G Laudicella R. Artificial intelligence applications on restaging [18F]FDG PET/CT in metastatic colorectal cancer: a preliminary report of morpho-functional radiomics classification for prediction of disease outcome. Appl Sci. (2022) 12: 2941. doi: 10.3390/app12062941 · DOI
  2. [2]Chaunzwa TL Hosny A Xu Y Shafer A Diao N Lanuti M. Deep learning classification of lung cancer histology using CT images. Sci Rep. (2021) 11: 5471. doi: 10.1038/s41598-021-84630-x, PMID: 33727623 PMC7943565 · DOI
  3. [3]de Vries BM Golla SSV Ebenau J Verfaillie SCJ Timmers T Heeman F. Classification of negative and positive 18F-florbetapir brain PET studies in subjective cognitive decline patients using a convolutional neural network. Eur J Nucl Med Mol Imaging. (2021) 48: 721– 8. doi: 10.1007/s00259-020-05006-3, PMID: 32875431 PMC8036183 · DOI
  4. [4]Prezioso E Izzo S Giampaolo F Piccialli F Dell'Aversana Orabona G Cuocolo R. Predictive medicine for salivary gland tumours identification through deep learning. IEEE J Biomed Health Inform. (2021) 26: 4869– 79. doi: 10.1109/JBHI.2021.3120178, PMID: 34648462 · DOI
  5. [5]Gunashekar DD Bielak L Hägele L Oerther B Benndorf M Grosu AL. Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology. Radiat Oncol. (2022) 17: 65. doi: 10.1186/s13014-022-02035-0, PMID: 35366918 PMC8976981 · DOI
  6. [7]González-Gonzalo C Thee EF Klaver CCW Lee AY Schlingemann RO Tufail A. Trustworthy AI: closing the gap between development and integration of AI systems in ophthalmic practice. Prog Retin Eye Res. (2022) 90: 101034. doi: 10.1016/j.preteyeres.2021.101034, PMID: 34902546 PMC11696120 · DOI · Paper page
  7. [8]Hasani N Morris MA Rahmim A Summers RM Jones E Siegel E. Trustworthy artificial intelligence in medical imaging. PET Clin. (2022) 17: 1– 12. doi: 10.1016/j.cpet.2021.09.007, PMID: 34809860 PMC8785402 · DOI
  8. [9]Ribeiro M. T. Singh S. Guestrin C. “Why should I trust you?”: explaining the predictions of any classifier. in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).
  9. [11]O'Connor JP Aboagye EO Adams JE Aerts HJWL Barrington SF Beer AJ. Imaging biomarker roadmap for cancer studies. Nat Rev Clin Oncol. (2017) 14: 169– 86. doi: 10.1038/nrclinonc.2016.162, PMID: 27725679 PMC5378302 · DOI · Paper page
  10. [13]Beckers R Kwade Z Zanca F. The EU medical device regulation: implications for artificial intelligence-based medical device software in medical physics. Phys Med. (2021) 83: 1– 8. doi: 10.1016/j.ejmp.2021.02.011, PMID: 33657513 · DOI
  11. [14]Zeineldin RA Karar ME Elshaer Z Coburger ·J Wirtz CR Burgert O. Explainability of deep neural networks for MRI analysis of brain tumors. Int J Comput Assist Radiol Surg. (2022) 17: 1673– 83. doi: 10.1007/s11548-022-02619-x, PMID: 35460019 PMC9463287 · DOI
  12. [15]Philbrick KA Yoshida K Inoue D Akkus Z Kline TL Weston AD. What does deep learning see? Insights from a classifier trained to predict contrast enhancement phase from CT images. AJR Am J Roentgenol. (2018) 211: 1184– 93. doi: 10.2214/ajr.18.20331, PMID: 30403527 · DOI
  13. [16]Martí-Juan G Frías M Garcia-Vidal A Vidal-Jordana A Alberich M Calderon W. Detection of lesions in the optic nerve with magnetic resonance imaging using a 3D convolutional neural network. Neuroimage Clin. (2022) 36: 103187. doi: 10.1016/j.nicl.2022.103187, PMID: 36126515 PMC9486565 · DOI
  14. [17]Zeiler M. Fergus R. Visualizing and understanding convolutional networks. arXiv:1311.2901 (2013). doi: 10.48550/arXiv.1311.2901 · DOI · Paper page
  15. [18]Chatterjee S das A Mandal C Mukhopadhyay B Vipinraj M Shukla A. TorchEsegeta: framework for interpretability and explainability of image-based deep learning models. Appl Sci. (2022) 12: 2022. doi: 10.3390/app12041834 · DOI
  16. [19]Springenberg J. Dosovitskiy A. Brox T. Riedmiller M. Striving for simplicity: the all convolutional net. arXiv:1412.6806 (2014). doi: 10.48550/arXiv.1412.6806 · DOI · Paper page
  17. [20]Wood DA Kafiabadi S Busaidi AA Guilhem E Montvila A Lynch J. Deep learning models for triaging hospital head MRI examinations. Med Image Anal. (2022) 78: 102391. doi: 10.1016/j.media.2022.102391, PMID: 35183876 · DOI
  18. [21]Saleem H Shahid AR Raza B. Visual interpretability in 3D brain tumor segmentation network. Comput Biol Med. (2021) 133: 104410. doi: 10.1016/j.compbiomed.2021.104410, PMID: 33894501 · DOI
  19. [22]Montavon G Binder A Lapuschkin S Samek W Müller K-R. Layer-wise relevance propagation: an overview In: Samek W Montavon G Vedaldi A Hansen L Müller KR, editors. Explainable AI: interpreting, explaining and visualizing deep learning. Lecture notes in computer science. Cham: Springer (2019). 193– 209.
  20. [23]Golla AK Tönnes C Russ T Bauer DF Froelich MF Diehl SJ. Automated screening for abdominal aortic aneurysm in CT scans under clinical conditions using deep learning (2021) Diagnostics, 11: 2131. doi: 10.3390/diagnostics11112131, PMID: 34829478 PMC8621263 · DOI
  21. [24]Shi W Tong L Zhu Y Wang MD. COVID-19 automatic diagnosis with radiographic imaging: explainable attention transfer deep neural networks. IEEE J Biomed Health Inform. (2021) 25: 2376– 87. doi: 10.1109/jbhi.2021.3074893, PMID: 33882010 PMC8545079 · DOI
  22. [25]Karim MR Jiao J Dohmen T Cochez M Beyan O Rebholz-Schuhmann D. DeepKneeExplainer: explainable knee osteoarthritis diagnosis from radiographs and magnetic resonance imaging. IEEE Access. (2021) 9: 39757– 80. doi: 10.1109/ACCESS.2021.3062493 · DOI
  23. [26]Lopatina A Ropele S Sibgatulin R Reichenbach JR Güllmar D. Investigation of deep-learning-driven identification of multiple sclerosis patients based on susceptibility-weighted images using relevance analysis. Front Neurosci. (2020) 14: 609468. doi: 10.3389/fnins.2020.609468, PMID: 33390890 PMC7775402 · DOI
  24. [27]Shrikumar A. Greenside P. Kundaje A. Learning important features through propagating activation differences. arXiv:1704.02685 (2017). doi: 10.48550/arXiv.1804.02391 · DOI · Paper page
  25. [28]Gulum MA Trombley CM Kantardzic M. A review of explainable deep learning cancer detection models in medical imaging. Appl Sci. (2021) 11: 2021– 5. doi: 10.3390/app11104573 · DOI
  26. [29]Singh A Sengupta S Lakshminarayanan V. Explainable deep learning models in medical image analysis. J Imaging. (2020) 6: 52. doi: 10.3390/jimaging6060052, PMID: 34460598 PMC8321083 · DOI · Paper page
  27. [30]Wang C Ma J Shao J Zhang S Liu Z Yu Y. Predicting EGFR and PD-L1 status in NSCLC patients using multitask AI system based on CT images. Front Immunol. (2022) 13: 813072. doi: 10.3389/fimmu.2022.813072, PMID: 35250988 PMC8895233 · DOI
  28. [31]Kumar A Manikandan R Kose U Gupta D Satapathy SC. Doctor's dilemma: evaluating an explainable subtractive spatial lightweight convolutional neural network for brain tumor diagnosis. ACM Trans Multimedia Comput Commun Appl. (2021) 17: 1– 26. doi: 10.1145/3457187 · DOI
  29. [32]Uyulan C Erguzel TT Turk O Farhad S Metin B Tarhan N. A class activation map-based interpretable transfer learning model for automated detection of ADHD from fMRI data. Clin EEG Neurosci. (2022): 15500594221122699. doi: 10.1177/15500594221122699 36052402 · DOI
  30. [33]Wang CJ Hamm CA Savic LJ Ferrante M Schobert I Schlachter T. Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features. Eur Radiol. (2019) 29: 3348– 57. doi: 10.1007/s00330-019-06214-8, PMID: 31093705 PMC7243989 · DOI
  31. [34]Akatsuka J Yamamoto Y Sekine T Numata Y Morikawa H Tsutsumi K. Illuminating clues of cancer buried in prostate MR image: deep learning and expert approaches. Biomolecules. (2019) 9: 673. doi: 10.3390/biom9110673, PMID: 31671711 PMC6920905 · DOI
  32. [35]Fuhrman JD Gorre N Hu Q Li H El Naqa I Giger ML. A review of explainable and interpretable AI with applications in COVID-19 imaging. Med Phys. (2022) 49: 1– 14. doi: 10.1002/mp.15359, PMID: 34796530 PMC8646613 · DOI
  33. [36]Alshazly H Linse C Barth E Martinetz T. Explainable COVID-19 detection using chest CT scans and deep learning. Sensors. (2021) 21: 455. doi: 10.3390/s21020455, PMID: 33440674 PMC7828058 · DOI
  34. [37]Hao J Xie J Liu R Hao H Ma Y Yan K. Automatic sequence-based network for lung diseases detection in chest CT. Front Oncol. (2021) 11: 781798. doi: 10.3389/fonc.2021.781798 34926297 PMC8674429 · DOI
  35. [38]Lahsaini I El Habib Daho M Chikh MA. Deep transfer learning based classification model for covid-19 using chest CT-scans. Pattern Recognit Lett. (2021) 152: 122– 8. doi: 10.1016/j.patrec.2021.08.035 34566222 PMC8455169 · DOI
  36. [39]Garg A Salehi S Rocca M Garner R Duncan D. Efficient and visualizable convolutional neural networks for COVID-19 classification using chest CT. Expert Syst Appl. (2022) 195: 116540. doi: 10.1016/j.eswa.2022.116540, PMID: 35075334 PMC8769906 · DOI
  37. [40]Ullah F Moon J Naeem H Jabbar S. Explainable artificial intelligence approach in combating real-time surveillance of COVID19 pandemic from CT scan and X-ray images using ensemble model. J Supercomput. (2022) 78: 19246– 71. doi: 10.1007/s11227-022-04631-z, PMID: 35754515 PMC9206105 · DOI
  38. [41]Lu SY Zhang Z Zhang YD Wang SH. CGENet: a deep graph model for COVID-19 detection based on chest CT. Biology. (2022) 11: 2022– 1. doi: 10.3390/biology11010033, PMID: 35053031 PMC8773037 · DOI
  39. [42]Jadhav S Deng G Zawin M Kaufman AE. COVID-view: diagnosis of COVID-19 using chest CT. IEEE Trans Vis Comput Graph. (2022) 28: 227– 37. doi: 10.1109/tvcg.2021.3114851, PMID: 34587075 PMC8981756 · DOI
  40. [43]Nagaoka T Kozuka T Yamada T Habe H Nemoto M Tada M. A deep learning system to diagnose COVID-19 pneumonia using masked lung CT images to avoid AI-generated COVID-19 diagnoses that include data outside the lungs. Adv Biomed Eng. (2022) 11: 76– 86. doi: 10.14326/abe.11.76 · DOI
  41. [44]Suri JS Agarwal S Chabert GL Carriero A Paschè A Danna PSC. COVLIAS 20-cXAI: cloud-based explainable deep learning system for COVID-19 lesion localization in computed tomography scans. Diagnostics. (2022) 12: 1482. doi: 10.3390/diagnostics12061482 35741292 PMC9221733 · DOI
  42. [45]Pennisi M Kavasidis I Spampinato C Schinina V Palazzo S Salanitri FP. An explainable AI system for automated COVID-19 assessment and lesion categorization from CT-scans. Artif Intell Med. (2021) 118: 102114. doi: 10.1016/j.artmed.2021.102114 34412837 PMC8139171 · DOI
  43. [46]Draelos RL Carin L. Explainable multiple abnormality classification of chest CT volumes. Artif Intell Med. (2022) 132: 2022. doi: 10.1016/j.artmed.2022.102372, PMID: 36207074 · DOI
  44. [47]Li CF Xu YD Ding XH Zhao JJ du RQ Wu LZ. MultiR-net: a novel joint learning network for COVID-19 segmentation and classification. Comput Biol Med. (2022) 144: 105340. doi: 10.1016/j.compbiomed.2022.105340, PMID: 35305504 PMC8912982 · DOI
  45. [48]Williamson BJ Khandwala V Wang D Maloney T Sucharew H Horn P. Automated grading of enlarged perivascular spaces in clinical imaging data of an acute stroke cohort using an interpretable, 3D deep learning framework. Sci Rep. (2022) 12: 788. doi: 10.1038/s41598-021-04287-4, PMID: 35039524 PMC8764081 · DOI
  46. [49]Kim KH Koo HW Lee BJ Yoon SW Sohn MJ. Cerebral hemorrhage detection and localization with medical imaging for cerebrovascular disease diagnosis and treatment using explainable deep learning. J Korean Phys Soc. (2021) 79: 321– 7. doi: 10.1007/s40042-021-00202-2 · DOI
  47. [50]Singh A Kwiecinski J Miller RJH Otaki Y Kavanagh PB van Kriekinge S. Deep learning for explainable estimation of mortality risk from myocardial positron emission tomography images. Circ Cardiovasc Imaging. (2022) 15: e014526. doi: 10.1161/circimaging.122.014526, PMID: 36126124 PMC10035936 · DOI
  48. [51]Jain V Nankar O Jerrish DJ Gite S Patil S Kotecha K. A novel AI-based system for detection and severity prediction of dementia using MRI. IEEE Access. (2021) 9: 154324– 46. doi: 10.1109/ACCESS.2021.3127394 · DOI
  49. [52]Hu M Qian X Liu S Koh AJ Sim K Jiang X. Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive convolutional neural networks. Schizophr Res. (2022) 243: 330– 41. doi: 10.1016/j.schres.2021.06.011, PMID: 34210562 · DOI
  50. [53]Islam MN Hasan M Hossain MK Alam MGR Uddin MZ Soylu A. Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from CT-radiography. Sci Rep. (2022) 12: 11440. doi: 10.1038/s41598-022-15634-4, PMID: 35794172 PMC9259587 · DOI
  51. [54]Zhang X Han L Zhu W Sun L Zhang D. An explainable 3D residual self-attention deep neural network for joint atrophy localization and Alzheimer's disease diagnosis using structural MRI. IEEE J Biomed Health Inform. (2021) 26: 5289– 97. doi: 10.1109/jbhi.2021.3066832 33735087 · DOI
  52. [55]Esmaeili M Vettukattil R Banitalebi H Krogh NR Geitung JT. Explainable artificial intelligence for human-machine interaction in brain tumor localization. J Pers Med. (2021) 11: 1213. doi: 10.3390/jpm11111213 34834566 PMC8618183 · DOI
  53. [56]Wang SH Govindaraj V Gorriz JM Zhang X Zhang YD. Explainable diagnosis of secondary pulmonary tuberculosis by graph rank-based average pooling neural network. J Ambient Intell Humaniz Comput. (2021). doi: 10.1007/s12652-021-02998-0 · DOI
  54. [57]Windisch P Weber P Fürweger C Ehret F Kufeld M Zwahlen D. Implementation of model explainability for a basic brain tumor detection using convolutional neural networks on MRI slices. Neuroradiology. (2020) 62: 1515– 8. doi: 10.1007/s00234-020-02465-1, PMID: 32500277 · DOI
  55. [58]Zhang F Pan B Shao P Liu P Alzheimer's Disease Neuroimaging Initiative Australian Imaging Biomarkers Lifestyle Flagship Study of Ageing. A single model deep learning approach for Alzheimer’s disease diagnosis. Neuroscience. (2022) 491: 200– 14. doi: 10.1016/j.neuroscience.2022.03.026, PMID: 35398507 · DOI
  56. [59]Chattopadhay A. Sarkar A. Howlader P. Balasubramanian V. N. Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (2018), 839– 847.
  57. [60]Termine A Fabrizio C Caltagirone C Petrosini L On Behalf Of The Frontotemporal Lobar Degeneration Neuroimaging. A reproducible deep-learning-based computer-aided diagnosis tool for frontotemporal dementia using MONAI and clinica frameworks. Life. (2022) 12: 947. doi: 10.3390/life12070947, PMID: 35888037 PMC9323676 · DOI
  58. [61]Lin QH Niu YW Sui J Zhao WD Zhuo C Calhoun VD. SSPNet: an interpretable 3D-CNN for classification of schizophrenia using phase maps of resting-state complex-valued fMRI data. Med Image Anal. (2022) 79: 102430. doi: 10.1016/j.media.2022.102430, PMID: 35397470 · DOI
  59. [62]Palatnik de Sousa I Vellasco M Costa da Silva E. Explainable artificial intelligence for bias detection in COVID CT-scan classifiers. Sensors. (2021) 21: 5657. doi: 10.3390/s21165657, PMID: 34451100 PMC8402377 · DOI · Paper page
  60. [63]Garcia M Dosenbach N Kelly C In: Kelly C, editor. BrainQCNet: a deep learning attention-based model for multi-scale detection of artifacts in brain structural MRI scans. Dublin: Trinity College Institute of Neuroscience (2022) PMC12290559 40800355
  61. [64]Dasanayaka S Shantha V Silva S Meedeniya D Ambegoda T. Interpretable machine learning for brain tumour analysis using MRI and whole slide images. Softw Imp. (2022) 13: 100340– 8. doi: 10.1016/j.simpa.2022.100340 · DOI
  62. [65]Shahamat H Saniee Abadeh M. Brain MRI analysis using a deep learning based evolutionary approach. Neural Netw. (2020) 126: 218– 34. doi: 10.1016/j.neunet.2020.03.017, PMID: 32259762 · DOI
  63. [66]Petsiuk V. Das A. Saenko K. Rise: randomized input sampling for explanation of black-box models. arXiv:180607421 (2018). doi: 10.48550/arXiv.1806.07421 · DOI
  64. [67]Gaur L Bhandari M Razdan T Mallik S Zhao Z. Explanation-driven deep learning model for prediction of brain tumour status using MRI image data. Front Genet. (2022) 13: 822666. doi: 10.3389/fgene.2022.822666, PMID: 35360838 PMC8964286 · DOI
  65. [68]Ahsan MM Gupta KD Islam MM Sen S Rahman ML Hossain MS. COVID-19 symptoms detection based on NasNetMobile with explainable AI using various imaging modalities. Mach. Learn. Knowl. Extr. (2020) 2: 490– 504. doi: 10.3390/make2040027 · DOI
  66. [69]Ahsan MM Nazim R Siddique Z Huebner P. Detection of COVID-19 patients from CT scan and chest X-ray data using modified MobileNetV2 and LIME. Healthcare. (2021) 9: 1099. doi: 10.3390/healthcare9091099 34574873 PMC8465084 · DOI
  67. [70]Lundberg S. Lee S.-I. A unified approach to interpreting model predictions. arXiv.1705.07874 (2017). doi: 10.48550/arXiv.1705.07874 · DOI · Paper page
  68. [71]van der Velden BH Janse MH Ragusi MA Loo CE Gilhuijs KG. Volumetric breast density estimation on MRI using explainable deep learning regression. Sci Rep. (2020) 10: 18095– 9. doi: 10.1038/s41598-020-75167-6, PMID: 33093572 PMC7581772 · DOI
  69. [72]Sanyal J Banerjee I Hahn L Rubin D. An automated two-step pipeline for aggressive prostate lesion detection from multi-parametric MR sequence. AMIA Jt Summits Transl Sci Proc. (2020) 2020: 552– 60. 32477677 PMC7233091
  70. [73]Roest C Kwee TC Saha A Fütterer JJ Yakar D Huisman H. AI-assisted biparametric MRI surveillance of prostate cancer: feasibility study. Eur Radiol. (2022) 33: 89– 96. doi: 10.1007/s00330-022-09032-7, PMID: 35960339 PMC9755080 · DOI
  71. [74]Gao KT Tibrewala R Hess M Bharadwaj UU Inamdar G Link TM. Automatic detection and voxel-wise mapping of lumbar spine Modic changes with deep learning. JOR Spine. (2022) 5: e1204. doi: 10.1002/jsp2.1204, PMID: 35783915 PMC9238279 · DOI
  72. [75]Wang X Jiang L Li L Xu M Deng X Dai L. Joint learning of 3D lesion segmentation and classification for explainable COVID-19 diagnosis. IEEE Trans Med Imaging. (2021) 40: 2463– 76. doi: 10.1109/tmi.2021.3079709, PMID: 33983881 PMC8544955 · DOI
  73. [76]Joshi A Sivaswamy J Joshi GD. Lung nodule malignancy classification with weakly supervised explanation generation. J Med Imaging. (2021) 8: 2021. doi: 10.1117/1.JMI.8.4.044502, PMID: 34423071 PMC8370883 · DOI
  74. [77]Wang WL Charkborty G. Automatic prognosis of lung cancer using heterogeneous deep learning models for nodule detection and eliciting its morphological features. Appl Intell. (2021) 51: 2471– 84. doi: 10.1007/s10489-020-01990-z · DOI
  75. [78]Yang Z Hu Z Ji H Lafata K Vaios E Floyd S. A neural ordinary differential equation model for visualizing deep neural network behaviors in multi‐parametric MRI‐based glioma segmentation. Med Phys. (2023): 1– 14. doi: 10.1002/mp.16286 36840621 PMC10440249 · DOI
  76. [79]Decuyper M Maebe J Van Holen R Vandenberghe S. Artificial intelligence with deep learning in nuclear medicine and radiology. EJNMMI Physics. (2021) 8: 81. doi: 10.1186/s40658-021-00426-y, PMID: 34897550 PMC8665861 · DOI
  77. [80]Angelov P Soares E. Towards explainable deep neural networks (xDNN). Neural Netw. (2020) 130: 185– 94. doi: 10.1016/j.neunet.2020.07.010 32682084 · DOI · Paper page
  78. [81]Teodoro AAM Silva DH Saadi M Okey OD Rosa RL Otaibi SA. An analysis of image features extracted by CNNs to design classification models for COVID-19 and non-COVID-19. J Sign Process Syst. (2023) 95: 101– 13. doi: 10.1007/s11265-021-01714-7 PMC8572648 34777680 · DOI
  79. [82]Singh G Yow KC. Object or background: an interpretable deep learning model for COVID-19 detection from CT-scan images. Diagnostics. (2021) 11: 1732. doi: 10.3390/diagnostics11091732 34574073 PMC8465137 · DOI
  80. [83]Qian X Fu H Shi W Chen T Fu Y Shan F. M (3)lung-sys: a deep learning system for multi-class lung pneumonia screening from CT imaging. IEEE J Biomed Health Inform. (2020) 24: 3539– 50. doi: 10.1109/jbhi.2020.3030853 33048773 PMC8545176 · DOI
  81. [84]Singh G. Think positive: an interpretable neural network for image recognition. Neural Netw. (2022) 151: 178– 89. doi: 10.1016/j.neunet.2022.03.034, PMID: 35439663 PMC8978459 · DOI
  82. [85]Kwabena Patrick M Felix Adekoya A Abra Mighty A Edward BY. Capsule networks – a survey. J King Saud Univ Comput Inf Sci. (2022) 34: 1295– 310. doi: 10.1016/j.jksuci.2019.09.014 · DOI
  83. [86]Afshar P Naderkhani F Oikonomou A Rafiee MJ Mohammadi A Plataniotis KN. MIXCAPS: a capsule network-based mixture of experts for lung nodule malignancy prediction. Pattern Recogn. (2021) 116: 107942– 8. doi: 10.1016/j.patcog.2021.107942 · DOI
  84. [87]Jetley S. Lord N. A. Lee N. Torr P. H. Learn to pay attention. arXiv:180402391 (2018). doi: 10.48550/arXiv.1804.02391 · DOI · Paper page
  85. [88]Jin D Zhou B Han Y Ren J Han T Liu B. Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer’s disease. Adv Sci. (2020) 7: 2000675. doi: 10.1002/advs.202000675, PMID: 32714766 PMC7375255 · DOI
  86. [89]Wang X Yuan Y Guo D Huang X Cui Y Xia M. SSA-net: spatial self-attention network for COVID-19 pneumonia infection segmentation with semi-supervised few-shot learning. Med Image Anal. (2022) 79: 102459. doi: 10.1016/j.media.2022.102459, PMID: 35544999 PMC9027296 · DOI
  87. [90]Li X Zhao H Ren T Tian Y Yan A Li W. Inverted papilloma and nasal polyp classification using a deep convolutional network integrated with an attention mechanism. Comput Biol Med. (2022) 149: 105976. doi: 10.1016/j.compbiomed.2022.105976, PMID: 36067631 · DOI
  88. [91]Wang J Yuan C Han C Wen Y Lu H Liu C. IMAL-net: interpretable multi-task attention learning network for invasive lung adenocarcinoma screening in CT images. Med Phys. (2021) 48: 7913– 29. doi: 10.1002/mp.15293, PMID: 34674280 · DOI
  89. [92]Gu R Wang G Song T Huang R Aertsen M Deprest J. CA-net: comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans Med Imaging. (2021) 40: 699– 711. doi: 10.1109/tmi.2020.3035253, PMID: 33136540 PMC7611411 · DOI
  90. [93]Han Z Wei B Hong Y Li T Cong J Zhu X. Accurate screening of COVID-19 using attention-based deep 3D multiple instance learning. IEEE Trans Med Imaging. (2020) 39: 2584– 94. doi: 10.1109/tmi.2020.2996256, PMID: 32730211 · DOI
  91. [94]Li M Li X Jiang Y Zhang J Luo H Yin S. Explainable multi-instance and multi-task learning for COVID-19 diagnosis and lesion segmentation in CT images. Knowl Based Syst. 252: (2022): 109278. doi: 10.1016/j.knosys.2022.109278 PMC9235304 35783000 · DOI
  92. [95]Zhao W Chen W Li G Lei D Yang J Chen Y. GMILT: a novel transformer network that can noninvasively predict EGFR mutation status. IEEE Trans Neural Netw Learn Syst. (2022). doi: 10.1109/tnnls.2022.3190671 35862326 · DOI
  93. [96]Zokaeinikoo M Kazemian P Mitra P Kumara S. AIDCOV: an interpretable artificial intelligence model for detection of COVID-19 from chest radiography images. ACM Trans Manag Inf Syst. (2021) 12: 1– 20. doi: 10.1145/3466690 · DOI
  94. [97]Grote T Berens P. On the ethics of algorithmic decision-making in healthcare. J Med Ethics. (2020) 46: 205– 11. doi: 10.1136/medethics-2019-105586, PMID: 31748206 PMC7042960 · DOI · Paper page

Authors
de Vries, B. M., Zwezerijnen, G. J. C., Burchell, G. L., van Velden, F. H. P., Menke-van der Houven van Oordt, C. W., & Boellaard, R.
Affiliations
  • Amsterdam University Medical Centers (NL)
  • Vrije Universiteit Amsterdam (NL)
ORCID
Venue
Frontiers in Medicine
Publisher
Frontiers Media SA
Published
2023-05-12
Volume
10
Pages
1180773-1180773
ISSN
2296-858X (ISSN-L)
DOI
10.3389/fmed.2023.1180773
PMC
PMC10213317
OpenAlex
W4378746207
Cited by
106
References
90
Citations by year
2026: 31 · 2025: 45 · 2024: 21 · 2023: 9
FWCI
2.22
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
  • Leiden Repository (Leiden University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
explainable AI, deep learning, radiology, nuclear medicine, positron emission tomography, magnetic resonance imaging, computed tomography, post-hoc explanation, black box transparency, clinical implementation, data minimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review