Interpretation and visualization techniques for deep learning models in medical imaging
Open accessThe DOI matches its Crossref record; 117 of the 120 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) approaches to medical image analysis tasks have recently become popular; however, they suffer from a lack of human interpretability critical for both increasing understanding of the methods’ operation and enabling clinical translation. This review summarizes currently available methods for performing image model interpretation and critically evaluates published uses of these methods for medical imaging applications. We divide model interpretation in two categories: (1) understanding model structure and function and (2) understanding model output. Understanding model structure and function summarizes ways to inspect the learned features of the model and how those features act on an image. We discuss techniques for reducing the dimensionality of high-dimensional data and cover autoencoders, both of which can also be leveraged for model interpretation. Understanding model output covers attribution-based methods, such as saliency maps and class activation maps, which produce heatmaps describing the importance of different parts of an image to the model prediction. We describe the mathematics behind these methods, give examples of their use in medical imaging, and compare them against one another. We summarize several published toolkits for model interpretation specific to medical imaging applications, cover limitations of current model interpretation methods, provide recommendations for DL practitioners looking to incorporate model interpretation into their task, and offer general discussion on the importance of model interpretation in medical imaging contexts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“This survey adds to the review of Huff et al. (2021), since they mostly focused on examples of visual explanation, while our survey aims for a more holistic approach including non-visual explanation, critiques on XAI, and methods for evaluating XAI.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Huff et al. (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
(2010) · Cited by 14,134
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
(2016) · Cited by 11,679
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
(2011) · Cited by 6,993
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 3×
“Layer-wise relevance propagation—Layer-wise relevance propagation (LRP) was introduced by Bach et al. in 2015 (Bach et al., 2015).”
BackgroundMethodAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 4×
“For example, (Kamnitsas et al., 2017) observed the network learning features to identify ventricles, cerebrospinal fluid (CSF), white, and gray matter on MRI images, indicating that differentiating between tissue types is useful for lesion segmentation.”
BackgroundInfluentialStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B. (2015) · arXiv · doi:10.48550/arxiv.1511.05644 · Cited by 1,579
Tutorial on Variational Autoencoders
Doersch, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05908 · Cited by 1,470
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Interpretation of Neural Networks is FragilePreprint
Ghorbani, A., Abid, A., & Zou, J. Y. (2017) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1710.10547 · Cited by 1,153
The statistics of natural images
Ruderman, D. (1994) · Network: Computation in Neural Systems · doi:10.1088/0954-898x/5/4/006 · Cited by 941
Is object localization for free? - Weakly-supervised learning with convolutional neural networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298668 · Cited by 886
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Hengstler, M., Enkel, E., & Duelli, S. (2016) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2015.12.014 · Cited by 661
Improving Computer-Aided Detection UsingConvolutional Neural Networks and Random View Aggregation
Roth, H. R., Lu, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2482920 · Cited by 608
Deep MR to CT Synthesis Using Unpaired Data
Wolterink, J. M., Dinkla, A. M., Savenije, M. H. F., Seevinck, P. R., van den Berg, C. A. T., & Išgum, I. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-68127-6_2 · Cited by 593
Deep learning for neuroimaging: a validation study
Plis, S. M., Hjelm, D. R., Salakhutdinov, R., Allen, E. A., Bockholt, H. J., Long, J. D., Johnson, H. J., Paulsen, J. S., Turner, J. A., & Calhoun, V. D. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00229 · Cited by 586
Kiran, B. R., Thomas, D. M., & Parakkal, R. (2018) · Journal of Imaging · doi:10.3390/jimaging4020036 · Cited by 557
Huynh, B. Q., Li, H., & Giger, M. L. (2016) · Journal of Medical Imaging · doi:10.1117/1.jmi.3.3.034501 · Cited by 539
Meaningful information and the right to explanation
Selbst, A. D., & Powles, J. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx022 · Cited by 516
On the interpretability of artificial intelligence in radiology: challenges and opportunities
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-M., Tengg-Kobligk, H. V., Summers, R. M., & Wiest, R. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190043 · Cited by 484
3D conditional generative adversarial networks for high-quality PET image estimation at low dose
Wang, Y., Yu, B., Wang, L., Zu, C., Lalush, D. S., Lin, W., Wu, X., Zhou, J., Shen, D., & Zhou, L. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.03.045 · Cited by 449
Wang, S., Shi, J., Ye, Z., Dong, D., Yu, D., Zhou, M., Liu, Y., Gevaert, O., Wang, K., Zhu, Y., et al. (2019) · European Respiratory Journal · doi:10.1183/13993003.00986-2018 · Cited by 435
Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418
Jetley, S., Lord, N. A., Lee, N., & Torr, P. H. S. (2018) · arXiv · doi:10.48550/arxiv.1804.02391 · Cited by 383
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354 · cited 1×
“MDNet MDNet, introduced in (Zhang et al., 2017), uses joint image and language models to provide diagnosis interpretation for images paired with text reports.”
MethodSharpness-Aware Low-Dose CT Denoising Using Conditional Generative Adversarial Network
Yi, X., & Babyn, P. (2018) · Journal of Digital Imaging · doi:10.1007/s10278-018-0056-0 · Cited by 343
Roth, H. R., Lu, L., Seff, A., Cherry, K. M., Hoffman, J., Wang, S., Liu, J., Turkbey, E., & Summers, R. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10404-1_65 · Cited by 340
Recent Advances in Autoencoder-Based Representation Learning
Tschannen, M., Bachem, O., & Lučić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05069 · Cited by 329
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291 · cited 23×
“…M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript Figure 7: Layerwise relevance propagation for model interpretation in (A) diagnosing multiple sclerosis on brain MRI (Eitel et al., 2019), and (B) visualizing evidence for Alzheimer’s disease on brain MRI (Böhle et al., 2019b).”
BackgroundMethodInfluentialCheng, P. M., & Malhi, H. S. (2017) · Journal of Digital Imaging · doi:10.1007/s10278-016-9929-2 · Cited by 253
Lin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252
Deep Convolutional AutoEncoder-based Lossy Image Compression
Cheng, Z., Sun, H., Takeuchi, M., & Katto, J. (2018) · 2018 Picture Coding Symposium (PCS) · doi:10.1109/pcs.2018.8456308 · Cited by 241
Multi-modal deep learning model for auxiliary diagnosis of Alzheimer’s disease
Zhang, F., Li, Z., Zhang, B., Du, H., Wang, B., & Zhang, X. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.04.093 · Cited by 230
Lossy Image Compression with Compressive Autoencoders
Theis, L., Shi, W., Cunningham, A., & Huszár, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00395 · Cited by 230
SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation
Sun, J., Darbehani, F., Zaidi, M., & Wang, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59719-1_77 · Cited by 207
Hu, P., Wu, F., Peng, J., Bao, Y., Chen, F., & Kong, D. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-016-1501-5 · Cited by 194
Liu, F., Guan, B., Zhou, Z., Samsonov, A., Rosas, H., Lian, K., Sharma, R., Kanarek, A., Kim, J., Guermazi, A., et al. (2019) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019180091 · Cited by 176
Deep convolutional networks for pancreas segmentation in CT imaging
Roth, H. R., Farag, A., Lu, L., Turkbey, E. B., & Summers, R. M. (2015) · SPIE Proceedings · doi:10.1117/12.2081420 · Cited by 173
Eitel, F., Soehler, E., Bellmann-Strobl, J., Brandt, A. U., Ruprecht, K., Giess, R. M., Kuchling, J., Asseyer, S., Weygandt, M., Haynes, J.-D., et al. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.102003 · Cited by 171 · cited 24×
“…M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript Figure 7: Layerwise relevance propagation for model interpretation in (A) diagnosing multiple sclerosis on brain MRI (Eitel et al., 2019), and (B) visualizing evidence for Alzheimer’s disease on brain MRI (Böhle et al., 2019b).”
MethodInfluentialAttention Dense-U-Net for Automatic Breast Mass Segmentation in Digital Mammogram
Li, S., Dong, M., Du, G., & Mu, X. (2019) · IEEE Access · doi:10.1109/access.2019.2914873 · Cited by 171
CT-GAN: Malicious Tampering of 3D Medical Imagery using Deep Learning
Mirsky, Y., Mahler, T., Shelef, I., & Elovici, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.03597 · Cited by 168
Guided Soft Attention Network for Classification of Breast Cancer Histopathology Images
Yang, H., Kim, J.-Y., Kim, H., & Adhikari, S. P. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2948026 · Cited by 166
Breast Mass Classification from Mammograms using Deep Convolutional Neural Networks
Lévy, D., & Jain, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00542 · Cited by 166
Promoting Trust Between Patients and Physicians in the Era of Artificial Intelligence
Nundy, S., Montgomery, T., & Wachter, R. M. (2019) · JAMA · doi:10.1001/jama.2018.20563 · Cited by 161
Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images
Hou, L., Nguyen, V., Kanevsky, A. B., Samaras, D., Kurc, T. M., Zhao, T., Gupta, R. R., Gao, Y., Chen, W., Foran, D., et al. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2018.09.007 · Cited by 160
Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images
Gondal, W. M., Kohler, J. M., Grzeszick, R., Fink, G. A., & Hirsch, M. (2017) · 2017 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2017.8296646 · Cited by 155
Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules
Feng, X., Yang, J., Laine, A. F., & Angelini, E. D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_65 · Cited by 155
Unsupervised lesion detection via image restoration with a normative prior
Chen, X., You, S., Tezcan, K. C., & Konukoğlu, E. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101713 · Cited by 150
Towards Best Practice in Explaining Neural Network Decisions with LRP
Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206975 · Cited by 145
Spatial Aggregation of Holistically-Nested Networks for Automated Pancreas Segmentation
Roth, H. R., Lu, L., Farag, A., Sohn, A., & Summers, R. M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_52 · Cited by 141
Salient Deconvolutional Networks
Mahendran, A., & Vedaldi, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_8 · Cited by 140
Please Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138
Self-Transfer Learning for Weakly Supervised Lesion Localization
Hwang, S., & Kim, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_28 · Cited by 130
Chest Radiographs in Congestive Heart Failure: Visualizing Neural Network Learning
Seah, J. C. Y., Tang, J. S. N., Kitchen, A., Gaillard, F., & Dixon, A. F. (2019) · Radiology · doi:10.1148/radiol.2018180887 · Cited by 128
Sammut, C., & Webb, G. I. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1_884 · Cited by 126
Interpretable Image Recognition with Hierarchical Prototypes
Hase, P., Chen, C., Li, O., & Rudin, C. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5265 · Cited by 123
Clinical implementation of AI technologies will require interpretable AI models
Jia, X., Ren, L., & Cai, J. (2020) · Medical Physics · doi:10.1002/mp.13891 · Cited by 113
Yu, Z., Tan, E.-L., Ni, D., Qin, J., Chen, S., Li, S., Lei, B., & Wang, T. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2705031 · Cited by 109 · cited 4×
“, 2017), fetal facial plan recognition on ultrasound images (Yu et al., 2018), classification of skin lesions on dermatology photographs (Van Molle et al.”
BackgroundMethodResultInfluentialRayan, J. C., Reddy, N. E., Kan, J. H., Zhang, W., & Annapragada, A. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180015 · Cited by 109
Adversarial Attacks Against Medical Deep Learning Systems
Finlayson, S. G., Chung, H. W., Kohane, I. S., & Beam, A. L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05296 · Cited by 98
Melanoma Recognition via Visual Attention
Yan, Y., Kawahara, J., & Hamarneh, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20351-1_62 · Cited by 96
Analyzing Neuroimaging Data Through Recurrent Deep Learning Models
Thomas, A. W., Heekeren, H. R., Müller, K.-R., & Samek, W. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01321 · Cited by 90
Jreige, M., Letovanec, I., Chaba, K., Renaud, S., Rusakiewicz, S., Cristina, V., Peters, S., Krueger, T., de Leval, L., Kandalaft, L. E., et al. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04348-x · Cited by 81
Relationship between the expression of PD-1/PD-L1 and 18F-FDG uptake in bladder cancer
Chen, R., Zhou, X., Liu, J., & Huang, G. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-018-4208-8 · Cited by 77
Kumar, D., Wong, A., & Taylor, G. W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.215 · Cited by 75
Visualizing Convolutional Neural Networks to Improve Decision Support for Skin Lesion Classification
Molle, P. V., Strooper, M. D., Verbelen, T., Vankeirsbilck, B., Simoens, P., & Dhoedt, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02628-8_13 · Cited by 73
SpineNet: Automatically Pinpointing Classification Evidence in Spinal MRIs
Jamaludin, A., Kadir, T., & Zisserman, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_20 · Cited by 70
Faust, K., Xie, Q., Han, D., Goyle, K., Volynskaya, Z., Djuric, U., & Diamandis, P. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2184-4 · Cited by 70
Yang, X., Yang, J. D., Hwang, H. P., Yu, H. C., Ahn, S. W., Kim, B., & You, H. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2017.12.008 · Cited by 69
3D regression neural network for the quantification of enlarged perivascular spaces in brain MRI
Dubost, F., Adams, H. H. H., Bortsova, G., Ikram, M. A., Niessen, W. J., Vernooij, M. W., & de Bruijne, M. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.10.008 · Cited by 69
Assessing Knee OA Severity with CNN attention-based end-to-end architectures
Blanch, M. G., Antony, J., McGuinness, K., Giró-i-Nieto, X., & O'Connor, N. E. (2019) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · doi:10.48550/arxiv.1908.08856 · Cited by 64
Shen, Y., & Gao, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00919-9_45 · Cited by 60
Optimizing and Visualizing Deep Learning for Benign/Malignant Classification in Breast TumorsPreprint
Yi, D., Sawyer, R. L., Cohn, D. M., Dunnmon, J., Lam, C., Xiao, X., & Rubin, D. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06362 · Cited by 58
Respond-CAM: Analyzing Deep Models for 3D Imaging Data by Visualizations
Zhao, G., Zhou, B., Wang, K., Jiang, R., & Xu, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_55 · Cited by 55
Is it time to get rid of black boxes and cultivate trust in AI?
Gastounioti, A., & Kontos, D. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200088 · Cited by 47
Takada, K., Toyokawa, G., Tagawa, T., Kohashi, K., Akamine, T., Takamori, S., Hirai, F., Shoji, F., Okamoto, T., Oda, Y., et al. (2017) · Anticancer Research · doi:10.21873/anticanres.12180 · Cited by 47
Discovery Radiomics With CLEAR-DR: Interpretable Computer Aided Diagnosis of Diabetic Retinopathy
Kumar, D., Taylor, G. W., & Wong, A. M. (2019) · IEEE Access · doi:10.1109/access.2019.2893635 · Cited by 46
GP-Unet: Lesion Detection from Weak Labels with a 3D Regression Network
Dubost, F., Bortsova, G., Adams, H., Ikram, A., Niessen, W. J., Vernooij, M., & De Bruijne, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_25 · Cited by 42
Uzunova, H., Ehrhardt, J., Kepp, T., & Handels, H. (2019) · Medical Imaging 2019: Image Processing · doi:10.1117/12.2511964 · Cited by 39
Gao, Y., & Noble, J. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66185-8_35 · Cited by 37
Hicks, S. A., Riegler, M. A., Pogorelov, K., Ånonsen, K. V., de Lange, T., Johansen, D., Jeppsson, M., Randel, K. R., Eskeland, S. L., & Halvorsen, P. (2018) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2018.00070 · Cited by 36
Chen, S.-W., Shen, W.-C., Lin, Y., Chen, R.-Y., Hsieh, T., Yen, K., & Kao, C. (2016) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-016-3580-5 · Cited by 33
DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation
Wu, J., Zhou, B., Peck, D., Hsieh, S. S., Dialani, V. M., Mackey, L. W., & Patterson, G. (2021) · Harvard Data Science Review · doi:10.1162/99608f92.8b81b005 · Cited by 30
Visually interpretable deep network for diagnosis of breast masses on mammograms
Kim, S. T., Lee, J.-H., Lee, H., & Ro, Y. M. (2018) · Physics in Medicine & Biology · doi:10.1088/1361-6560/aaef0a · Cited by 29
Garg, P., Davenport, E. M., Murugesan, G. K., Wagner, B. C., Whitlow, C. T., Maldjian, J. A., & Montillo, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_43 · Cited by 24
Interpretable Image Recognition with Hierarchical PrototypesPreprint
Hase, P., Chen, C., Li, O., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10651 · Cited by 24
Hicks, S. A., Eskeland, S., Lux, M., de Lange, T., Randel, K. R., Jeppsson, M., Pogorelov, K., Halvorsen, P., & Riegler, M. (2018) · Proceedings of the 9th ACM Multimedia Systems Conference · doi:10.1145/3204949.3208129 · Cited by 21
SAUNet: Shape Attentive U-Net for Interpretable Medical Image SegmentationPreprint
Sun, J., Darbehani, F., Zaidi, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07645 · Cited by 21
Expert identification of visual primitives used by CNNs during mammogram classification
Hsieh, S. S., Lehman, C. D., Dialani, V., Zhou, B., Peck, D., Patterson, G., Mackey, L. W., Syrgkanis, V., & Wu, J. (2018) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2293890 · Cited by 20
Learning Curves in Machine Learning
Perlich, C. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1_452 · Cited by 5
Deep Mask For X-ray Based Heart Disease ClassificationPreprint
Chen, X., & Shi, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08277 · Cited by 4
Encoding High-Level Visual Attributes in Capsules for Explainable Medical Diagnoses.Preprint
LaLonde, R., Torigian, D. A., & Bağcı, U. (2019) · arXiv (Cornell University) · Cited by 4
Visualizing evidence for Alzheimer's disease in deep neural networks trained on structural MRI data
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Cited by 3
Research works printed but not linked: 6
- 1. Adebayo J, Gilmer J, Muelly M, Goodfellow I, Hardt Mand Kim B Advances in Neural Information Processing Systems,2018), vol. Series) pp 9505–15
- 25. Glorot X and Bengio Y Proceedings of the thirteenth international conference on artificial intelligence and statistics,2010), vol. Series) pp 249–56
- 26. Glorot X, Bordes A and Bengio Y Proceedings of the fourteenth international conference on artificial intelligence and statistics,2011), vol. Series) pp 315–23
- 52. Krizhevsky A, Sutskever I and Hinton GE Advances in neural information processing systems,2012), vol. Series) pp 1097–105
- 81. Sabour S, Frosst N and Hinton GE Advances in neural information processing systems,2017), vol. Series) pp 3856–66
- 108. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R and Bengio Y International conference on machine learning,2015), vol. Series) pp 2048–57
Cited by
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
ŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
Bhati, D., Neha, F., & Amiruzzaman, M. (2024) · Journal of Imaging · doi:10.3390/jimaging10100239 · Cited by 213
Iqbal, S., N. Qureshi, A., Li, J., & Mahmood, T. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09899-9 · Cited by 210
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2025) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 184
Metadata
- Authors
- Huff, D. T., Weisman, A. J., & Jeraj, R.
- Affiliations
- University of Wisconsin–Madison (US)
- Corresponding author
- Jeraj, Robert
- Venue
- Physics in Medicine & Biology
- Publisher
- IOP Publishing
- Published
- 2021-02-01
- Volume
- 66
- Issue
- 4
- Pages
- 04TR01
- ISSN
- 0031-9155 (ISSN-L)
- 1361-6560
- PMC
- PMC8236074
- OpenAlex
- W3108932871
- Cited by
- 172
- References
- 120
- Citations by year
- 2026: 29 · 2025: 40 · 2024: 50 · 2023: 24 · 2022: 21 · 2021: 8
- FWCI
- 0.46
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- publishingsupport.iopscience.iop.org/iop-standard/v1 (vor)
- iopscience.iop.org/info/page/text-and-data-mining (tdm)
- Locations
- Physics in Medicine and Biology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc-nd)
- Funders
- National Institutes of Health (T32CA009206)
- University of Wisconsin Carbone Cancer Center (P30CA014520)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- deep learning interpretability, medical image analysis, model visualization, saliency maps, class activation maps, autoencoder, dimensionality reduction, feature visualization, attribution methods, explainable AI, clinical translation
- MeSH terms
- Deep Learning, Diagnostic Imaging, Humans, Image Processing, Computer-Assisted
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.