Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Open accessAbstract
This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work on effective receptive fields and salient regions reveals that only a small proportion of pixels in the theoretical receptive field contribute significantly to the final network prediction.
“Recent works on effective receptive fields [31] and salient regions [41, 44, 13, 7] reveal that only a small proportion of pixels in the theoretical receptive field contribute significantly to the final network prediction.”
From Deformable ConvNets V2: More Deformable, Better Results (2019) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deformable ConvNets V2: More Deformable, Better Results
Zhu, X., Hu, H., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00953 · Cited by 2,943 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · see citing works
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · see citing works
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328 · see citing works
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · see citing works
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Machine Learning Methods for Histopathological Image Analysis
Komura, D., & Ishikawa, S. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.01.001 · Cited by 937 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Data Augmentation Using Random Image Cropping and Patching for Deep CNNs
Takahashi, R., Matsubara, T., & Uehara, K. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2935128 · Cited by 477 · see citing works
Interpretability of machine learning-based prediction models in healthcare
Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., & Cilar, L. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1379 · Cited by 541 · see citing works
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · see citing works
Explainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372 · see citing works
Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226 · see citing works
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171 · see citing works
Transparency: Motivations and Challenges
Weller, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_2 · Cited by 198 · see citing works
Eitel, F., Soehler, E., Bellmann-Strobl, J., Brandt, A. U., Ruprecht, K., Giess, R. M., Kuchling, J., Asseyer, S., Weygandt, M., Haynes, J.-D., et al. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.102003 · Cited by 171 · see citing works
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171 · see citing works
A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137 · see citing works
Four Principles of Explainable Artificial Intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Broniatowski, D. A., & Przybocki, M. A. (2020) · doi:10.6028/nist.ir.8312-draft · Cited by 136 · see citing works
DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
Han, D., Wang, Z., Chen, W., Zhong, Y., Wang, S., Zhang, H., Yang, J., Shi, X., & Yin, X. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484589 · Cited by 121 · see citing works
TSViz: Demystification of Deep Learning Models for Time-Series Analysis
Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. (2019) · IEEE Access · doi:10.1109/access.2019.2912823 · Cited by 95 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Contrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83 · see citing works
Predicting phenotypes from genetic, environment, management, and historical data using CNNs
Washburn, J. D., Cimen, E., Ramstein, G., Reeves, T., O’Briant, P., McLean, G., Cooper, M., Hammer, G., & Buckler, E. S. (2021) · Theoretical and Applied Genetics · doi:10.1007/s00122-021-03943-7 · Cited by 57 · see citing works
Kumar, D., Sankar, V., Clausi, D., Taylor, G. W., & Wong, A. (2019) · IEEE Access · doi:10.1109/access.2019.2945524 · Cited by 26 · see citing works
- Authors
- Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2622-8912 (ISSN-L)
- arXiv
- 1702.04595
- OpenAlex
- W2590082389
- Cited by
- 487
- References
- 2
- Citations by year
- 2025: 15 · 2024: 31 · 2023: 52 · 2022: 64 · 2021: 90 · 2020: 98 · 2019: 68 · 2018: 47 · 2017: 20 · 2016: 1
- FWCI
- 29.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- deep neural network interpretability, image classification, black-box classifiers, medical image classification, brain MRI, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
