CoChat
Sign Up
Cited by 923Explainable Artificial Intelligence (XAI)Software Testing and Debugging TechniquesAdversarial Robustness in Machine Learning

A Survey on Neural Network Interpretability

Open access
Zhang, Yu, Tino, Peter, Leonardis, Ales, Tang, Ke·2021·IEEE Transactions on Emerging Topics in Computational Intelligence
DOI 10.1109/tetci.2021.3100641
Open this paper in CoChat — free account

Abstract

Along with the great success of deep neural networks, there is also growing concern about their black-box nature. The interpretability issue affects people's trust on deep learning systems. It is also related to many ethical problems, e.g., algorithmic discrimination. Moreover, interpretability is a desired property for deep networks to become powerful tools in other research fields, e.g., drug discovery and genomics. In this survey, we conduct a comprehensive review of the neural network interpretability research. We first clarify the definition of interpretability as it has been used in many different contexts. Then we elaborate on the importance of interpretability and propose a novel taxonomy organized along three dimensions: type of engagement (passive vs. active interpretation approaches), the type of explanation, and the focus (from local to global interpretability). This taxonomy provides a meaningful 3D view of distribution of papers from the relevant literature as two of the dimensions are not simply categorical but allow ordinal subcategories. Finally, we summarize the existing interpretability evaluation methods and suggest possible research directions inspired by our new taxonomy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tetci.2021.3100641

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Machine learning systems are hard to interpret.

    “and are hard to interpret [12]”

    From Warning: Humans cannot reliably detect speech deepfakes (2023) · cites this paper as [12]

    The survey describes deep neural networks as black boxes and focuses on explaining their inner workings (Definition of Interpretability, pp. 1–2). That supports the claim for neural-network systems, but not for machine-learning systems generally: the same section says some decision trees and linear models are interpretable (p. 2). (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 125

  1. [4][4] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury et al. , 'Deep neural networks for acoustic modeling in speech recognition,' IEEE Signal processing magazine , vol. 29, 2012. -
  2. [5][5] I. Sutskever, O. Vinyals, and Q. V. Le, 'Sequence to sequence learning with neural networks,' in Advances in neural information processing systems , 2014. -
  3. [6][6] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al. , 'Mastering the game of go without human knowledge,' Nature , vol. 550, 2017. -
  4. [7][7] S. Dong, P. Wang, and K. Abbas, 'A survey on deep learning and its applications,' Computer Science Review , 2021. -
  5. [8][8] P. Dixit and S. Silakari, 'Deep learning algorithms for cybersecurity applications: A technological and status review,' Computer Science Review , 2021. -
  6. [9][9] T. Bouwmans, S. Javed, M. Sultana, and S. K. Jung, 'Deep neural network concepts for background subtraction: A systematic review and comparative evaluation,' Neural Networks , 2019. -
  7. [11][11] M. Wainberg, D. Merico, A. Delong, and B. J. Frey, 'Deep learning in biomedicine,' Nature biotechnology , vol. 36, 2018. -
  8. [12][12] H. Y. Xiong, B. Alipanahi, L. J. Lee, H. Bretschneider, D. Merico, R. K. Yuen, Y. Hua, S. Gueroussov, H. S. Najafabadi, T. R. Hughes et al. , 'The human splicing code reveals new insights into the genetic determinants of disease,' Science , vol. 347, 2015. -
  9. [13][13] S. Zhang, H. Hu, T. Jiang, L. Zhang, and J. Zeng, 'Titer: predicting translation initiation sites by deep learning,' Bioinformatics , vol. 33, 2017. -
  10. [14][14] D. Parks, J. X. Prochaska, S. Dong, and Z. Cai, 'Deep learning of quasar spectra to discover and characterize damped ly α systems,' Monthly Notices of the Royal Astronomical Society , vol. 476, 2018. -
  11. [15][15] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, 'Intriguing properties of neural networks,' arXiv preprint arXiv:1312.6199 , 2013. - · arXiv
  12. [16][16] A. Nguyen, J. Yosinski, and J. Clune, 'Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015. -
  13. [19][19] D. Pedreschi, F. Giannotti, R. Guidotti, A. Monreale, S. Ruggieri, and F. Turini, 'Meaningful explanations of black box ai decision systems,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019. -
  14. [20][20] M. T. Ribeiro, S. Singh, and C. Guestrin, 'Why should i trust you?: Explaining the predictions of any classifier,' in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , 2016. -
  15. [21][21] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, 'Network dissection: Quantifying interpretability of deep visual representations,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017. -
  16. [22][22] G. D. Stormo, T. D. Schneider, L. Gold, and A. Ehrenfeucht, 'Use of the 'perceptron' algorithm to distinguish translational initiation sites in e. coli,' Nucleic acids research , vol. 10, 1982. -
  17. [23][23] A. B. Arrieta, N. D´ ıaz-Rodr´ ıguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garc´ ıa, S. Gil-L´ opez, D. Molina, R. Benjamins et al. , 'Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,' Information Fusion , vol. 58, 2020. -
  18. [24][24] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba, 'Gan dissection: Visualizing and understanding generative adversarial networks,' in Proceedings of the International Conference on Learning Representations (ICLR) , 2019. -
  19. [25][25] C. Yang, Y. Shen, and B. Zhou, 'Semantic hierarchy emerges in deep generative representations for scene synthesis,' International Journal of Computer Vision , 2021. -
  20. [26][26] A. Voynov and A. Babenko, 'Unsupervised discovery of interpretable directions in the gan latent space,' in International Conference on Machine Learning , 2020. -
  21. [27][27] A. Plumerault, H. L. Borgne, and C. Hudelot, 'Controlling generative models with continuous factors of variations,' in International Conference on Learning Representations , 2020. -
  22. [28][28] E. H¨ ark¨ onen, A. Hertzmann, J. Lehtinen, and S. Paris, 'Ganspace: Discovering interpretable gan controls,' in Advances in Neural Information Processing Systems , 2020. -
  23. [29][29] M. Kahng, N. Thorat, D. H. Chau, F. B. Vi´ egas, and M. Wattenberg, 'Gan lab: Understanding complex deep generative models using interactive visual experimentation,' IEEE transactions on visualization and computer graphics , 2018. -
  24. [30][30] R. Vidal, J. Bruna, R. Giryes, and S. Soatto, 'Mathematics of deep learning,' arXiv preprint arXiv:1712.04741 , 2017. - · arXiv
  25. [31][31] J. Bruna and S. Mallat, 'Invariant scattering convolution networks,' IEEE transactions on pattern analysis and machine intelligence , 2013. -
  26. [32][32] R. Eldan and O. Shamir, 'The power of depth for feedforward neural networks,' in Conference on learning theory , 2016. -
  27. [33][33] M. Nouiehed and M. Razaviyayn, 'Learning deep models: Critical points and local openness,' arXiv preprint arXiv:1803.02968 , 2018. - · arXiv
  28. [34][34] C. Yun, S. Sra, and A. Jadbabaie, 'Global optimality conditions for deep neural networks,' in International Conference on Learning Representations , 2018. -
  29. [36][36] B. D. Haeffele and R. Vidal, 'Global optimality in neural network training,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017. -
  30. [37][37] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: a simple way to prevent neural networks from overfitting,' The journal of machine learning research , 2014. -
  31. [38][38] P. Mianjy, R. Arora, and R. Vidal, 'On the implicit bias of dropout,' in International Conference on Machine Learning , 2018. -
  32. [39][39] H. Salehinejad and S. Valaee, 'Ising-dropout: A regularization method for training and compression of deep neural networks,' in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019. -
  33. [40][40] B. Sengupta and K. J. Friston, 'How robust are deep neural networks?' arXiv preprint arXiv:1804.11313 , 2018. - · arXiv
  34. [41][41] S. Zheng, Y. Song, T. Leung, and I. Goodfellow, 'Improving the robustness of deep neural networks via stability training,' in Proceedings of the ieee conference on computer vision and pattern recognition , 2016. -
  35. [43][43] B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham, 'Reversible architectures for arbitrarily deep residual neural networks,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2018. -
  36. [44][44] K. K. Thekumparampil, A. Khetan, Z. Lin, and S. Oh, 'Robustness of conditional gans to noisy labels,' in Advances in Neural Information Processing Systems , 2018. -
  37. [45][45] A. Creswell and A. A. Bharath, 'Denoising adversarial autoencoders,' IEEE transactions on neural networks and learning systems , 2018. -
  38. [47][47] K. R. Mopuri, U. Ojha, U. Garg, and R. V. Babu, 'Nag: Network for adversary generation,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  39. [48][48] Z. Zheng and P. Hong, 'Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks,' in Proceedings of the 32nd International Conference on Neural Information Processing Systems , 2018. -
  40. [49][49] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, 'A survey of methods for explaining black box models,' ACM computing surveys (CSUR) , vol. 51, 2018. -
  41. [51][51] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, 'Universal adversarial perturbations,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017. -
  42. [52][52] E. Gawehn, J. A. Hiss, and G. Schneider, 'Deep learning in drug discovery,' Molecular Informatics , vol. 35, 2016. -
  43. [54][54] European Parliament, Council of the European Union, 'Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation),' Official Journal of the European Union, Apr. 2016, https://eurlex.europa.eu/eli/reg/2016/679/oj. -
  44. [55][55] Y. Park and M. Kellis, 'Deep learning for regulatory genomics,' Nature biotechnology , vol. 33, 2015. -
  45. [56][56] P. Baldi, P. Sadowski, and D. Whiteson, 'Searching for exotic particles in high-energy physics with deep learning,' Nature communications , vol. 5, 2014. -
  46. [58][58] G. Montavon, W. Samek, and K.-R. M¨ uller, 'Methods for interpreting and understanding deep neural networks,' Digital Signal Processing , vol. 73, 2018. -
  47. [59][59] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, 'Explaining explanations: An overview of interpretability of machine learning,' in 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) , 2018. -
  48. [60][60] Q.-s. Zhang and S.-c. Zhu, 'Visual interpretability for deep learning: a survey,' Frontiers of Information Technology & Electronic Engineering , vol. 19, 2018. -
  49. [61][61] W. Samek and K.-R. M¨ uller, 'Towards explainable artificial intelligence,' in Explainable AI: interpreting, explaining and visualizing deep learning . Springer, 2019. -
  50. [62][62] F. Bodria, F. Giannotti, R. Guidotti, F. Naretto, D. Pedreschi, and S. Rinzivillo, 'Benchmarking and survey of explanation methods for black box models,' arXiv preprint arXiv:2102.13076 , 2021. - · arXiv
  51. [63][63] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres, 'Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV),' in International Conference on Machine Learning , 2018. -
  52. [64][64] C. Molnar, G. Casalicchio, and B. Bischl, 'Interpretable machine learning-a brief history, state-of-the-art and challenges,' in ECML PKDD 2020 Workshops , 2020. -
  53. [66][66] P. Adler, C. Falk, S. A. Friedler, T. Nix, G. Rybeck, C. Scheidegger, B. Smith, and S. Venkatasubramanian, 'Auditing black-box models for indirect influence,' Knowledge and Information Systems , vol. 54, 2018. -
  54. [67][67] S. Lapuschkin, S. W¨ aldchen, A. Binder, G. Montavon, W. Samek, and K.-R. M¨ uller, 'Unmasking clever hans predictors and assessing what machines really learn,' Nature communications , vol. 10, 2019. -
  55. [68][68] K. Natesan Ramamurthy, B. Vinzamuri, Y. Zhang, and A. Dhurandhar, 'Model agnostic multilevel explanations,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  56. [69][69] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, and P. Das, 'Explanations based on the missing: Towards contrastive explanations with pertinent negatives,' in Advances in Neural Information Processing Systems 31 . Curran Associates, Inc., 2018. -
  57. [70][70] Q. Zhang, Y. Nian Wu, and S.-C. Zhu, 'Interpretable convolutional neural networks,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  58. [72][72] M. D. Zeiler and R. Fergus, 'Visualizing and understanding convolutional networks,' in European conference on computer vision , 2014. -
  59. [73][73] P. W. Koh and P. Liang, 'Understanding black-box predictions via influence functions,' in Proceedings of the 34th International Conference on Machine Learning-Volume 70 , 2017. -
  60. [74][74] B. Zhou, A. Khosla, ` A. Lapedriza, A. Oliva, and A. Torralba, 'Object detectors emerge in deep scene cnns,' in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. -
  61. [75][75] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, 'Sanity checks for saliency maps,' in Advances in Neural Information Processing Systems 31 , 2018. -
  62. [76][76] O. Li, H. Liu, C. Chen, and C. Rudin, 'Deep learning for casebased reasoning through prototypes: A neural network that explains its predictions,' in Thirty-Second AAAI Conference on Artificial Intelligence , 2018. -
  63. [77][77] Y. Wang, H. Su, B. Zhang, and X. Hu, 'Interpret neural networks by identifying critical data routing paths,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. -
  64. [78][78] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, 'Counterfactual visual explanations,' in Proceedings of the 36th International Conference on Machine Learning , vol. 97, 2019. -
  65. [79][79] K. Kanamori, T. Takagi, K. Kobayashi, and H. Arimura, 'Dace: Distribution-aware counterfactual explanation by mixed-integer linear optimization,' in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization , 2020. -
  66. [80][80] T. Wang, 'Gaining free or low-cost interpretability with interpretable partial substitute,' in International Conference on Machine Learning , 2019. -
  67. [81][81] L. Fu, 'Rule Learning by Searching on Adapted Nets,' AAAI , 1991. -
  68. [82][82] G. G. Towell and J. W. Shavlik, 'Extracting refined rules from knowledge-based neural networks,' Machine Learning , vol. 13, 1993. -
  69. [83][83] R. Setiono and H. Liu, 'Understanding Neural Networks via Rule Extraction,' IJCAI , 1995. -
  70. [84][84] --, 'NeuroLinear: From neural networks to oblique decision rules,' Neurocomputing , vol. 17, 1997. -
  71. [85][85] K. Odajima, Y. Hayashi, G. Tianxia, and R. Setiono, 'Greedy rule generation from discrete data and its use in neural network rule extraction,' Neural Networks , vol. 21, 2008. -
  72. [86][86] R. Nayak, 'Generating rules with predicates, terms and variables from the pruned neural networks,' Neural Networks , vol. 22, 2009. -
  73. [87][87] J. M. Benitez, J. L. Castro, and I. Requena, 'Are artificial neural networks black boxes?' IEEE Transactions on Neural Networks , vol. 8, 1997. -
  74. [88][88] J. L. Castro, C. J. Mantas, and J. M. Benitez, 'Interpretation of artificial neural networks by means of fuzzy rules,' IEEE Transactions on Neural Networks , vol. 13, 2002. -
  75. [89][89] M. Craven and J. W. Shavlik, 'Extracting Tree-Structured Representations of Trained Networks,' Advances in Neural Information Processing Systems , 1996. -
  76. [90][90] R. Krishnan, G. Sivakumar, and P. Bhattacharya, 'Extracting decision trees from trained neural networks,' Pattern recognition , vol. 32, 1999. -
  77. [91][91] O. Boz, 'Extracting decision trees from trained neural networks,' in Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , 2002. -
  78. [92][92] T. Pedapati, A. Balakrishnan, K. Shanmugam, and A. Dhurandhar, 'Learning global transparent models consistent with local contrastive explanations,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  79. [93][93] D. Erhan, Y. Bengio, A. Courville, and P. Vincent, 'Visualizing higherlayer features of a deep network,' University of Montreal , vol. 1341, 2009. -
  80. [94][94] F. Wang, H. Liu, and J. Cheng, 'Visualizing deep neural network by alternately image blurring and deblurring,' Neural Networks , vol. 97, 2018. -
  81. [95][95] A. Mahendran and A. Vedaldi, 'Understanding deep image representations by inverting them,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 5188-5196. -
  82. [96][96] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, 'Understanding Neural Networks Through Deep Visualization,' ICML Deep Learning Workshop , Jun. 2015. -
  83. [97][97] A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune, 'Synthesizing the preferred inputs for neurons in neural networks via deep generator networks,' NIPS , 2016. -
  84. [99][99] R. Fong and A. Vedaldi, 'Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks,' in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. -
  85. [100][100] F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, D. A. Bau, and J. Glass, 'What is one grain of sand in the desert? analyzing individual neurons in deep nlp models,' in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2019. -
  86. [102][102] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, 'Striving for simplicity: The all convolutional net,' arXiv preprint arXiv:1412.6806 , 2014. - · arXiv
  87. [103][103] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, 'Grad-cam: Visual explanations from deep networks via gradient-based localization,' in Proceedings of the IEEE International Conference on Computer Vision , 2017. -
  88. [104][104] E. ˇ Strumbelj and I. Kononenko, 'An efficient explanation of individual classifications using game theory,' Journal of Machine Learning Research , vol. 11, 2010. -
  89. [106][106] M. Ancona, C. Oztireli, and M. Gross, 'Explaining deep neural networks with a polynomial time algorithm for shapley value approximation,' in Proceedings of the 36th International Conference on Machine Learning , vol. 97, 2019. -
  90. [107][107] T. Heskes, E. Sijben, I. G. Bucur, and T. Claassen, 'Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  91. [108][108] R. C. Fong and A. Vedaldi, 'Interpretable explanations of black boxes by meaningful perturbation,' in Proceedings of the IEEE International Conference on Computer Vision , 2017. -
  92. [109][109] L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling, 'Visualizing deep neural network decisions: Prediction difference analysis,' arXiv preprint arXiv:1702.04595 , 2017. - · arXiv
  93. [110][110] J. Chen, L. Song, M. Wainwright, and M. Jordan, 'Learning to explain: An information-theoretic perspective on model interpretation,' in Proceedings of the 35th International Conference on Machine Learning , vol. 80, 2018. -
  94. [111][111] S. Wang, T. Zhou, and J. Bilmes, 'Bias also matters: Bias attribution for deep neural network explanation,' in International Conference on Machine Learning , 2019. -
  95. [112][112] A. Shrikumar, P. Greenside, and A. Kundaje, 'Learning important features through propagating activation differences,' in Proceedings of the 34th International Conference on Machine Learning , 2017. -
  96. [113][113] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. M¨ uller, and W. Samek, 'On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,' PloS one , vol. 10, 2015. -
  97. [114][114] M. Sundararajan, A. Taly, and Q. Yan, 'Axiomatic attribution for deep networks,' in Proceedings of the 34th International Conference on Machine Learning-Volume 70 , 2017. -
  98. [115][115] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim, 'Towards automatic concept-based explanations,' in Advances in Neural Information Processing Systems , 2019, pp. 9277-9286. -
  99. [116][116] S. Salman, S. N. Payrovnaziri, X. Liu, P. Rengifo-Moreno, and Z. He, 'Deepconsensus: Consensus-based interpretable deep neural networks with application to mortality prediction,' in 2020 International Joint Conference on Neural Networks (IJCNN) , 2020. -
  100. [117][117] C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, 'Representer point selection for explaining deep neural networks,' in Advances in Neural Information Processing Systems 31 , 2018. -
  101. [118][118] M. Wu, S. Parbhoo, M. C. Hughes, R. Kindle, L. A. Celi, M. Zazzi, V. Roth, and F. Doshi-Velez, 'Regional tree regularization for interpretability in deep neural networks.' in AAAI , 2020, pp. 6413-6421. -
  102. [120][120] G. Plumb, M. Al-Shedivat, ´ A. A. Cabrera, A. Perer, E. Xing, and A. Talwalkar, 'Regularizing black-box models for improved interpretability,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  103. [121][121] E. Weinberger, J. Janizek, and S.-I. Lee, 'Learning deep attribution priors based on prior knowledge,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  104. [122][122] M. Wojtas and K. Chen, 'Feature importance ranking for deep learning,' Advances in Neural Information Processing Systems , vol. 33, 2020. -
  105. [123][123] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su, 'This looks like that: deep learning for interpretable image recognition,' in Advances in Neural Information Processing Systems , 2019. -
  106. [124][124] S. Wachter, B. Mittelstadt, and C. Russell, 'Counterfactual explanations without opening the black box: Automated decisions and the gdpr,' Harvard Journal of Law & Technology , 2018. -
  107. [125][125] M. W. Craven and J. W. Shavlik, 'Using sampling and queries to extract rules from trained neural networks,' in Machine learning proceedings 1994 , 1994. -
  108. [126][126] R. Andrews, J. Diederich, and A. B. Tickle, 'Survey and critique of techniques for extracting rules from trained artificial neural networks,' Knowledge-Based Systems , vol. 8, 1995. -
  109. [127][127] A. B. Tickle, R. Andrews, M. Golea, and J. Diederich, 'The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks,' IEEE Transactions on Neural Networks , vol. 9, 1998. -
  110. [129][129] J. R. Quinlan, 'C4.5: Programs for machine learning,' The Morgan Kaufmann Series in Machine Learning, San Mateo, CA: Morgan Kaufmann,- c1993 , 1993. -
  111. [130][130] G. D. Plotkin, 'A note on inductive generalization,' Machine intelligence , vol. 5, 1970. -
  112. [131][131] S. Mitra and Y. Hayashi, 'Neuro-fuzzy rule generation: survey in soft computing framework,' IEEE Transactions on Neural Networks , vol. 11, 2000. -
  113. [132][132] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in Advances in neural information processing systems , 2014. -
  114. [133][133] A. Radford, L. Metz, and S. Chintala, 'Unsupervised representation learning with deep convolutional generative adversarial networks,' ICLR , 2016. -
  115. [135][135] T. Minematsu, A. Shimada, and R.-i. Taniguchi, 'Analytics of deep neural network in change detection,' in 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) , 2017. -
  116. [136][136] T. Minematsu, A. Shimada, H. Uchiyama, and R.-i. Taniguchi, 'Analytics of deep neural network-based background subtraction,' Journal of Imaging , 2018. -
  117. [137][137] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. M ˜ Aˇ zller, 'How to explain individual classification decisions,' Journal of Machine Learning Research , vol. 11, 2010. -
  118. [138][138] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, 'Learning deep features for discriminative localization,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016. -
  119. [139][139] M. Ancona, E. Ceolini, C. ¨ Oztireli, and M. Gross, 'Towards better understanding of gradient-based attribution methods for deep neural networks,' in International Conference on Learning Representations , 2018. -
  120. [142][142] J. Heo, S. Joo, and T. Moon, 'Fooling neural network interpretations via adversarial model manipulation,' in Advances in Neural Information Processing Systems , 2019. -
  121. [144][144] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, 'Distributed representations of words and phrases and their compositionality,' in Advances in neural information processing systems , 2013. -
  122. [145][145] L. S. Shapley, 'A value for n-person games,' Contributions to the Theory of Games , vol. 2, 1953. -
  123. [146][146] R. Caruana, H. Kangarloo, J. Dionisio, U. Sinha, and D. Johnson, 'Casebased explanation of non-case-based learning methods.' in Proceedings of the AMIA Symposium , 1999, p. 212. -
  124. [147][147] J. Bien, R. Tibshirani et al. , 'Prototype selection for interpretable classification,' The Annals of Applied Statistics , vol. 5, 2011. -
  125. [148][148] B. Zhou, D. Bau, A. Oliva, and A. Torralba, 'Interpreting deep visual representations via network dissection,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018. -

Cited by

Sorted by citations

Authors
Zhang, Y., Tino, P., Leonardis, A., & Tang, K.
Affiliations
Southern University of Science and Technology (CN)
ORCID
Venue
IEEE Transactions on Emerging Topics in Computational Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-10
Volume
5
Issue
5
Pages
726-742
ISSN
2471-285X (ISSN-L)
DOI
10.1109/tetci.2021.3100641
OpenAlex
W3117379604
Cited by
923
References
21
Citations by year
2026: 102 · 2025: 270 · 2024: 243 · 2023: 166 · 2022: 110 · 2021: 32
FWCI
48.68
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Emerging Topics in Computational Intelligence (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Guangdong Provincial Key Laboratory (2020B121201001)
  • Program for Guangdong Introducing Innovative and Entrepreneurial Teams (2017ZT07X386)
  • Support Plan Program of Shenzhen Natural Science Fund (20200925154942002)
  • Science and Technology Commission of Shanghai Municipality (19511120602)
  • National Leading Youth Talent Support Program of China
  • MOE University Scientific-Technological Innovation Plan Program
  • European Commission Horizon 2020 Innovative Training Network SUNDIAL
  • Alan Turing Institute, ATI (1056900)
  • Machine Learning in the Space of Inferential Models
  • Survey Network for Deep Imaging Analysis, and Learning (721463)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
neural network interpretability, explainable AI, black-box models, explanation types, interpretability evaluation, algorithmic discrimination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review