Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Strong intelligent machines powered by deep neural networks are increasingly deployed as black boxes to make decisions in risk-sensitive domains, such as finance and medical. To reduce potential risk and build trust with users, it is critical to interpret how such machines make their decisions. Existing works interpret a pre-trained neural network by analyzing hidden neurons, mimicking pre-trained models or approximating local predictions. However, these methods do not provide a guarantee on the exactness and consistency of their interpretations. In this paper, we propose an elegant closed form solution named $OpenBox$ to compute exact and consistent interpretations for the family of Piecewise Linear Neural Networks (PLNN). The major idea is to first transform a PLNN into a mathematically equivalent set of linear classifiers, then interpret each linear classifier by the features that dominate its prediction. We further apply $OpenBox$ to demonstrate the effectiveness of non-negative and sparse constraints on improving the interpretability of PLNNs. The extensive experiments on both synthetic and real world data sets clearly demonstrate the exactness and consistency of our interpretation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Examples include rank order correlation [27,78,230,257], top-k intersection [78], cosine similarity [52], rule match [136], normalized distance [8], and the structural similarity index (SSIM) [60].”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [52]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
(2006) · Springer eBooks · Cited by 8,123
(2011) · Cited by 6,993
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“n Methods The local interpretation methods [11, 37, 39, 41] compute and visualize the important features for an input instance by analyzing the predictions of its local perturbations. Simonyan et al. [38] generated a class-representative image and a class-saliency map for each class of images by computing the gradient…”
MethodAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Efficient sparse coding algorithms
Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0105 · Cited by 2,394
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“ing [15]. As the impact of machine-made decisions increases, the demand on clear interpretations of machine learning systems is growing ever stronger against the blind deployments of decision machines[17].Accuratelyandreliablyinterpretingamachinelearning model is the key to many significant tasks, such as identifying…”
BackgroundUnderstanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 1×
“…image and Ribeiro et al. [36] proposed LIME to interpret the predictions of any classifier Zhou et al. [48] proposed CAM to identify discriminative image Selvaraju et al. [37] generalized CAM [48] by Grad-CAM, Koh et al. [23] used influence functions to trace a model’s prediction”
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 2×
“The local interpretation methods [11, 37, 39, 41] compute and visualize the important features for an input instance by analyzing the predictions of its local perturbations.”
MethodUnderstanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358
Do Deep Nets Really Need to be Deep?
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,335
Interpretation of Neural Networks is FragilePreprint
Ghorbani, A., Abid, A., & Zou, J. Y. (2017) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1710.10547 · Cited by 1,153
(2013) · Cited by 1,126
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Hoyer, P. O. (2003) · Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing · doi:10.1109/nnsp.2002.1030067 · Cited by 804
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Visualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625
Inverting Visual Representations with Convolutional Networks
Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.522 · Cited by 603
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_14 · Cited by 580
Prediction as a candidate for learning deep hierarchical models of data
Palm, R. (2012) · Cited by 444
Distilling a neural network into a soft decision tree
Frosst, N., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1711.09784 · Cited by 407 · cited 2×
“operties of each hidden neuron. However, qualitatively analyzing every neuron does not provide much actionable and quantitative interpretation about the overall mechanism of the entire neural network [12]. 2.2 Model Mimicking Methods By imitating the classification function of a neural network, the model mimicking met…”
MethodInfluentialCao, C., Liu, X., Yang, Y., Yu, Y., Wang, J., Wang, Z., Huang, Y., Wang, L., Huang, C., Xu, W., et al. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.338 · Cited by 392
Self-citationInterpreting Deep Visual Representations via Network Dissection
Zhou, B., Bau, D., Oliva, A., & Torralba, A. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858759 · Cited by 330
Analyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297 · cited 2×
“ing ever stronger against the blind deployments of decision machines[17].Accuratelyandreliablyinterpretingamachinelearning model is the key to many significant tasks, such as identifying failuremodels[1],buildingtrustwithhumanusers[35],discovering new knowledge [34], and avoiding unfairness issues [45]. The interpreta…”
BackgroundDo Deep Nets Really Need to be DeepPreprint
Ba, L. J., & Caruana, R. (2013) · arXiv (Cornell University) · Cited by 282
Visualizing and Understanding Neural Models in NLPPreprint
Li, J., Chen, X., Hovy, E. H., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01066 · Cited by 165
Learning Understandable Neural Networks With Nonnegative Weight Constraints
Chorowski, J., & Zurada, J. M. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2310059 · Cited by 161
The (Un)reliability of saliency methodsPreprint
Kindermans, P.-J., Hooker, S., Adebayo, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00867 · Cited by 160
On the number of response regions of deep feed forward networks with piece-wise linear activations
Pascanu, R., Montufar, G., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6098 · Cited by 153
VC dimension of neural networks
Sontag, E. D. (1998) · Cited by 126
Interpreting Blackbox Models via Model Extraction
Bastani, O., Kim, C., & Bastani, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08504 · Cited by 119
Beyond Sparsity: Tree Regularization of Deep Models for InterpretabilityPreprint
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Röth, V., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06178 · Cited by 106
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.03542 · Cited by 105
Nearly-tight VC-dimension bounds for piecewise linear neural networksPreprint
Harvey, N., Liaw, C., & Mehrabian, A. (2017) · arXiv (Cornell University) · Cited by 103
Neural Networks with Quadratic VC Dimension
Koiran, P., & Sontag, E. D. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1479 · Cited by 91
Caron, R. J., McDonald, J. F., & Ponic, C. M. (1989) · Journal of Optimization Theory and Applications · doi:10.1007/bf00941055 · Cited by 77
Interpreting Deep Visual Representations via Network DissectionPreprint
Zhou, B., Bau, D. A., Oliva, A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05611 · Cited by 45
Deep Embedding Forest: Forest-based Serving with Deep Embedding Features
Zhu, J., Shan, Y., Mao, J. C., Yu, D., Rahmanian, H., & Zhang, Y. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098059 · Cited by 40
Using Deep Learning Towards Biomedical Knowledge Discovery
Rather, N., Patel, C. O., & Khan, S. A. (2017) · International Journal of Mathematical Sciences and Computing · doi:10.5815/ijmsc.2017.02.01 · Cited by 20
Research works printed but not linked: 3
- Erhan D., 2009, Visualizing higher-layer features of a deep network, University of Montreal, 1341, 3
- Goodfellow Ian, Deep Learning
- Goodfellow Ian J, 2013, Maxout networks. arXiv:1302.4389 · arXiv
Cited by
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
On Interpretability of Artificial Neural Networks: A Survey
Fan, F.-L., Xiong, J., Li, M., & Wang, G. (2021) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2021.3066428 · Cited by 561
Metadata
- Authors
- Chu, L., Hu, X., Hu, J., Wang, L., & Pei, J.
- Affiliations
- Simon Fraser University (CA)
- Simon Fraser University, Burnaby, BC, Canada
- Huawei Technologies (China) (CN)
- Venue
- Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2018-07-19
- Pages
- 1244-1253
- OpenAlex
- W2962824341
- Cited by
- 87
- References
- 48
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 8 · 2023: 13 · 2022: 6 · 2021: 19 · 2020: 16 · 2019: 9 · 2018: 3
- FWCI
- 4.78
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- University of Washington Tacoma Digital Commons (University of Washington Tacoma) (submittedVersion)
- Funders
- the NSERC Strategic Grant program
- the NSERC Discovery Grant program
- the Canada Research Chair program
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- piecewise linear neural networks, neural network interpretability, nonnegativity constraints, sparsity constraints, explainable AI, black-box decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.