The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Higher-Order Explanations of Graph Neural Networks via Relevant Walks
Open accessAbstract
Graph Neural Networks (GNNs) are a popular approach for predicting graph structured data. As GNNs tightly entangle the input graph into the neural network structure, common explainable AI approaches are not applicable. To a large extent, GNNs have remained black-boxes for the user so far. In this paper, we show that GNNs can in fact be naturally explained using higher-order expansions, i.e., by identifying groups of edges that jointly contribute to the prediction. Practically, we find that such explanations can be extracted using a nested attribution scheme, where existing techniques such as layer-wise relevance propagation (LRP) can be applied at each step. The output is a collection of walks into the input graph that are relevant for the prediction. Our novel explanation method, which we denote by GNN-LRP, is applicable to a broad range of graph neural networks and lets us extract practically relevant insights on sentiment analysis of text data, structure-property relationships in quantum chemistry, and image classification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Statistical mechanics of complex networks
Albert, R., & Barabási, A.-L. (2002) · Reviews of Modern Physics · doi:10.1103/revmodphys.74.47 · Cited by 20,471
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000
Inductive Representation Learning on Large Graphs
Hamilton, W. L., Ying, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1706.02216 · Cited by 10,639
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582
Dynamic Graph CNN for Learning on Point Clouds
Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., & Solomon, J. M. (2019) · ACM Transactions on Graphics · doi:10.1145/3326362 · Cited by 7,199
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Self-citationJurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting
Yu, B., Yin, H., & Zhu, Z. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/505 · Cited by 3,728
Geometric Deep Learning: Going beyond Euclidean data
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2693418 · Cited by 3,661
Sentiment Analysis and Opinion Mining
Liu, B. (2012) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02145-9 · Cited by 3,281
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Self-citationSchNet – A deep learning architecture for molecules and materials
Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A., & Müller, K.-R. (2018) · The Journal of Chemical Physics · doi:10.1063/1.5019779 · Cited by 2,217
Self-citationQuantum chemistry structures and properties of 134 kilo molecules
Ramakrishnan, R., Dral, P. O., Rupp, M., & von Lilienfeld, O. A. (2014) · Scientific Data · doi:10.1038/sdata.2014.22 · Cited by 1,966
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Unke, O. T., Chmiela, S., Sauceda, H. E., Gastegger, M., Poltavsky, I., Schütt, K. T., Tkatchenko, A., & Müller, K.-R. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.0c01111 · Cited by 1,647
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615 · cited 1×
“In NLP model biases are well studied areas and some approaches to tackle that problem have already been developed [53], [54].”
BackgroundModeling polypharmacy side effects with graph convolutional networks
Zitnik, M., Agrawal, M., & Leskovec, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty294 · Cited by 1,501
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · cited 2×
“For a more comprehensive set of related works, we refer the reader to the review papers [17], [18] for XAI and [2], [19] for GNNs.”
BackgroundSelf-citationQuantum-chemical insights from deep tensor neural networks
Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., & Tkatchenko, A. (2017) · Nature Communications · doi:10.1038/ncomms13890 · Cited by 1,445
Self-citationExplainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · cited 6×
“In the following, we consider an extension of the Taylor expansion method, called deep Taylor decomposition (DTD) [41].”
BackgroundMethodInfluentialSelf-citationEvaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Self-citationUnmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Self-citationMachine Learning for Molecular Simulation
Noé, F., Tkatchenko, A., Müller, K.-R., & Clementi, C. (2020) · Annual Review of Physical Chemistry · doi:10.1146/annurev-physchem-042018-052331 · Cited by 956
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
Marcheggiani, D., & Titov, I. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1159 · Cited by 907
Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
Keith, J. A., Vassilev-Galindo, V., Cheng, B., Chmiela, S., Gastegger, M., Müller, K.-R., & Tkatchenko, A. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00107 · Cited by 849
Self-citationLayer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_10 · Cited by 750
Self-citationExplainability in Graph Neural Networks: A Taxonomic Survey
Yuan, H., Yu, H., Gui, S., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204236 · Cited by 583
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
Self-citationInceptionism: Going Deeper into Neural Networks
Mordvintsev, A., Olah, C., & Tyka, M. (2015) · Cited by 574
Explainability Methods for Graph Convolutional Neural Networks
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01103 · Cited by 553
Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 496
Gonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454
SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
Schütt, K. T., Kindermans, P.-J., Sauceda, H. E., Chmiela, S., Tkatchenko, A., & Müller, K. (2017) · Open Repository and Bibliography (University of Luxembourg) · doi:10.48550/arxiv.1706.08566 · Cited by 453
SchNetPack: A Deep Learning Toolbox For Atomistic Systems
Schütt, K. T., Kessel, P., Gastegger, M., Nicoli, K. A., Tkatchenko, A., & Müller, K.-R. (2019) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.8b00908 · Cited by 433
GraphLIME: Local Interpretable Model Explanations for Graph Neural Networks
Huang, Q., Yamada, M., Tian, Y., Singh, D., & Chang, Y. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3187455 · Cited by 424
Graph-to-Sequence Learning using Gated Graph Neural Networks
Beck, D., Haffari, G., & Cohn, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1026 · Cited by 380
Exploring chemical compound space with quantum-based machine learning
von Lilienfeld, O. A., Müller, K.-R., & Tkatchenko, A. (2020) · Nature Reviews Chemistry · doi:10.1038/s41570-020-0189-9 · Cited by 369
Parameterized Explainer for Graph Neural Network
Luo, D., Cheng, W., Xu, D., Yu, W., Zong, B., Chen, H., & Zhang, X. (2020) · arXiv · doi:10.48550/arxiv.2011.04573 · Cited by 355
XGNN: Towards Model-Level Explanations of Graph Neural Networks
Yuan, H., Tang, J., Hu, X., & Ji, S. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403085 · Cited by 272
Gonen, H., & Goldberg, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03862 · Cited by 225
Machine Learning Meets Quantum Physics
Schütt, K. T., Chmiela, S., von Lilienfeld, O. A., Tkatchenko, A., Tsuda, K., & Müller, K. (2020) · Lecture notes in physics · doi:10.1007/978-3-030-40245-7 · Cited by 218
Explanations can be manipulated and geometry is to blame
Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019) · arXiv · doi:10.48550/arxiv.1906.07983 · Cited by 171
Self-citationExplaining and Interpreting LSTMs
Arras, L., Arjona-Medina, J., Widrich, M., Montavon, G., Gillhofer, M., Müller, K.-R., Hochreiter, S., & Samek, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_11 · Cited by 135
Self-citationA Roadmap for a Rigorous Science of Interpretability.Preprint
Doshi‐Velez, F., & Kim, B. (2017) · arXiv (Cornell University) · Cited by 119
Hybrid Graph Convolutional Networks for Semi-Supervised Classification
(2019) · International Workshop on Computer Science and Engineering (WCSE) · doi:10.18178/wcse.2019.06.016 · Cited by 118
Layerwise Relevance Visualization in Convolutional Text Graph Classifiers
Schwarzenberg, R., Hübner, M., Harbecke, D., Alt, C., & Hennig, L. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5308 · Cited by 110
Evaluating Recurrent Neural Network Explanations
Arras, L., Osman, A., Müller, K., & Samek, W. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4813 · Cited by 105
Building and Interpreting Deep Similarity Models
Eberle, O., Büttner, J., Kräutli, F., Müller, K., Valleriani, M., & Montavon, G. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3020738 · Cited by 98
Self-citationGradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
Montavon, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_13 · Cited by 48
Self-citationQuantum-Chemical Insights from Interpretable Atomistic Neural Networks
Schütt, K. T., Gastegger, M., Tkatchenko, A., & Müller, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28954-6_17 · Cited by 47
SAM: The Sensitivity of Attribution Methods to Hyperparameters
Bansal, N., Agarwal, C., & Nguyen, A. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00009 · Cited by 27
Rieck, K., Krueger, T., Brefeld, U., & Müller, K. (2010) · Cited by 22
Learning Representations of Molecules and Materials with Atomistic Neural Networks
Schütt, K. T., Tkatchenko, A., & Müller, K. (2020) · Lecture notes in physics · doi:10.1007/978-3-030-40245-7_11 · Cited by 17
Perturb more, trap more: Understanding behaviors of graph neural networks
Ji, C., Wang, R., & Wu, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.070 · Cited by 15
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Cited by
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“At the same time, these methods were able to handle different types of data, including images [13], speech [20], text [10], [38], and structured data, such as molecules [162], [165] or genetic sequences [191].”
BackgroundMethodInfluentialSelf-citationGraph Neural Networks in Recommender Systems: A Survey
Wu, S., Sun, F., Zhang, W., Xie, X., & Cui, B. (2023) · ACM Computing Surveys · doi:10.1145/3535101 · Cited by 1,326
Explainability in Graph Neural Networks: A Taxonomic Survey
Yuan, H., Yu, H., Gui, S., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204236 · Cited by 583
Geometric deep learning on molecular representations
Atz, K., Grisoni, F., & Schneider, G. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00418-8 · Cited by 373
From attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181
Metadata
- Authors
- Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schutt, K. T., Muller, K.-R., & Montavon, G.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-11-01
- Volume
- 44
- Issue
- 11
- Pages
- 7581-7596
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3108823960
- Cited by
- 259
- References
- 63
- FWCI
- 13.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.