The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Open accessAbstract
Recently, a technique called Layer-wise Relevance Propagation (LRP) was shown to deliver insightful explanations in the form of input space relevances for understanding feed-forward neural network classification decisions.In the present work, we extend the usage of LRP to recurrent neural networks.We propose a specific propagation rule applicable to multiplicative connections as they arise in recurrent network architectures such as LSTMs and GRUs.We apply our technique to a word-based bi-directional LSTM model on a five-class sentiment prediction task, and evaluate the resulting LRP relevances both qualitatively and quantitatively, obtaining better results than a gradient-based related method which was used in previous work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 6,122
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Self-citationGevrey, M., Dimopoulos, I. F., & Lek, S. (2003) · Ecological Modelling · doi:10.1016/s0304-3800(02)00257-0 · Cited by 1,263
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Self-citationUse of some sensitivity criteria for choosing networks with good generalization ability
Dimopoulos, Y., Bourret, P., & Lek, S. (1995) · Neural Processing Letters · doi:10.1007/bf02309007 · Cited by 267
Research works printed but not linked: 16
- Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016a. Explaining Predictions of Non-Linear Classifiers in NLP. In Proceedings of the ACL 2016 Workshop on Representation Learning for NLP. Association for Computational Linguistics, pages 1-7.
- Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016b. "What is Relevant in a Text Document?": An In- terpretable Machine Learning Approach. arXiv 1612.07843. · arXiv
- Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learn- ing Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Pro- ceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pages 1724-1734.
- Misha Denil, Alban Demiraj, and Nando de Freitas. 2014. Extraction of Salient Sentences from Labelled Documents. arXiv 1412.6815. · arXiv
- Will Landecker, Michael D. Thomure, Luís M. A. Bet- tencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting Individual Classifications of Hierarchical Networks. In IEEE Symposium on Computational Intelligence and Data Mining (CIDM). pages 32-38.
- Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. The Layer-wise Relevance Propaga- tion Toolbox for Artificial Neural Networks. Jour- nal of Machine Learning Research 17(114):1-5.
- Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Juraf- sky. 2016a. Visualizing and Understanding Neural Models in NLP. In Proceedings of the 2016 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (NAACL-HLT). pages 681-691.
- Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Un- derstanding Neural Networks through Representa- tion Erasure. arXiv 1612.08220. · arXiv
- Saif M. Mohammad. 2017. Challenges in Sentiment Analysis. In Erik Cambria, Dipankar Das, Sivaji Bandyopadhyay, and Antonio Feraco, editors, A Practical Guide to Sentiment Analysis. Springer In- ternational Publishing, pages 61-83.
- Grégoire Montavon, Wojciech Samek, and Klaus- Robert Müller. 2017b. Methods for Interpreting and Understanding Deep Neural Networks. arXiv 1706.07979. · arXiv
- James Murdoch and Arthur Szlam. 2017. Automatic Rule Extraction from Long Short Term Memory Networks. In International Conference on Learning Representations Conference Track (ICLR).
- Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. SemEval- 2016 Task 4: Sentiment Analysis in Twitter. In Pro- ceedings of the 10th International Workshop on Se- mantic Evaluation (SemEval-2016). Association for Computational Linguistics, pages 1-18.
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Ex- plaining the Predictions of Any Classifier. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing. pages 1135-1144.
- Mike Schuster and Kuldip K. Paliwal. 1997. Bidirec- tional Recurrent Neural Networks. IEEE Transac- tions on Signal Processing 45(11):2673-2681.
- Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive Deep Mod- els for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP). Association for Computational Linguistics, pages 1631-1642.
- Matthew D. Zeiler and Rob Fergus. 2014. Visualiz- ing and Understanding Convolutional Networks. In Computer Vision ECCV 2014: 13th European Con- ference. pages 818-833.
Cited by
Sorted by citations
Items 301–359 of 359
Parker, C., Nelson, E., & Zhang, T. (2024) · Frontiers in Genetics · doi:10.3389/fgene.2024.1375468 · Cited by 1
Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation
Sarmiento, P. Y., Witzke, S., Klein, N., & Renard, B. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70359-1_20 · Cited by 1
Nonlinearized Relevance Propagation
Zhang, Q., & Ohsawa, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-97304-3_69 · Cited by 1
Concept Cloud-Based Sentiment Visualization for Financial Reviews
Ito, T., Tsubouchi, K., Sakaji, H., Yamashita, T., & Izumi, K. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-38227-8_21 · Cited by 1
An Overview of Computational Approaches for Interpretation AnalysisPreprint
Blandfort, P., Hees, J., & Patton, D. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04028 · Cited by 2
Human-grounded Evaluations of Explanation Methods for Text ClassificationPreprint
Lertvittayakumjorn, P., & Toni, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11355 · Cited by 15
Shapley Interpretation and Activation in Neural NetworksPreprint
Li, Y., & Cui, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06143 · Cited by 1
FIND: Human-in-the-Loop Debugging Deep Text ClassifiersPreprint
Lertvittayakumjorn, P., Specia, L., & Toni, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04987 · Cited by 13
Collaborative Technologies and Data Science in Artificial Intelligence Applications
Online, C. (. 2., Hajian, A., Baloian, N., Inoue, T., Luther, W. J., & Berlin, L. V. (2020) · Logos Verlag Berlin eBooks · doi:10.30819/5141 · Cited by 2
It’s All in the Name: A Character Based Approach to Infer ReligionPreprint
Chaturvedi, R., & Chaturvedi, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4012431 · Cited by 1
The Analysis and Development of an XAI Process on Feature Contribution ExplanationPreprint
Huang, J., Wang, Z., LI, D., & Liu, Y. (2022) · doi:10.36227/techrxiv.21545478.v1 · Cited by 1
From medical imaging to explainable artificial intelligence
Tjoa, E. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/164493 · Cited by 1
Explainable Deep Learning for Analysing Brain Data
Müller, K. R. (2019) · International Winter Conference on Brain-Computer Interface (BCI) · doi:10.1109/iww-bci.2019.8737321
Self-citationUnderstanding Memory Modules on Learning Simple AlgorithmsPreprint
Wang, K., Yu, Z., Wang, S., Zhang, J., & Zong, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00820
Word-level Sentiment Visualizer for Financial Documents
Ito, T., Tsubouchi, K., Sakaji, H., Yamashita, T., & Izumi, K. (2019) · IEEE Conference on Computational Intelligence for Financial Engineering & Economics · doi:10.1109/cifer.2019.8759116
SSNN: Sentiment Shift Neural Network
Ito, T., Tsubouch, K., Sakaji, H., Yamashita, T., & Izumi, K. (2020) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976236.30
Detecting Preventable Disease Risk on Social Media
Bell, D. (2018) · UA Campus Repository (The University of Arizona)
Q-FIT: The Quantifiable Feature Importance Technique for Explainable Machine Learning.Preprint
Adamczewski, K., Harder, F., & Park, M. (2020) · arXiv (Cornell University)
How to predict relapse in leukemia using time series data: A comparative in silico studyPreprint
Hoffmann, H., Baldow, C., Zerjatke, T., Gottschalk, A., Wagner, S., Karg, E., Niehaus, S., Roeder, I., Glauche, I., & Scherf, N. (2020) · medRxiv · doi:10.1101/2020.12.04.20243907
DoLFIn: Distributions over Latent Features for Interpretability
Le, P., & Zuidema, W. H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.127
Training with Adversaries to Improve Faithfulness of Attention in Neural Machine Translation
Moradi, P., Kambhatla, N., & Sarkar, A. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing: Student Research Workshop (AACL-IJCNLP SRW) · doi:10.18653/v1/2020.aacl-srw.14
Split and Expand: An inference-time improvement for Weakly Supervised Cell Instance SegmentationPreprint
Foo, L. G., Ho, R. E., Sun, J., & Binder, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10817
Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning ModelsPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01037
Self-citationTowards a Better Understanding of Gradient-Based Explanatory Methods in NLP
Du, Q. (2021) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2021-045
A Framework for Rationale Extraction for Deep QA modelsPreprint
Ramnath, S., Nema, P., Sahni, D., & Khapra, M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04620
How to Predict Relapse in Leukaemia Using Time Series Data: A Comparative in silico StudyPreprint
Hoffmann, H., Baldow, C., Zerjatke, T., Gottschalk, A., Wagner, S., Karg, E., Niehaus, S., Roeder, I., Glauche, I., & Scherf, N. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3751755
The Comparative Performance of Machine Learning Models for COVID-19 Sentiment Analysis
Rubab, S. F., Mushtaq, M. F., Tahir, M. H., Amna, Samad, A., Gilanie, G., & Ghouse, M. G. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-00828-3_37
Language Inference Using Elman Networks with Evolutionary Training
Anastasopoulos, N., Tsoulos, I. G., Dermatas, E., & Karvounis, E. C. (2022) · Signals · doi:10.3390/signals3030037
Sensor Dynamic Modeling Based on GWO-LSSVM
An, Q., Fu, L., & Zhang, A. (2022) · International Conference on Mechanical Engineering and Automation Science (ICMEAS) · doi:10.1109/icmeas57305.2022.00052
The Analysis and Development of an XAI Process on Feature Contribution ExplanationPreprint
Huang, J., Wang, Z., LI, D., & Liu, Y. (2022) · doi:10.36227/techrxiv.21545478
The Analysis and Development of an XAI Process on Feature Contribution ExplanationPreprint
Huang, J., Wang, Z., LI, D., & Liu, Y. (2022) · doi:10.36227/techrxiv.21545478.v2
User Feedback Severity Level Identification and Classification through Deeper Analysis of Text
Umair, M. B., Irtaza, A., & Salim, S. (2023) · International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) · doi:10.1109/icomet57998.2023.10099177
DARE: Towards Robust Text Explanations in Biomedical and Healthcare Applications
Ivankay, A., Rigotti, M., & Pascal Frossard, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.644
Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation
Korakakis, M., & Vlachos, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.536 · Cited by 2
Interpreting Deep Text Quantification Models
Bang, Y., Khaleel, M., & Tavanapong, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39821-6_25
Cohesive Attention-Based Explanations for Sequences and Explainability in Presence of Event Types
Ger, S., Jambunath, Y. S., Klabjan, D., & Utke, J. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386904
Deep Learning made transferable: studying Brain decoding
Müller, K. R. (2024) · International Winter Conference on Brain-Computer Interface (BCI) · doi:10.1109/bci60775.2024.10480517
Self-citationRadke, T., Fuchs, S., Wilms, C., et al. (2024) · Geoscientific model development · doi:10.5194/gmd-2024-60-cec1
Gomes, A. A. A., Braga, L. J. F., Azevedo, M. P. C., Assunção, G., Carvalho, A., Brandão, M. A., Dalip, D. H., & Pádua, F. L. C. (2024) · doi:10.5753/wperformance.2024.2348
Radke, T., Fuchs, S., Wilms, C., et al. (2024) · Geoscientific model development · doi:10.5194/gmd-2024-60-rc1
Radke, T., Fuchs, S., Wilms, C., et al. (2024) · Geoscientific model development · doi:10.5194/gmd-2024-60-rc2
Unveiling the black box: Enhancing transparency in deep learning through explainable AI
Tingare, B. A., -, P. U., -, A. C., & -, A. K. (2025) · AIP conference proceedings · doi:10.1063/5.0254216
New analytical approaches to life course epidemiology
Börnhorst, C., Wright, M. N., & Didelez, V. (2025) · doi:10.1093/oso/9780198895961.003.0005
Zhang, L., Song, H., Patel, A., Pollack, M. M., & Watson, L. T. (2025) · Research Square · doi:10.21203/rs.3.rs-6787167/v1
Artificial Intelligence Techniques for Sentimental Analysis During Pandemic
Thakur, A., & Agarwal, R. (2025) · Transactions on computer systems and networks · doi:10.1007/978-981-96-5269-3_15
Gantla, S. R., Kurbanova, S., Nandihal, P., & Gulsara, R. (2025) · International Conference on Innovations in Intelligent Systems: Advancements in Computing, Communication, and Cybersecurity (ISAC3) · doi:10.1109/isac364032.2025.11156588 · Cited by 1
Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Models
Hirano, S., Wada, Y., Iida, T., & Sugiura, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4097-6_27
Wu, L., An, J., & He, J. (2025) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2025.180901
From Enigma to Insight: A Pragmatic Survey on Explainable AI (XAI)
Samih, A., Errakha, K., & Marzouk, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14430-0_30
Toward interpretable machine learning, with applications to large-scale industrial systems data
Mita, G. (2021) · theses.fr (ABES) · doi:10.70675/acfd6da1z8046z49d6z8c4ez2da37571763c
Interpretable machine learning for CLAS12 data analysis
Cherrier, N. (2021) · theses.fr (ABES) · doi:10.70675/171a54daz808cz478az83bcz0276fd7612c3 · Cited by 3
A Neural-Symbolic learning framework to produce interpretable predictions for image classification
Bennetot, A. (2022) · theses.fr (ABES) · doi:10.70675/2b59f598z2028z49afz8a20z29b21f19f394
Granular Insights into Financial Discourse : Fine-Grained Opinion Analysis of Expert Texts
Hu, J. (2023) · theses.fr (ABES) · doi:10.70675/ed1d7748z8ebaz4550zbfb8z54bab2a6ac0a
Sparks of explainability : recent advancements in explaining large vision models
Fel, T. (2024) · theses.fr (ABES) · doi:10.70675/2306eab0z8851z470cz9dc4z01481861fb3e · Cited by 1
Tyagi, R., Tomar, N., Chaubey, R. R., Chauhan, P., Garg, P., & Bohra, M. (2026) · International Conference on Intelligent Systems in Engineering, Secured Systems and Cybersecurity (ICISESSC) · doi:10.1109/icisessc68634.2026.11542649
Et-Tousy, J., Malek, C., & Zyane, A. (2026) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset68627.2026.11538732
Beyond Accuracy: Explainability of Transformer Models for Sentiment Analysis
Shahid, S., Aziz, A., Bilal, M., Shoaib, M., & Aaqib, M. (2025) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/1704
Metadata
- Authors
- Arras, L., Montavon, G., Müller, K.-R., & Samek, W.
- Affiliations
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Technische Universität Berlin (DE)
- Freie Universität Berlin (DE)
- Venue
- Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis
- Published
- 2017
- Pages
- 159-168
- OpenAlex
- W2639206954
- Cited by
- 361
- References
- 5
- Citations by year
- 2026: 6 · 2025: 22 · 2024: 42 · 2023: 49 · 2022: 47 · 2021: 63 · 2020: 58 · 2019: 42 · 2018: 26 · 2017: 6
- FWCI
- 16.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft
- Bundesministerium für Bildung und Forschung
- Institute for Information and Communications Technology Promotion (2017-0-00451)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- layer-wise relevance propagation, explainable AI, recurrent neural networks, sentiment analysis, LSTM, GRU, bidirectional LSTM
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
