Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Open accessThe DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
Deep learning models have achieved remarkable success in natural language inference (NLI) tasks.While these models are widely explored, they are hard to interpret and it is often unclear how and why they actually work.In this paper, we take a step toward explaining such deep learning based models through a case study on a popular neural model for NLI.In particular, we propose to interpret the intermediate layers of NLI models by visualizing the saliency of attention and LSTM gating signals.We present several examples for which our methods are able to reveal interesting insights and identify the critical information contributing to the model decisions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While these methods are widely adopted [24, 43, 49, 75], alternative approaches may provide more benefit to team performance.”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“There are a few attempts toward explaining/interpreting deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Bahdanau et al., 2014; Denil et al., 2014; Li et al., 201…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 2×
“The concept of saliency was first introduced in vision for visualizing the spatial support on an image for a particular object class (Simonyan et al., 2013).”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“Using the SNLI (Bowman et al., 2015) data, we train two variants of ESIM, with dimensionality 50 and 300 respectively, referred to as ESIM-50 and ESIM-300 in the remainder of the paper.”
MethodInfluentialA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 1×
“Several pieces of prior work in NLI have attempted to visualize the attention layer to provide some understanding of their models (Ghaeini et al., 2018b; Parikh et al., 2016).”
BackgroundEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2017), are variants of the ESIM model (Chen et al., 2017), which we choose to analyze in this paper.”
MethodBilateral Multi-Perspective Matching for Natural Language Sentences
Wang, Z., Hamza, W., & Florian, R. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/579 · Cited by 739 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodVisualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625 · cited 4×
“…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
BackgroundMethodInfluentialLearned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodUnderstanding Neural Networks through Representation Erasure
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412 · cited 2×
“…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
Background
Research works printed but not linked: 2
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. Proceedings of the 2015 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Cited by
Jain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 2,023
“Comparing such importance measures to attention scores may provide additional insights into the working of attention based models (Ghaeini et al., 2018).”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“ghlights, where the interface highlights spans of text that are most salient to the model’s prediction directly in the input text, so the user need not go back and forth between input and explanation [18, 36, 42, 62]. Figure2shows one example beer review used in our study. We also experimented with a more challenging…”
BackgroundMethodInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
“ghlights, where the interface highlights spans of text that are most salient to the model’s prediction directly in the input text, so the user need not go back and forth between input and explanation [18, 36, 42, 62]. Figure2shows one example beer review used in our study. We also experimented with a more challenging…”
BackgroundMethodInfluentialSerrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690
“Since attention calculates a distribution over inputs, prior work has used attention as a tool for interpretation of model decisions (Wang et al., 2016; Lee et al., 2017; Lin et al., 2017; Ghaeini et al., 2018).”
MethodAn Empirical Study of Spatial Attention Mechanisms in Deep Networks
Zhu, X., Cheng, D., Zhang, Z., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00679 · Cited by 659
“Many works [15, 35, 16] suggest that attention weight assignment in encoder-decoder attention plays a role similar to word alignment in traditional approaches [1, 7, 27, 5].”
BackgroundMethodTowards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Attention and gating are frequently used in NLP models, and there have been plenty of work aiming to understand the model through these weights [2, 40, 111, 112] and investigate the reliability of using them as explanations [55, 103, 129].”
BackgroundAnalysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
“Saliency can also be computed with respect to intermediate values, rather than input features (Ghaeini et al., 2018).”
BackgroundJain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 467
“Comparing such importance measures to attention scores may provide additional insights into the working of attention based models (Ghaeini et al., 2018).”
BackgroundAttention Interpretability Across NLP Tasks
Vashishth, S., Upadhyay, S., Tomar, G. S., & Faruqui, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11218 · Cited by 63
“Apart from substantial performance benefit (Vaswani et al., 2017), attention also provides interpretability to neural models (Wang et al., 2016; Lin et al., 2017; Ghaeini et al., 2018) which are usually criticized for being black-box function approximators (Chakraborty et al., 2017).”
BackgroundExplaining Explanations: Axiomatic Feature Interactions for Deep Networks
Janizek, J. D., Sturmfels, P., & Lee, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04138 · Cited by 37
“Previous work suggested investigating the internal weights of the attention mechanisms in attention-based models (Ghaeini et al., 2018; Lee et al., 2017; Lin et al., 2017; Wang et al., 2016).”
BackgroundA Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing · doi:10.18653/v1/2020.aacl-main.46 · Cited by 203
“It is possible to utilize (and even combine) other operations presented here to interpret gating signals to aid feature importance explainability (Ghaeini et al., 2018).”
BackgroundSelf-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 189
Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
Chen, H., Zheng, G., & Ji, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.494 · Cited by 143
On Model Stability as a Function of Random Seed
Madhyastha, P., & Jain, R. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1087 · Cited by 123
Making Sense of Machine Learning: A Review of Interpretation Techniques and Their Applications
Tursunalieva, A., Alexander, D. L., Dunne, R. A., Li, J., Riera, L., & Zhao, Y. (2024) · Applied Sciences · doi:10.3390/app14020496 · Cited by 121
Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., & Labi, S. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104358 · Cited by 122
Illuminating the Black Box: Interpreting Deep Neural Network Models for Psychiatric Research
Sheu, Y.-H. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.551299 · Cited by 112
“Ghaeini et al. (75) propose “attention saliency,” which, instead of looking at attention per se , visualizes a score defined by calculating the absolute value of the derivative of the model output with respect to the unnormalized attention weight and show that the attention saliency score provide more meaningful interp…”
BackgroundHuman-grounded Evaluations of Explanation Methods for Text Classification
Lertvittayakumjorn, P., & Toni, F. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1523 · Cited by 60
“Besides, there are other explanation methods designed for specific deep learning architectures such as attention mechanism (Ghaeini et al., 2018) and extractive rationale generation (Lei et al.”
MethodSaliency-driven Word Alignment Interpretation for Neural Machine Translation
Ding, S., Xu, H., & Koehn, P. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5201 · Cited by 82
“Koehn and Knowles (2017) showed that attention-based word alignment interpretation may be subject to “off-by-one” errors.”
BackgroundInterpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 82
“For natural language inference, [63] examined the attention weights and the attention saliency, illustrating that attention heatmaps identify the alignment evidence supporting model predictions, and the attention saliency shows how much such alignment impacts the decisions.”
BackgroundEnhanced aspect-based sentiment analysis models with progressive self-supervised attention learning
Su, J., Tang, J., Jiang, H., Lu, Z., Ge, Y., Song, L., Xiong, D., Sun, L., & Luo, J. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103477 · Cited by 70
Chrysostomou, G., & Αλέτρας, Ν. (2021) · doi:10.18653/v1/2021.acl-long.40 · Cited by 69
“Attention weights have been extensively used to interpret model predictions in NLP; i.e. (Cho et al., 2014; Xu et al., 2015; Barbieri et al., 2018; Ghaeini et al., 2018).”
MethodVisual Correspondence-Based Explanations Improve AI Robustness and human-AI Team Accuracy
Taesiri, M. R., Nguyen, G., & Nguyen, A. K. (2022) · neural information processing systems · doi:10.52202/068431-2485 · Cited by 65
Self-Explaining Structures Improve NLP ModelsPreprint
Sun, Z., Fan, C. C., Han, Q., Sun, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01786 · Cited by 62
“Using attentions as a tool for model interpretation, Ghaeini et al. (2018) visualized attention heatmaps to understand how natural language inference models build interactions between two sentences; Vig and Belinkov (2019); Tenney et al. (2019); Clark et al. (2019); Htut et al. (2019) analyzed the…”
MethodUnderstanding Neural Abstractive Summarization Models via Uncertainty
Xu, J., Desai, S., & Durrett, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.508 · Cited by 37
The Role of Human Knowledge in Explainable AI
Tocchetti, A., & Brambilla, M. (2022) · Data · doi:10.3390/data7070093 · Cited by 55
“Ghaeini et al. [26] utilised saliency visualisations to explain a neural model performing Natural Language Inference (NLI).”
BackgroundExploring Interpretable Predictive Models for Business Processes
Sindhgatta, R., Moreira, C., Ouyang, C., & Barros, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58666-9_15 · Cited by 55
“Current studies in the literature have used attention-based models mainly in Natural Language Processing tasks [17, 6, 4].”
BackgroundMethodT.y.s.s, S., Xu, S., Ichim, O., & Grabmair, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.74 · Cited by 47
The Past, Present, and Prospective Future of XAI: A Comprehensive Review
Islam, M. U., Mottalib, M. M., Hassan, M., Alam, Z. I., Zobaed, S., & Rabby, M. F. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-96630-0_1 · Cited by 46
A Comprehensive Survey on Self-Interpretable Neural Networks
Ji, Y., Sun, Y., Zhang, Y., Wang, Z., Zhuang, Y., Gong, Z., Shen, D., Qin, C., Zhu, H., & Xiong, H. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3635153 · Cited by 45
“Techniques such as sampling-based approaches [135], [250], attention mechanisms [19], [251], [252], and sparsity-inducing constraints [86], [253] highlight important input elements.”
Kang, P., Li, J., Jiang, S., & Shull, P. B. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3232159 · Cited by 43
“These studies clarified basic concepts and definitions of XAI and provided precursory demonstrations in the fields of medical image processing [13], autonomous driving [14], and nat-ural language processing [15].”
BackgroundExplainable Natural Language Processing
Søgaard, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01118ed1v01y202107hlt051 · Cited by 39
Understanding Multi-Head Attention in Abstractive SummarizationPreprint
Baan, J., ter Hoeve, M., van der Wees, M., Schuth, A., & de Rijke, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03898 · Cited by 36
“Attention (Bah-danau et al., 2014; Luong et al., 2015) has gained popularity as a means of obtaining insight in the inner workings of deep neural networks (e.g., Lei, 2017; Choi et al., 2016; Gilpin et al., 2018; Ghaeini et al., 2018).”
Finding Experts in Transformer ModelsPreprint
Suau, X., Zappella, L., & Apostoloff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07647 · Cited by 35
“Saliency Some works focus on analyzing the self-attention layers in the Transformer blocks, visualizing saliency [12] or studying how attention heads attend to different word families [7].”
BackgroundTowards Explainability in Using Deep Learning for the Detection of Anorexia in Social Media
Amini, H., & Kosseim, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-51310-8_21 · Cited by 33
Multimodal Wearable Biosensing Meets Multidomain AI: A Pathway to Decentralized Healthcare
Liu, C., Fan, H., Kim, M., Zhou, T., Yang, P., Zhao, L., Wang, Y., Che, Z., Liu, C., Li, B., et al. (2026) · Advanced Science · doi:10.1002/advs.202522900 · Cited by 32
“Such evaluations can be supported by interpretability and causal analysis tools, including feature attribution [ 198 ], attention maps [ 199 ], causal graphs [ 200, 201 ], and counterfactual explanations [ 197 ], which probe decision sensitivity and help distinguish clinically meaningful…”
Fooling Explanations in Text ClassifiersPreprint
Ivankay, A., Girardi, I., Marchiori, C., & Frossard, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.03178 · Cited by 31
“This is especially problematic given the increased reliance on and fragility of attention mechanisms as inherently explainable methods, as stated in Ghaeini et al. (2018).”
Saliency Learning: Teaching the Model Where to Pay Attention
Ghaeini, R., Fern, X. Z., Shahbazi, H., & Tadepalli, P. V. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1404 · Cited by 29
“Recent work on explanation and interpretation has introduced a few approaches (Simonyan et al., 2013; Ribeiro et al., 2016; Lei et al., 2016; Li et al., 2016, 2017; Ghaeini et al., 2018b; Ribeiro et al., 2018) for explanation.”
BackgroundInfluentialSelf-citationStructured Self-Attention Weights Encode Semantics in Sentiment Analysis
Wu, Z., Nguyen, T.-S., & Ong, D. C. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.24 · Cited by 29
More Interpretable Graph Similarity Computation via Maximum Common Subgraph Inference
Lan, Z., Hong, B., Ma, Y., & Ma, F. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3387044 · Cited by 26
“Attention provides a way to interpret the model’s inner workings [18], [19], [20], [21] and can improve its performance.”
BackgroundSaliency Learning: Teaching the Model Where to Pay AttentionPreprint
Ghaeini, R., Fern, X. Z., Shahbazi, H., & Tadepalli, P. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08649 · Cited by 26
“Recent work on explanation and interpretation has introduced a few approaches (Simonyan et al., 2013; Ribeiro et al., 2016; Lei et al., 2016; Li et al., 2016, 2017; Ghaeini et al., 2018b; Ribeiro et al., 2018) for explanation.”
BackgroundInfluentialSelf-citationProceedings of the 3rd Workshop on Neural Generation and Translation
(2019) · doi:10.18653/v1/d19-56 · Cited by 24
Supervised Contrastive Learning for Interpretable Long-Form Document Matching
Jha, A., Rakesh, V., Chandrashekar, J., Samavedhi, A., & Reddy, C. K. A. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3542822 · Cited by 23
“While some works do demonstrate how attention scores can be used to provide insights into attention-based models [13, 39, 42], we define interpretability as the weighted similarity scores between different sections and different chunks of text in a document.”
BackgroundAttention Is All You Need for LLM-Based Code Vulnerability Localization
Li, Y., Li, X., Wu, H., Zhang, Y., Cheng, X., Liu, Y., Xu, F., & Zhong, S. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3725564 · Cited by 22
“There has been significant research [9], [17], [26], [37], [44], [48] on leveraging attention for the interpretability of language models.”
Interrogating the Explanatory Power of Attention in Neural Machine Translation
Moradi, P., Kambhatla, N., & Sarkar, A. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5624 · Cited by 22
“Saliency methods use the gradient of the inputs to define importance (Li et al., 2016a; Ghaeini et al., 2018; Ding et al., 2019).”
MethodMeasuring and Improving Faithfulness of Attention in Neural Machine Translation
Moradi, P., Kambhatla, N., & Sarkar, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.243 · Cited by 22
“…the model’s prediction at the time step t (Tu et al., 2016; Mi et al., 2016; Liu et al., 2016; Wang et al., 2016; Lee et al., 2017; Ding et al., 2017; Ghaeini et al., 2018 (Jain and Wal-lace, 2019; Wiegreffe and Pinter, 2019) have used a binary measure: either attention is faithful or it is not.”
BackgroundMethodExplaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
Chen, H., Feng, S., Ganhotra, J., Wan, H., Gunasekara, C., Joshi, S., & Ji, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.306 · Cited by 20
Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains
Shnarch, E., Choshen, L., Moshkowich, G., Aharonov, R., & Slonim, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.243 · Cited by 19
DNN2LR: Interpretation-inspired Feature Crossing for Real-world Tabular DataPreprint
Liu, Z., Liu, Q., Zhang, H., & Chen, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09775 · Cited by 17
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 17
Huo, Y., Wong, D. F., Ni, L. M.-S., Chao, L. S., & Zhang, J. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106389 · Cited by 17
Human-grounded Evaluations of Explanation Methods for Text ClassificationPreprint
Lertvittayakumjorn, P., & Toni, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11355 · Cited by 17
“Besides, there are other explanation methods designed for specific deep learning architectures such as attention mechanism (Ghaeini et al., 2018) and extractive rationale generation (Lei et al.”
MethodDas, R. K., & Dodge, J. E. (2026) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2026.339 · Cited by 15
“In prior works, researchers used other techniques, such as causal frameworks [58], attention score-based methods [59], saliency visualizations [60], and Layer-wise Relevance Propagation (LRP) [61] to explain the output of models.”
A Natural Language Process-Based Framework for Automatic Association Word Extraction
Hu, Z., Luo, J., Zhang, C., & Li, W. (2019) · IEEE Access · doi:10.1109/access.2019.2962154 · Cited by 15
“The proposed first-derivative saliency of attention succeeds in pinpointing how the alignment process contributes to the final predictions [17].”
BackgroundRelation Extraction with Explanation
Shahbazi, H., Fern, X. Z., Ghaeini, R., & Tadepalli, P. V. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.579 · Cited by 15
Self-citationSalKG: Learning From Knowledge Graph Explanations for Commonsense ReasoningPreprint
Chan, A., Xu, J., Long, B., Sanyal, S., Gupta, T., & Ren, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08793 · Cited by 15
“…gradient-based (Sundararajan et al., 2017; Li et al., 2015; Denil et al., 2014), attention-based (Mohankumar et al., 2020; Tutek and Šnajder, 2020; Ghaeini et al., 2018; Lee et al., 2017), and occlusion-based (DeYoung et al., 2019; Poerner et al., 2018; Kádár et al., 2017; Li et al., 2016)…”
Pair the Dots: Jointly Examining Training History and Test Stimuli for Model InterpretabilityPreprint
Meng, Y., Fan, C. C., Sun, Z., Hovy, E. H., Wu, F., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06943 · Cited by 13
“For example, Ghaeini et al. (2018) A less relevant line of work is model poisoning, which manipulates the training process using poisoned data (Biggio et al., 2012; Mei and Zhu, 2015; Muñoz-González et al., 2017; Shafahi et al., 2018), or adversarial examples (Goodfellow et al., 2014; Kurakin et…”
BackgroundLioNets: a neural-specific local interpretation technique exploiting penultimate layer information
Mollas, I., Bassiliades, N., & Tsoumakas, G. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03351-4 · Cited by 12
“Several works have exploited the attention layer, extracting interpretations [52] and even visualising this knowledge [53].”
BackgroundA Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
Nguyen, D. H., Gravier, G., & Sébillot, P. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00259 · Cited by 12
“Most recent studies on explanability derived from attention maps assess the faithfulness of the attention mechanism, e.g., qualitatively looking at attention weights on a few examples [9], [14]–[16].”
KW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification
Jang, H., Bang, S., Xiao, W., Carenini, G., Ng, R. T., & Lee, Y. J. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.10 · Cited by 13
“However, a recent and popular method in NLP has been the use of an attention mechanism, which was found to be effective in helping interpret complex models by highlighting which inputs are informative to prediction (Wang et al., 2016; Lin et al., 2017; Ghaeini et al., 2018; Seo et al., 2016).”
MethodCorpus-level and Concept-based Explanations for Interpretable Document Classification
Shi, T. Z., Zhang, X., Wang, P., & Reddy, C. K. A. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3477539 · Cited by 12
“visualizing heat-maps of attention weights [11, 40, 42, 45], namely attention visualization.”
Toward a traceable, explainable, and fairJD/Resume recommendation systemPreprint
Barrak, A., Adams, B., & Zouaq, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.08960 · Cited by 12
“These operations are: (1) Layer-wise relevance propagation [52], (2) input perturbations (3) Attention base models feature importance [53], (4) LSTM and feature importance explainability [54], and (5) Explainability-aware architecture design [55].”
BackgroundResultTowards Reconciling Usability and Usefulness of Explainable AI MethodologiesPreprint
Tambwekar, P., & Gombolay, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.05347 · Cited by 12
“Such approaches include attention visualization [15], behavior summarization [16,17], model reconciliation [18], and much more.”
BackgroundNguyen, T. H., & Rudra, K. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583861 · Cited by 11
“attention weights as a mode of explanation [3, 10].”
BackgroundOffensive text detection on English Twitter with deep learning models and rule-based systems
Gémes, K., Kovacs, A., Reichel, M., & Recski, G. (2021) · reposiTUm (TU Wien) · doi:10.34726/4342 · Cited by 11
Tambwekar, P., & Gombolay, M. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1375490 · Cited by 10
Shin, Y.-M., Li, S., Cao, X., & Shin, W.-Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i19.34254 · Cited by 10
“The choice of attention as a baseline is natural, as self-attention itself can be viewed as a direct way to provide model interpretations without any separate explanation method (Lee, Shin, and Kim 2017; Ghaeini, Fern, and Tadepalli 2018; Hao et al. 2021; Aflalo et al. 2022; Deiseroth et al. 2023).”
GHOST at SemEval-2021 Task 5: Is explanation all you need?
Pluciński, K., & Klimczak, H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.114 · Cited by 9
“Attention has been commonly used by many researchers as a medium of explanation for the predictions made by the model (Ghaeini et al., 2018).”
BackgroundHierarchical Attention Network for Interpretable ECG-based Heart Disease ClassificationPreprint
Rodriguez, M. P., & Nafea, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.03703 · Cited by 9
On Explaining with Attention Matrices
Naim, O., & Asher, N. M. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240594 · Cited by 8
“As such it stands to reason that AW should have an explanatory role [8, 20, 43, 10].”
BackgroundSaliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification
Lai, S., Huang, H., Jing, D., Chen, Y., Xu, J., & Liu, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.55 · Cited by 8
“Since attention mechanisms (Bahdanau et al., 2014) boosted performance on many current NLP tasks, using attention weight as explanation of model predictions is a general approach for many models (Wang et al., 2016; Lin et al., 2017; Ghaeini et al., 2018).”
BackgroundDo Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition
Lyu, S., Wu, X., Li, J., Chen, Q., & Chen, H. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3136598 · Cited by 8
Emergent Structures and Training Dynamics in Large Language Models
Teehan, R., Clinciu, M., Serikov, O., Szczechla, E., Seelam, N., Mirkin, S., & Gokaslan, A. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.11 · Cited by 7
“To address these limi-653 tations, Ghaeini et al. (2018) examine the saliency of attention and LSTM gating signal in the inter-655 mediate layers of ESIM models, an architecture designed for NLI tasks (Chen et al., 2017).”
MethodRelevance-based Infilling for Natural Language Counterfactuals
Betti, L., Abrate, C., Bonchi, F., & Kaltenbrunner, A. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615029 · Cited by 7
Uddin, M. R., Mahbub, S., Rahman, M. S., & Bayzid, M. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/786921 · Cited by 7
“while generating the output [50, 73–76], and hence it was used to develop interpretable models [77–80].”
MethodZeldes, A., & Liu, Y. (2022) · Dialogue & Discourse · doi:10.5087/dad · Cited by 7
“While earlier work on interpreting neural models has focused on token-wise softmax probability (Zeldes, 2018a) or attention weights (Ghaeini et al., 2018), using contextualized embeddings complicates the evaluation: since word representations are adjusted to reflect neighboring words, the model may assign higher impor…”
BackgroundUse all tokens method to improve semantic relationship learning
Lee, K., Choi, G. H., & Choi, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120911 · Cited by 6
Context-Aware Feature Selection and Classification
Wang, J., & Bilgic, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/480 · Cited by 6
“While several papers used attention mechanism for interpretability [Wang et al., 2016; Ghaeini et al., 2018], other papers pointed out that attention weights often reflect how much the model attend to the hidden representation of each input, which might already have mixed in information from other inputs [Bastings and…”
MethodControlling the Focus of Pretrained Language Generation Models
Ji, J., Kim, Y., Glass, J. R., & He, T. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.260 · Cited by 6
Attentional Multi-Reading Sarcasm DetectionPreprint
Ghaeini, R., Fern, X. Z., & Tadepalli, P. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03051 · Cited by 6
“Finally, we present an extensive data analysis to (1) provide explanations regarding our model’s decisions and behavior by visualizing attention and attention saliency(Ghaeini et al., 2018b); (2) study the impact and effect of utterance and the conversational context on our model’s final prediction.”
BackgroundMethodSelf-citationZhang, Y., Wittrup, E. M., Hodgman, M. W., Mathis, M. R., & Najarian, K. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3551312 · Cited by 5
Conditional Natural Language Inference
Kim, Y., Rahimi, R., & Allan, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.456 · Cited by 5
An Eye Opener Regarding Task-Based Text Gradient Saliency
Wu, G., Bolliger, L. S., Reich, D. R., & Jäger, L. A. (2024) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2024.cmcl-1.22 · Cited by 5
“Attention mechanisms assign dynamic weights to input tokens, offering a method to understand a model’s internal functioning and decision-making processes (Wang et al., 2016; Ghaeini et al., 2018).”
MethodA Systematic Review of Attention Models in Natural Language Processing
Shabir, A., Shabir, A., Ahmed, K. T., Ahmed, K. T., Kanwal, K., Almas, A., Raza, S., Fatima, M., & Abbas, T. (2024) · STATISTICS COMPUTING AND INTERDISCIPLINARY RESEARCH · doi:10.52700/scir.v6i1.157 · Cited by 5
Using Interpretation Methods for Model Enhancement
Chen, Z., Jiang, C., & Tu, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.28 · Cited by 4
“The attention mechanism calculates a distribution over input tokens, and some previous works use the attention weights as interpretations derived from the model (Wang et al., 2016; Ghaeini et al., 2018).”
MethodRelation Extraction with ExplanationPreprint
Shahbazi, H., Fern, X. Z., Ghaeini, R., & Tadepalli, P. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14271 · Cited by 4
“However, recent studies (Ghaeini et al., 2018; Jain et al., 2019; Wiegreffe and Pinter, 2019) have questioned the validity of attention as a faithful explanation of model’s behavior.”
BackgroundSelf-citationDeep Active Learning by Model InterpretabilityPreprint
Liu, Q., Liu, Z., Zhu, X., & Xiu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12100 · Cited by 4
“In some views, attention in deep models can also be regarded as local interpretations (Ghaeini, Fern, and Tadepalli 2018; Wang et al. 2019).”
BackgroundWhere to Look When Repairing Code? Comparing the Attention of Neural Models and DevelopersPreprint
Huber, D., Paltenghi, M., & Pradel, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07287 · Cited by 6
“Besides improving the prediction accuracy, this special layer, called attention layer [7], reveals which parts of the code are most relevant for the model to make its predictions and has been widely used to understand how a model makes decisions [15, 19, 32, 37, 64].”
BackgroundTowards Explainable Table Interpretation Using Multi-view Explanations
Gao, Y., Wang, P., Zeng, X., Chen, L., Mao, Y., Wei, Z., & Li, M. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00094 · Cited by 3
“Moreover, attention is a valuable tool to explain model decisions in recent work [32], [33].”
MethodReason induced visual attention for explainable autonomous drivingPreprint
Chen, S., Dong, J., Du, R., Li, Y., & Labi, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07380 · Cited by 3
Jouis, G. (2023) · theses.fr (ABES) · doi:10.70675/e2608d6az0bd1z43cezbffez4d7f535ef83e · Cited by 3
How (Un)Faithful is Attention?
Amini, H., & Kosseim, L. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.10 · Cited by 2
“A common approach has been to provide attention heat maps to which input elements the attention component has attended to (e.g. Wang et al., 2016; Lee et al., 2017; Lin et al., 2017; Ghaeini et al., 2018).”
MethodCATAN: Chart-aware temporal attention network for adverse outcome prediction
Gero, Z., & Ho, J. C. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00024 · Cited by 2
“Attention-based RNNs have been proposed to offer better explainability and transparency of the prediction [19]–[21].”
BackgroundSpindola, S., José, M. M., Oliveira, A. S., Cação, F. N., & Cozman, F. G. (2021) · doi:10.5753/eniac.2021.18302 · Cited by 2
“Similarly, [Ghaeini et al. 2020] uses the attention weights to interpret decisions from natural language inference (NLI) models, and [Vig 2019a] found even evidence of model biases with the help of them, which is noteworthy for several current practical reasons [Bender and Friedman 2018].”
BackgroundMethodInfluentialYadav, V., Bethard, S. J., & Surdeanu, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401311 · Cited by 2
“Especially the attention contribution on lexical-matched tokens is often studied as a proxy to interpret neural models on (retrieval based) QA [7] and natural language inference task[8].”
BackgroundTang, R., Chen, H., & Ji, Y. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.30 · Cited by 2
Detecting semantic similarity : biases, evaluation and models
Peinelt, N. (2026) · Warwick Research Archive Portal (University of Warwick) · doi:10.82444/warw.33824893 · Cited by 2
Metadata
- Authors
- Ghaeini, R., Fern, X., & Tadepalli, P.
- Affiliations
- Oregon State University (US)
- ORCID
- Tadepalli, Prasad
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 4952-4957
- OpenAlex
- W2885396331
- Cited by
- 91
- References
- 20
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 10 · 2022: 15 · 2021: 22 · 2020: 20 · 2019: 16
- FWCI
- 5.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language inference, model interpretability, attention visualization, saliency maps, explainable AI, neural network decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.