Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
Open accessThe DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
Deep learning models have achieved remarkable success in natural language inference (NLI) tasks.While these models are widely explored, they are hard to interpret and it is often unclear how and why they actually work.In this paper, we take a step toward explaining such deep learning based models through a case study on a popular neural model for NLI.In particular, we propose to interpret the intermediate layers of NLI models by visualizing the saliency of attention and LSTM gating signals.We present several examples for which our methods are able to reveal interesting insights and identify the critical information contributing to the model decisions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While these methods are widely adopted [24, 43, 49, 75], alternative approaches may provide more benefit to team performance.”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“There are a few attempts toward explaining/interpreting deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Bahdanau et al., 2014; Denil et al., 2014; Li et al., 201…”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 2×
“The concept of saliency was first introduced in vision for visualizing the spatial support on an image for a particular object class (Simonyan et al., 2013).”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“Using the SNLI (Bowman et al., 2015) data, we train two variants of ESIM, with dimensionality 50 and 300 respectively, referred to as ESIM-50 and ESIM-300 in the remainder of the paper.”
MethodInfluentialA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 1×
“Several pieces of prior work in NLI have attempted to visualize the attention layer to provide some understanding of their models (Ghaeini et al., 2018b; Parikh et al., 2016).”
BackgroundEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2017), are variants of the ESIM model (Chen et al., 2017), which we choose to analyze in this paper.”
MethodBilateral Multi-Perspective Matching for Natural Language Sentences
Wang, Z., Hamza, W., & Florian, R. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/579 · Cited by 739 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodVisualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625 · cited 4×
“…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
BackgroundMethodInfluentialLearned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2×
“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
MethodUnderstanding Neural Networks through Representation Erasure
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412 · cited 2×
“…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
Background
Research works printed but not linked: 2
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. Proceedings of the 2015 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Cited by
Items 101–117 of 117
Dong, J., Chen, S., & Labi, S. (2023) · Purdue e-Pubs (Purdue University System) · doi:10.5703/1288284317701 · Cited by 5
Training with Adversaries to Improve Faithfulness of Attention in Neural Machine Translation
Moradi, P., Kambhatla, N., & Sarkar, A. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing: Student Research Workshop (AACL-IJCNLP SRW) · doi:10.18653/v1/2020.aacl-srw.14 · Cited by 1
“These weights are often implicitly or explicitly regarded as an interpretation for the model’s prediction at the time step t (Tu et al., 2016; Mi et al., 2016; Liu et al., 2016; Wang et al., 2016; Lee et al., 2017; Ding et al., 2017; Ghaeini et al., 2018).”
BackgroundFormality Style Transfer Within and Across Languages with Limited Supervision
Niu, X. (2019) · University Libraries (University of Maryland) · doi:10.13016/nb3v-spmo · Cited by 1
Naeem, S., & Beg, O. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07848 · Cited by 1
“(Ghaeini et al., 2018).”
Understanding Memory Modules on Learning Simple AlgorithmsPreprint
Wang, K., Yu, Z., Wang, S., Zhang, J., & Zong, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00820
“Some visualize the dynamics of the LSTM gates in terms of absolute value and saturation rates that keep track of the structure hints [Karpathy et al., 2015; Ghaeini et al., 2018].”
BackgroundNovel Algorithms for Understanding Online Reviews
Shi, T. (2021) · VTechWorks (Virginia Tech)
Morgan, J. (2022) · Cronfa (Swansea University) · doi:10.23889/suthesis.59258
An empirical user-study of text-based nonverbal annotation systems for human-human conversationsPreprint
Kim, J. Y., & Yacef, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.14928
“Instead of producing heatmaps [28] or highlighting the words with different colours [29], we directly enlarge and darken the words in the visualization proportional to the attention weights to try reduce the cognitive load of the reader.”
The model thinks what?! Interpreting deep NLP models with rationales and influence
Jain, S. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467212
Explaining semantic role labelling outputs with word feature importance
Tuan-Nguyen, H.-D. (2025) · Science and Technology Development Journal - Natural Sciences · doi:10.32508/stdjns.v9i3.1397
Mudrakarta, P. K. (2019) · Knowledge@UChicago (University of Chicago) · doi:10.6082/h6jh7-cph18
Interpretable classification and summarization of crisis events from microblogs
Nguyen, T. H. (2024) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/17782
ニューラルネットワークを用いたソーシャルメディア分析へのユーザ関係と投稿関係の導入
Miura, Y. (2019) · Institutional Repositories DataBase (IRDB)
Wang, J., Liang, Y., Sun, Y., & Gong, B. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777962
Nguyen, D. H. (2024) · theses.fr (ABES) · doi:10.70675/97b1b150zc009z4f96z9ec6z2790870387a0
Explainable Deep Learning for the Early Detection of Mental Health Risks in Social Media
Amini, H. (2025) · Spectrum Research Repository (Concordia University)
Representation-Aware Modularity: Efficient Cross-Task Generalization for LLMs
Gong, Z., Sun, Y., Wang, C., Huo, X., Li, P., Zheng, Y., & Wang, Z. (2026) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2026/638
“A higher attention score implies that a token’s representation is more influential in the context of the entire sequence [Ghaeini et al. , 2018; Luo et al. , 2018; Huang et al. , 2025a].”
Metadata
- Authors
- Ghaeini, R., Fern, X., & Tadepalli, P.
- Affiliations
- Oregon State University (US)
- ORCID
- Tadepalli, Prasad
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 4952-4957
- OpenAlex
- W2885396331
- Cited by
- 91
- References
- 20
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 10 · 2022: 15 · 2021: 22 · 2020: 20 · 2019: 16
- FWCI
- 5.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language inference, model interpretability, attention visualization, saliency maps, explainable AI, neural network decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.