CoChat

Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference

Open access
Ghaeini, Reza, Fern, Xiaoli, Tadepalli, Prasad · 2018 · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.

Abstract

Deep learning models have achieved remarkable success in natural language inference (NLI) tasks.While these models are widely explored, they are hard to interpret and it is often unclear how and why they actually work.In this paper, we take a step toward explaining such deep learning based models through a case study on a popular neural model for NLI.In particular, we propose to interpret the intermediate layers of NLI models by visualizing the saliency of attention and LSTM gating signals.We present several examples for which our methods are able to reveal interesting insights and identify the critical information contributing to the model decisions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d18-1537

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Normalized attention and attention saliency visualization. Each column shows visualization of one sample. Top plots depict attention visualization and bottom ones represent attention saliency visualization. Predicted (the same as Gold) label of each sample is shown on top of each column.
Figure 1 Normalized attention and attention saliency visualization. Each column shows visualization of one sample. Top plots depict attention visualization and bottom ones represent attention saliency visualization. Predicted (the same as Gold) label of each sample is shown on top of each column.CC-BY · From the PDF
Figure 2 Normalized attention and attention saliency visualizations of two examples (p1 and p2) for ESIM-50 (a) and ESIM-300 (b) models. Each column indicates visualization of a model and each row represents visualization of one example.
Figure 2 Normalized attention and attention saliency visualizations of two examples (p1 and p2) for ESIM-50 (a) and ESIM-300 (b) models. Each column indicates visualization of a model and each row represents visualization of one example.CC-BY · From the PDF
Figure 3 Normalized signal and saliency norms for the input and inference LSTMs (forward) of ESIM-50 for three examples. The bottom (top) three rows show the signals of the input (inference) LSTM. Each row shows one of the three gates (input, forget and output).
Figure 3 Normalized signal and saliency norms for the input and inference LSTMs (forward) of ESIM-50 for three examples. The bottom (top) three rows show the signals of the input (inference) LSTM. Each row shows one of the three gates (input, forget and output).CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Neural Machine Translation by Jointly Learning to Align and Translate

    Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×

    “There are a few attempts toward explaining/interpreting deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Bahdanau et al., 2014; Denil et al., 2014; Li et al., 201…”
    Background
  • Deep Contextualized Word Representations

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×

    “Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
    Method
  • Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

    Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 2×

    “The concept of saliency was first introduced in vision for visualizing the spatial support on an image for a particular object class (Simonyan et al., 2013).”
    Background
  • A large annotated corpus for learning natural language inference

    Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×

    “Using the SNLI (Bowman et al., 2015) data, we train two variants of ESIM, with dimensionality 50 and 300 respectively, referred to as ESIM-50 and ESIM-300 in the remainder of the paper.”
    MethodInfluential
  • A Decomposable Attention Model for Natural Language Inference

    Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 1×

    “Several pieces of prior work in NLI have attempted to visualize the attention layer to provide some understanding of their models (Ghaeini et al., 2018b; Parikh et al., 2016).”
    Background
  • Enhanced LSTM for Natural Language Inference

    Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×

    “, 2017), are variants of the ESIM model (Chen et al., 2017), which we choose to analyze in this paper.”
    Method
  • Bilateral Multi-Perspective Matching for Natural Language Sentences

    Wang, Z., Hamza, W., & Florian, R. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/579 · Cited by 739 · cited 2×

    “Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
    Method
  • Visualizing and Understanding Neural Models in NLP

    Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625 · cited 4×

    “…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
    BackgroundMethodInfluential
  • Learned in Translation: Contextualized Word Vectors

    McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2×

    “Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”
    Method
  • Understanding Neural Networks through Representation Erasure

    Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412 · cited 2×

    “…deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Li et al., 2016; Arras et al., 2017; Li et al., 2017; Rei and Søgaard, 2018).”
    Background

Show all 19 references

Research works printed but not linked: 2

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. Proceedings of the 2015 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642.
  2. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.

Cited by

Metadata

Authors
Ghaeini, R., Fern, X., & Tadepalli, P.
Affiliations
Oregon State University (US)
Venue
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Published
2018
Pages
4952-4957
OpenAlex
W2885396331
Cited by
91
References
20
Citations by year
2026: 2 · 2025: 2 · 2024: 4 · 2023: 10 · 2022: 15 · 2021: 22 · 2020: 20 · 2019: 16
FWCI
5.90
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
natural language inference, model interpretability, attention visualization, saliency maps, explainable AI, neural network decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.