Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 1,001–1,008 of 1,008
On Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2023) · arXiv · doi:10.48550/arxiv.2311.07466 · Cited by 27
“…methods correlate high attention weights with high feature importance, which is debated (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). iii) Perturbation methods like RISE (Petsiuk et al., 2018) and SHAP (Lundberg and Lee, 2017) compute importance scores…”
Díaz-Rodríguez, N., Lamas, A., Sanchez, J., Franchi, G., Donadello, I., Tabik, S., Filliat, D., Cruz, P., Montes, R., & Herrera, F. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.09.022 · Cited by 2
“they are not easy to quantify, correct, nor convey to non technical audiences [47, 95, 93, 61, 40, 3, 49].”
BackgroundWhy context matters in VQA and Reasoning: Semantic interventions for VLM input modalities
Amara, K., Klein, L., Lüth, C., Jäger, P., Strobelt, H., & El-Assady, M. (2024) · arXiv · doi:10.48550/arxiv.2410.01690 · Cited by 1
“(Serrano & Smith, 2019; Jain & Wallace, 2019) questioned the validity of attention as an indicator of importance, while (Wiegreffe & Pinter, 2019) challenged their arguments, asserting that attention can serve as an explanation, albeit not the definitive one.”
BackgroundCAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents
Hossain, S. M. A., Shayoni, R. K., & Morol, M. K. (2026) · arXiv · doi:10.48550/arxiv.2609.02265 · Cited by 3
“For interpretability we do not report attention or retrieval rank, both of which are known to be unreliable as explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Turpin et al., 2023).”
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
ERP-XTTN: interpretable prototype-guided cross-attention for cross-subject ERP classification
Genevier Wyman, C., & Hirshfield, L. (2026) · Journal of Neural Engineering · doi:10.1088/1741-2552/ae9f95
“Peak detection is performed inside the forward pass, on the augmented signal, rather than as a pre-processing step; tokenization is therefore part of the model and is subject to the same augmentation as the rest of the input (section 2.5).”
InfluentialTell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
Zhang, Q., Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2311.02262
“…aimed to mechanistically understand attention scores (Zou et al., 2023), e.g. by studying them through feature importance (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Deb et al., 2023), probing (Conneau et al., 2018; Liu & Avci, 2019), visualization (Karpathy et al., 2015; Olah et al.,…”
BackgroundAI in Control: Rethinking Cybersecurity Compliance and Auditing
Loumachi, F. Y., Lacerda, M., Ouazzane, K., Adnane, A., & Adamyk, O. (2025) · doi:10.2139/ssrn.5731707
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.