Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 901–1,000 of 1,008
Interpreting Text Classifiers by Learning Context-sensitive Influence of Words
Kumar, S., Dixit, K., & Shah, K. (2021) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2021.trustnlp-1.7 · Cited by 4
“A global learning objective: Several existing approaches rely on some heuristic to come up with importance scores, such as gradients (Wallace et al., 2019), attentions (Wiegreffe and Pinter, 2019), or locally valid classifiers (Ribeiro et al., 2016) (see Atanasova et al. (2020) for a broad survey).”
BackgroundMethodSources of Understanding in Supervised Machine Learning Models
Pirozelli, P. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00524-4 · Cited by 4
“Second, attention and convolutional filters do not always provide straightforward understanding (Wiegreffe & Pinter, 2019).”
BackgroundDeep Dynamic Effective Connectivity Estimation from Multivariate Time Series
Mahmood, U., Fu, Z., Calhoun, V. D., & Plis, S. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892436 · Cited by 4
“These attributes address the questions regarding explainability of attention mentioned in [62], [63].”
Enhancing Attention’s Explanation Using Interpretable Tsetlin Machine
Yadav, R. K., & Nicolae, D. (2022) · Algorithms · doi:10.3390/a15050143 · Cited by 4
“Despite this, recent research has revealed that learned attention weights are frequently unrelated to the relevance of input components as judged by various explainability approaches [12], and that alternative attention distributions can provide identical predictions [13,14].”
BackgroundWhy Bother with Geometry? On the Relevance of Linear Decompositions of Transformer Embeddings
Mickus, T., & Vázquez, R. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.10 · Cited by 4
Yue, J., Wang, J., Zhang, S., Ma, Z., Shi, Y., & Lin, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106896 · Cited by 4
Integrating mechanics and machine learning for build-up rate prediction
Li, Z., Song, X., Yu, Q., Gong, N., Jiang, Z., Zhu, Z., & Zhang, C. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.213594 · Cited by 4
Navigating the informativeness-compression trade-off in XAI
Oldenburg, N., & Søgaard, A. (2025) · AI and Ethics · doi:10.1007/s43681-025-00733-5 · Cited by 4
“(iii) Attention maps [1, 111] provide us with saliency maps or dependency graphs.”
Motokawa, Y., & Sugawara, T. (2023) · Applied Sciences · doi:10.3390/app13148454 · Cited by 4
Offensive Text Detection Across Languages and Datasets Using Rule-based and Hybrid Methods
Gémes, K., Kovacs, A., & Recski, G. (2022) · doi:10.34726/3742 · Cited by 4
McInerney, D. J., Young, G., van de Meent, J.-W., & Wallace, B. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.238 · Cited by 4
Harnessing the richness of the linguistic signal in predicting pragmatic inferences.
Schuster, S., Chen, Y., & Degen, J. (2019) · Cited by 4
Attention Meets Normalization and Beyond
Ma, X., Guo, J., Chen, Q., Tang, S., Yang, Q. X., & Fu, S. (2020) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme46284.2020.9102909 · Cited by 4
“However, recent studies on attention’s interpretability [4, 5, 6] in NLP find that the attention weights are not closely related to fea-tures’ importance.”
BackgroundColoring the Black Box: What Synesthesia Tells Us about Character Embeddings
Kann, K., & Monsalve‐Mercado, M. M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.230 · Cited by 4
Sparse Attention with Linear Units
Zhang, B., Titov, I., & Sennrich, R. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-208886 · Cited by 4
Robust Complaint Processing in Portuguese
Cardoso, H. L., Osório, T. F., Barbosa, L. V., Rocha, G., Reis, L. P., Machado, J. P., & Oliveira, A. M. (2021) · Information · doi:10.3390/info12120525 · Cited by 4
Recent Development on Extractive Rationale for Model Interpretability: A Survey
Wang, H., & Dou, Y. (2022) · International Conference on Cloud Computing, Big Data and Internet of Things (3CBIT) · doi:10.1109/3cbit57391.2022.00080 · Cited by 4
Relation Extraction with ExplanationPreprint
Shahbazi, H., Fern, X. Z., Ghaeini, R., & Tadepalli, P. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14271 · Cited by 4
“However, recent studies (Ghaeini et al., 2018; Jain et al., 2019; Wiegreffe and Pinter, 2019) have questioned the validity of attention as a faithful explanation of model’s behavior.”
BackgroundWang, K., Jose Oramas, M., & Tuytelaars, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07954 · Cited by 4
“We use the attention as the explanation [29, 3].”
MethodUnderstanding Attention in Machine Reading ComprehensionPreprint
Cui, Y., Zhang, W., Che, W., Liu, T., & Chen, Z. (2021) · arXiv (Cornell University) · Cited by 4
Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Barba, J. Á. G. (2021) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/172245 · Cited by 4
SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free ModelsPreprint
Kumar, Y., Ilin, A., Salo, H., Kulathinal, S., Leinonen, M. K., & Marttinen, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13672 · Cited by 4
“As explained in [56], the GLU operation is applied by splitting the input tensor along the embedding dimension into two chunks, adding GELU activation on one of the chunks and combining them with a multiplicative gate.”
MethodCLEAR: Causal Explanations from Attention in Neural RecommendersPreprint
Nisimov, S., Rohekar, R. Y., Gurwicz, Y., Koren, G., & Novik, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10621 · Cited by 4
“In a paper contradicting Jain and Wallace [12], it was shown that explainability is task dependent [28].”
BackgroundRizzo, M., Conati, C., Jang, D., & Hu, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.07982 · Cited by 4
“Such evaluations concentrated on textual data [4; 2; 3] with only one investigation on visual data [18] and none involving video data.”
BackgroundMethodInfluentialAttention-likelihood relationship in transformersPreprint
Ruscio, V., Maiorca, V., & Silvestri, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.08288 · Cited by 4
“…as specialized attention heads and the informative nature of attention weights (Vig & Belinkov, 2019; Kovaleva et al., 2019; Clark et al., 2019), other studies inquire whether attention alone can fully explain the transformer’s output (Wiegreffe & Pinter, 2019; Jain & Wallace, 2019; Bibal et al.).”
BackgroundWhere to Look When Repairing Code? Comparing the Attention of Neural Models and DevelopersPreprint
Huber, D., Paltenghi, M., & Pradel, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07287 · Cited by 6
Explaining Deep Neural Networks by Leveraging Intrinsic MethodsPreprint
Rosa, B. L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12243 · Cited by 4
Validating Mechanistic Interpretations: An Axiomatic ApproachPreprint
Palumbo, N., Mangal, R., Wang, Z., Vijayakumar, S., Păsăreanu, C. S., & Jha, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.13594 · Cited by 4
T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLPPreprint
Li, R., Xiao, W., Wang, L., Jang, H., & Carenini, G. (2021) · arXiv (Cornell University) · doi:10.14288/1.0406310 · Cited by 4
HB-Eval: A System-Level Reliability Evaluation and Certification Framework for Agentic AIPreprint
Adam, A. M. I. (2025) · Preprints.org · doi:10.20944/preprints202512.2186.v1 · Cited by 4
TimeSliver : Symbolic-Linear Decomposition for Explainable Time Series ClassificationPreprint
Pandey, A., Mohapatra, P., Chen, W., Zhu, Q., & Keten, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.21289 · Cited by 4
Detecting Contextual Hallucinations in LLMs with Frequency-Aware AttentionPreprint
Qi, S., Chen, Y., Zhao, R., Zhu, Q., Hu, Z., Liu, W., He, Y., Yuan, Z., & Gui, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.18145 · Cited by 4
“Among these signals, attention is particularly informative, as it directly reflects how the model allocates content focus during generation (Wiegreffe & Pinter, 2019).”
Interpretable time series classification
Wang, Y. (2021) · theses.fr (ABES) · doi:10.70675/07ed6f76z6e51z4803zb2cez6cdac7921444 · Cited by 4
Vision-language models for chest radiography do not always need the imagePreprint
Lotfinia, M., Ziegelmayer, S., Adams, L. C., Nguyen, T.-T., Truhn, D., Maier, A., & Arasteh, S. T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.17710 · Cited by 4
“Post hoc saliency and attention maps [15,16,17] likewise describe where a model attends without establishing that its output causally depends on the image.”
Context-Aware Neural Machine Translation Models Analysis And Evaluation Through Attention
Dinarelli, M., Niaouri, D., Lopez, F., González-Sáez, G., Nakhlé, M., Esperança-Rodier, E., Rossi, C., Schwab, D., & Ballier, N. (2024) · HAL · doi:10.57896/2023-tal-64_3_3 · Cited by 3
“While some works question the explanatory power of attention, others acknowledge that it is one of the diverse explanation tools (Wiegreffe and Pinter, 2019).”
FRIC: a framework for feature importance explanations and evaluation in span-level NLP
Tuan-Nguyen, H.-D., & Le, B. H. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11628-1 · Cited by 3
“Explanation approaches are classified by various criteria, including the techniques used to generate explanations [13, 58] or the methods of presenting them [22, 41].”
Explainable Multi-Hop Question Answering: A Rationale-Based Approach
Han, K., Jang, Y., & Kim, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9110273 · Cited by 3
“…three axes: (i) causal sufficiency—the selected set alone can reproduce the output, in contrast to attention scores that need not be causal [23,25]; (ii) an information-bottleneck perspective—the pointer compresses inputs into minimal-sufficient evidence while preserving task-relevant meaning…”
Attention-based explainability for structure–property relationships
Slautin, B. N., Pratiush, U., Liu, Y., Funakubo, H., Shvartsman, V. V., Lupascu, D. C., & Kalinin, S. V. (2026) · Journal of Applied Physics · doi:10.1063/5.0314260 · Cited by 3
Integrating attention into explanation frameworks for language and vision transformers
Eggen, M., Lysnæs-Larsen, J., & Strümke, I. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12257-y · Cited by 3
“While attention weights do not directly determine output contributions, they are known to encode explanatory information [1, 5, 28, 29], and thus have unexplored potential when integrated into existing explainability approaches.”
Zhang, J., & Oh, S.-S. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01196-0 · Cited by 3
DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models
Bousselham, W., Boggust, A., Strobelt, H., & Kuehne, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37550-6_20 · Cited by 3
CROWN: Cross-attention reinforcement learning for O-RAN wireless networks
Monaco, D., Sacco, A., & Marchetto, G. (2026) · Computer Networks · doi:10.1016/j.comnet.2026.112276 · Cited by 3
“Such probabilities do not explain the reason for a given action [43,44], mainly because multiple linear layers act after the cross-attention to determine the neural network’s output.”
Pedersen, T. M., Øvlisen, K., Connelly, L. C., Assent, I., & Petersen, M. G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791457 · Cited by 3
“Research has explored techniques for XAI for LLMs [56, 63], such as visualizing attention mechanisms [30, 67, 70], leveraging uncertainty measures [6, 71, 72], and utilizing Chain-of-Thought prompting [65, 68].”
LLMs Decide Early and Explain Later: You Can Stop Them Early
Datta, A., Zhao, Z., Marreddy, M., Verma, B., Mamidi, R., & Mehler, A. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.2412 · Cited by 3
Wu, M.-L., & Liu, C.-W. (2026) · IEEE Access · doi:10.1109/access.2026.3691973 · Cited by 3
Hanafy, A. E., Hessane, A., & Farhaoui, Y. (2026) · Information Processing in Agriculture · doi:10.1016/j.inpa.2026.05.006 · Cited by 3
Onawole, A. T. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c01047 · Cited by 3
Wen, S., Wang, Y., & Qian, Y. H. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c01213 · Cited by 3
A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents
Wu, B., Li, Q., Zhou, C., Wang, T., & Ji, S. (2026) · ACM Computing Surveys · doi:10.1145/3837083 · Cited by 4
“Direct prompt injection places the adversarial instruction in the user-controlled input or immediate interaction context, whereas indirect prompt injection embeds it in external content that is later ingested during normal execution.”
Pavelski, L. M., & Braga, R. D. S. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3561164 · Cited by 3
“This step will apply advanced deep learning models based on transformers for NLP [8, 15], where multi-language support [6] and explainability [22] are active areas of ML research.”
BackgroundExplainable and Accurate Natural Language Understanding for Voice Assistants and Beyond
Gunaratna, K., Srinivasan, V., & Jin, H. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615277 · Cited by 3
“Wiegreffe et al. [19] showed that attentions can be used for model explanation purposes since attention mechanisms capture important details of the model decisions.”
BackgroundHow Does a Transformer Learn Compression? An Attention Study on Huffman and LZ4
Seo, B., & No, A. (2023) · IEEE Access · doi:10.1109/access.2023.3341512 · Cited by 3
“Yet, its genuine interpretative power remains contested [12], [13], [14], [15].”
BackgroundTowards Interpretability of Automatic Phoneme Analysis in Cleft Lip and Palate Speech
Baumann, I., Wagner, D., Schuster, M. E., Nöth, E., & Bocklet, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447632 · Cited by 3
“There are numerous debates [16, 17, 18] whether attention can be used to make decisions of a model interpretable.”
BackgroundNo Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models
Cheng, K., & Bhat, S. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.272 · Cited by 3
“Indeed, there is a long debate about the explanatory abilities of attention (Jain and Wal-lace, 2019; Wiegreffe and Pinter, 2019; Bibal et al., 2022), and whether researchers can learn anything about the inference process by examining attention weights and maps.”
BackgroundThe Optimal Choice of the Encoder–Decoder Model Components for Image Captioning
Bartosiewicz, M., & Iwanowski, M. (2024) · Information · doi:10.3390/info15080504 · Cited by 3
Anomaly Detection and Interpretation from Tabular Data Using Transformer Architecture
Homayouni, H., Aghayarzadeh, H., Ray, I., & Shirazi, H. M. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00091 · Cited by 3
Pasupuleti, M., & Satapathy, S. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02539-7 · Cited by 3
Huang, J., Parthasarathi, P., Rezagholizadeh, M., Chen, B., & Chandar, S. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.60 · Cited by 3
SentireCache: Accelerate Sentiment Classification With Saliency-Based Caching
Zhu, Y., Jia, J., Dong, M., & Qi, J. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3578574 · Cited by 3
“There has been some debate regarding the interpretability of the attention mechanism [42], with some studies suggesting that “Attention is Not Explanation” [43], while some others [44].”
Dual-Stream Time-Series Transformer-Based Encrypted Traffic Data Augmentation Framework
Choi, D., Kim, Y., Lee, C., & Sohn, K. (2025) · Applied Sciences · doi:10.3390/app15189879 · Cited by 3
Nare, B. (2025) · Advances in intelligent systems research/Advances in Intelligent Systems Research · doi:10.2991/978-94-6463-948-3_8 · Cited by 3
Contextualizing and Expanding Conversational Queries without Supervision
Krasakis, A. M., Yates, A., & Kanoulas, E. (2023) · ACM Transactions on Information Systems · doi:10.1145/3632622 · Cited by 3
“In fact, the value of attention weights for explainability is an on-going debate in the research community [7, 35, 71], while various methods have been introduced to measure the flow of information in the transformer through attention weights [1, 22].”
BackgroundMethodVargas‐Solar, G. (2024) · Electronics · doi:10.3390/electronics13183688 · Cited by 3
“…ensures equitable performance across different demographic groups [22]; • using attention mechanisms can enhance the transparency of NLP models [23]; • helping to pinpoint and correct biases in how models process data [24]; • adjusting model training to calibrate outputs to reduce bias [25]; •…”
Pruning Literals for Highly Efficient Explainability at Word Level
Yadav, R. K., Bhattarai, B., Jana, A., Jiao, L., & Yimam, S. M. (2024) · International Symposium on the Tsetlin Machine (ISTM) · doi:10.1109/istm62799.2024.10931519 · Cited by 3
“In addition, it is recently suggested in [6], [4], and [5] three separate ways for assessing the explainability of attention.”
Justifying Multi-label Text Classifications for Healthcare Applications
Figueira, J., Correia, G. M., Strzyz, M., & Mendes, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_30 · Cited by 3
Understanding stance classification of BERT models: an attention-based framework
Sáenz, C. A. C., & Becker, K. E. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01962-y · Cited by 3
“(a) Method : The second assessment involved the LOO technique, used to assess feature influence in classifications in related work [16, 17].”
On the Learning Dynamics of Attention Networks
Vashisht, R., & Ramaswamy, H. G. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230541 · Cited by 3
“[17] Sarah Wiegreffe and Yuval Pinter, ‘Attention is not not explanation’, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.”
BackgroundSentiment interpretability analysis on Chinese texts employing multi-task and knowledge base
Quan, X., Xie, X., & Liu, Y. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1104064 · Cited by 3
SDSK2BERT: Explore the Specific Depth with Specific Knowledge to Compress BERT
Ding, L., & Yang, Y. (2020) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/icbk50248.2020.00066 · Cited by 3
“However, [9] and [21] argue about whether attention could be the reason, making previous explanations untrustworthy.”
BackgroundHow does Attention Affect the Model?
Zhang, C., Li, Q., Hua, L., & Song, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.22 · Cited by 3
“Wiegreffe and Pinter (2019) and Vashishth et al. (2019) considered attention to be interpretable, using a more model-driven approach and manual verification.”
MethodHIVE: Hierarchical Information Visualization for Explainability
Juan, Y., Chiang, Y.-S., Liu, S.-C., Tsai, M.-F., & Wang, C. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/709 · Cited by 3
“Thus, to some extent, the attention mechanism is capable of identifying meaningful information for model prediction and in some ways alleviate the explainability problem [Wiegreffe and Pinter, 2019].”
BackgroundInterpretable sentiment analysis based on sentiment words’ syntax information
Zhao, Q., Zhang, H., & Shang, J. (2022) · International Conference on Industrial Automation, Robotics and Control Engineering (IARCE) · doi:10.1109/iarce57187.2022.00025 · Cited by 3
Unsupervised Extraction of Market Moving Events with Neural Attention.
Corro, L. D., & Hoffart, J. (2020) · San Andres Digital Repository (University of San Andrés) · Cited by 3
Assessing the Bilingual Knowledge Learned by Neural Machine Translation ModelsPreprint
He, S., Wang, X., Shi, S., Lyu, M. R.-T., & Tu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13270 · Cited by 3
“, 2019a) and attention (Voita et al., 2019b; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Li et al., 2018); and (2) behavior interpretability, which aims to explain particular behaviors of a NMT model, such as the input-output behavior (Alvarez-Melis and Jaakkola, 2017; Ding et al.”
BackgroundOn the Dynamics of Training Attention ModelsPreprint
Lu, H., Mao, Y., & Nayak, A. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10036 · Cited by 3
“One focus of these works pertains to understanding whether an attention mechanism can provide meaningful explanations (Voita et al., 2019; Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Serrano & Smith, 2020; Vashishth et al., 2020).”
BackgroundExplainable question answering beyond F1: metrics, models and human evaluation
Schuff, H. (2020) · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · doi:10.18419/opus-11224 · Cited by 3
Polarization and toxicity in political discourse online
Saveski, M. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 3
Explainability-aided Domain Generalization for Image ClassificationPreprint
Schmidt, R. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01742 · Cited by 3
“There has also been a discussion on whether attention counts as explanation and to which degree the process offers insights into the inner workings of a neural network [91, 165, 196].”
BackgroundYou Can Do Better! If You Elaborate the Reason When Making PredictionPreprint
Li, D., Tao, J., Chen, Q., & Hu, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14919 · Cited by 3
“However, it remains uncertain whether attention weights can provide reliable insights into the decision process of the model (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Pruthi et al., 2020b).”
Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection ModelsPreprint
Chandu, K. R., Sharma, P. K., Changpinyo, S., Thapliyal, A., & Soricut, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05175 · Cited by 3
“To mitigate this, researchers began to dissect intermediate components of models with the goal of interpretability to humans (Wiegreffe and Pinter, 2019; Thorne et al., 2019; Lipton, 2018) as opposed to implicit explanation (Xu et al.”
BackgroundUsing Self-Supervised Feature Extractors with Attention for Automatic COVID-19 Detection from SpeechPreprint
Mendonça, J., Solera-Ureña, R., Abad, A., & Trancoso, I. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00112 · Cited by 3
“Such conclusions have been refuted on the basis that existence does not entail exclusivity [15].”
BackgroundGENERALIZED OPTIMAL LINEAR ORDERSPreprint
Bommasani, R. (2020) · arXiv (Cornell University) · doi:10.7298/5x0j-me63 · Cited by 3
Reader comment analysis on online news platforms
Risch, J. (2021) · publish.UP (University of Potsdam) · doi:10.25932/publishup-48922 · Cited by 3
On-the-Fly Attention Modulation for Neural GenerationPreprint
Dong, Y., Bhagavatula, C., Lu, X., Hwang, J. D., Bosselut, A., Cheung, J. C. K., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00371 · Cited by 3
“Nevertheless, the inductive biases learned in attention patterns are often noisy and imperfect (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Kobayashi et al., 2020).”
BackgroundExplainability of Traditional and Deep Learning Models on Longitudinal Healthcare RecordsPreprint
Cheong, L. L., Meharizghi, T., Black, W., Guang, Y., & Meng, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12002 · Cited by 3
“This has very recently been called into question and is area of active debate, as evidenced by Jain and Wallace (2019)’s work titled ‘Attention is not Explanation’ and a responding paper by Wiegreffe and Pinter (2019) titled ‘Attention is not not Explanation’.”
BackgroundBenchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language TasksPreprint
Ambsdorf, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08174 · Cited by 3
Interpretable Sentence Representation with Variational Autoencoders and AttentionPreprint
Felhi, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02810 · Cited by 3
Combining Transformers with Natural Language ExplanationsPreprint
Ruggeri, F., Lippi, M., & Torroni, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4531749 · Cited by 3
New Faithfulness-Centric Interpretability Paradigms for Natural Language ProcessingPreprint
Madsen, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.17992 · Cited by 3
AttentionDep: Domain-Aware Attention for Explainable Depression Severity AssessmentPreprint
Ibrahimov, Y., Anwar, T., Yuan, T., Mutallimov, T., & Hasanov, E. E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00706 · Cited by 3
“Although the consensus is weak among scholars regarding whether attention weights represent feature importance, some studies have empirically validated this association [49].”
Baur, S., Benova, A., Cantú, E. D., & Ma, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.06558 · Cited by 3
“We assess vision trans-former attention maps for capabilities of ROI localization, with prior research acknowledging their explanatory potential [8] as well as pitfalls of qualitative heatmap evaluation [1,3].”
Gigiu, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.11590 · Cited by 3
Combining Causal Models for More Accurate Abstractions of Neural Networks
Pîslar, T.-M., Magliacane, S., & Geiger, A. (2025) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2503.11429 · Cited by 3
Tan, X. W., & Kok, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.23767 · Cited by 3
Analyzing Latent Concepts in Code Language ModelsPreprint
Sharma, A., Pungliya, V., Quinn, C. J., & Jannesari, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00476 · Cited by 3
“Attention-based methods are not considered reliable explanations [10, 29].”
Orlov, A. V., Makus, Y. V., Ashniev, G. A., Orlova, N. N., & Nikitin, P. I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.04.709491 · Cited by 3
“Subsequent work clarified conditions under which attention can be informative [43,44], but identified further complications: information mixing across layers undermines single-layer interpretation [45], and attention weights capture only part of token influence, as transformed vector norms also…”
The Dual-Stream Transformer: Channelized Architecture for Interpretable Language ModelingPreprint
Kerce, J. C., & Fox, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.07461 · Cited by 3
“A key question for interpretability is whether models learn discrete algorithms or rely on distributed soft mixing Jain and Wallace [2019], Wiegreffe and Pinter [2019].”
Mitigating Object Hallucinations in LVLMs via Attention Imbalance RectificationPreprint
Sun, H., Li, Q., Wang, P., & Zhang, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.24058 · Cited by 3
“By the erasure-based attribution method [1, 15, 50, 53], a technique that evaluates the importance of model components by removing them and observing changes in predictions, we erase individual attention heads to measure their impact on the probability of generating hallucinated tokens (more…”
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.