Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 701–800 of 1,008
Multi-Perspective Abstractive Answer SummarizationPreprint
Fabbri, A. R., Wu, X., Iyer, S. V., & Diab, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08536 · Cited by 9
“Aligning the source and generated output offers a potential interpretable output during inference, which goes beyond using attention for interpretation (Wiegreffe and Pinter, 2019).”
BackgroundTraining dynamics of neural language models
Saphra, N. (2021) · ERA · doi:10.7488/era/1421 · Cited by 9
Connecting Attributions and QA Model Behavior on Realistic CounterfactualsPreprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04515 · Cited by 9
“However, it avoids the pitfalls of directly inspecting attention (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019) by running an integrated gradients procedure over all the attention links within transformers, yielding attribution scores for each link.”
BackgroundSNVformer: An Attention-based Deep Neural Network for GWAS DataPreprint
Elmes, K., Benavides‐Prado, D., Tan, N. Ö., Nguyen, T. B., Sumpter, N. A., Leask, M., Witbrock, M., & Gavryushkin, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.07.499217 · Cited by 9
“A popular method for explaining predictions in Transformers is by analysing the values of the attention matrix, often by visualisation (Vig, 2019), since these reflect the task-dependent importance of each token (Wiegreffe & Pinter, 2019).”
MethodExplaining Hypergraph Neural Networks: From Local Explanations to Global ConceptsPreprint
SU, S. C. K., Duta, I., Magister, L. C., & Lió, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.07764 · Cited by 9
“However, there is ongoing debate about whether attention mechanisms offer valid explanations (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; ( Top ) Instance-level explanations are obtained by optimizing the subhypergraph structure using a loss function that incentivizes faithfulness (the…”
BackgroundHierarchical Attention Network for Interpretable ECG-based Heart Disease ClassificationPreprint
Rodriguez, M. P., & Nafea, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.03703 · Cited by 9
“Nevertheless, despite its popularity, the direct explainability of attention has been questioned by prior research [19], [22], [23].”
How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal ReasoningPreprint
Wu, H., Zhang, Y., & Zhou, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.15580 · Cited by 9
AgentWatcher: A Rule-based Prompt Injection MonitorPreprint
Wang, Y., Zou, W., Geng, R., & Jia, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.01194 · Cited by 10
Explainable AI: Context-aware layer-wise integrated gradients for explaining transformer models
Mersha, M. A., & Kalita, J. K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133050 · Cited by 8
“Furthermore, numerous studies have shown that attention weights alone are insu ffi cient for reliable explanations [12, 50], as they do not necessarily correlate with a model’s actual decision-making process.”
Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation
Sun, Y., Zhan, R., Cheang, C. S., Wu, H., Liu, X., Niu, Y., Ye, F., Lan, K., Chao, L. S., & Wong, D. F. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i39.40597 · Cited by 8
Noise-Filtering Enhanced Graph Transformer for Robust Fake News Detection
Zhu, J., Gao, C., Yin, Z., Li, X., Wang, Z., & Kurths, J. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3677544 · Cited by 8
Idaji, M. J., Hense, J., Neuhauser, T., Krause, A., Luo, Y., Eberle, O., Schnake, T., Ciernik, L., Jafari, F. R., Vahidimajd, R., et al. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104148 · Cited by 10
“The direct interpretability of attention scores is insufficient to faithfully reflect the model predictions [56, 57, 58].”
Kendiukhov, I. (2026) · BMC Genomics · doi:10.1186/s12864-026-12965-8 · Cited by 8
“These findings extend the attention-as-explanation debate from NLP [17, 19, 27] to biological foundation models.”
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 8
Reinforced GNNs for Multiple Instance Learning
Zhao, X., Dai, Q., Bai, X., Wu, J. X., Peng, H., Peng, H., Yu, Z., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3392575 · Cited by 8
“For instance, the attention-based pooling in RGMIL enables focusing on essential instances that lead to bag prediction results, but the explainability of its mechanism remains a subject of ongoing debate [57], [58].”
Kobayashi, K., Takamizawa, Y., Miyake, M., Ito, S., Gu, L., Nakatsuka, T., Akagi, Y., Harada, T., Kanemitsu, Y., & Hamamoto, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102929 · Cited by 8
Interpreting Deepcode, a Learned Feedback Code
Zhou, Y., Devroye, N., Turán, G., & Žefran, M. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619390 · Cited by 8
“The validity of explanations can be controversial as well [18], [19].”
BackgroundSkipPLUS: Skip the First Few Layers to Better Explain Vision Transformers
Mehri, F., Fayyaz, M., Baghshah, M. S., & Pilehvar, M. T. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00025 · Cited by 8
“Recent studies have questioned the reliability of this approach, given that self-attention is only a small part of a Transformer block [14, 34, 63, 72].”
On Explaining with Attention Matrices
Naim, O., & Asher, N. M. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240594 · Cited by 8
“[24, 45, 42] provide empirical evidence that at least in some NLP tasks, what appear to be numerically significant shifts in AW do not affect the predictions over a large number of instances in an NLP task.”
BackgroundMethodInfluentialSelf-Powered LLM Modality Expansion for Large Speech-Text Models
Yu, T., Liu, X., Hou, Z., Ding, L., Tao, D., & Zhang, M. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.690 · Cited by 8
Wang, X., & Liu, W. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110938 · Cited by 8
Explaining transformer-based next activity prediction by using attention scores
Käppel, M., Ackermann, L., Jablonski, S., & Härtl, S. (2025) · Process Science · doi:10.1007/s44311-025-00018-4 · Cited by 8
“This finding that the attention scores appear less reliable for very simple datasets is also consistent with the results in the study of Wiegreffe and Pinter (2019), where the same observation was made for datasets from other domains.”
InfluentialAutomated Trustworthiness Oracle Generation for Machine Learning Text Classifiers
Tung, L. N., Cho, S. B., Du, X., Neelofar, N., Terragni, V., Ruberto, S., & Aleti, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729376 · Cited by 8
“We derive trustwor-thiness datasets from existing studies, which do not directly label predictions as trustworthy or not, by measuring the explanations’ plausibility.”
Hong, C., & He, Q. (2025) · Applied Sciences · doi:10.3390/app15126893 · Cited by 9
Libovický, J., & Fraser, A. (2022) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2022.spnlp-1.6 · Cited by 8
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation
Liu, Y., Meng, F., Lin, Z., Wang, W., & Zhou, J. (2021) · doi:10.18653/v1/2021.acl-long.228 · Cited by 8
“However, prior work (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Brunner et al., 2020; Hao et al., 2020) finds that attention weights usually fail to correlate well with their contributions to the final prediction.”
BackgroundInterpreting BERT-based stance classification: a case study about the Brazilian COVID vaccination
Sáenz, C. A. C., & Becker, K. E. (2021) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2021.17867 · Cited by 8
“…a better and faster reception of users to these techniques in everyday life, works have discussed the value of attention weights to provide some level of interpretability to BERT-based models [Jain and Wallace 2019, Wiegreffe and Pinter 2019], with arguments in favor and against their contribution.”
BackgroundSupervised Attention in Sequence-to-Sequence Models for Speech Recognition
Yang, G.-P., & Tang, H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746310 · Cited by 8
“There is an ongoing debate whether attention can be interpreted as alignments [7, 6, 26].”
BackgroundFault Report Generation for ICT Systems by Jointly Learning Time-series and Text Data
Fukuda, N., Wu, C., Horiuchi, S., & Tayama, K. (2022) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms54207.2022.9789784 · Cited by 8
“It illustrates the names of the extracted metrics and the attention weights [26]–[28] of the metadata encoder of the proposed method when the stress-cpu is injected into cartservice .”
BackgroundFelhi, G., Roux, J. L., & Seddah, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.423 · Cited by 8
Attentional Extractive Summarization
González, J. Á., Segarra, E., García, F., Sanchis, E., & Hurtado, L.-F. (2023) · Applied Sciences · doi:10.3390/app13031458 · Cited by 8
Argumentative explanations for pattern-based text classifiers
Lertvittayakumjorn, P., & Toni, F. (2023) · Argument & Computation · doi:10.3233/aac-220004 · Cited by 8
Normal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction
Li, L., Chen, X., Bi, Z., Xie, X. Y., Deng, S., Zhang, N., Tan, C., Chen, M., & Chen, H. (2021) · International Joint Conference on Knowledge Graphs (IJCKG) · doi:10.1145/3502223.3502237 · Cited by 8
“Attention-based attribution [28] is not adopted as Bastings and Filippova [2] point out saliency methods are more suitable than attention mechanism in providing faithful explanations.”
Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models
Erliksson, K. F., Arpteg, A., Matskin, M., & Payberah, A. H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80599-9_8 · Cited by 8
“At the same time, deep neural networks (NNs) have achieved and even surpassed human performance in many tasks in natural language processing (NLP) and computer vision [15, 43], which has motivated a large body of research over the last few years focusing on making NN predictions more explainable.”
BackgroundInfluentialXRR: Explainable Risk Ranking for Financial Reports
Lin, T.-W., Sun, R.-Y., Chang, H., Wang, C., & Tsai, M.-F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86514-6_16 · Cited by 8
“To some extent, attention mechanisms alleviate the explainability problem [26], since attention layers explicitly weight the components’ representations; thus, it can be said that attention mechanisms are in some way capable of identifying meaningful information as a post-hoc justification of the model prediction.”
BackgroundAttExplainer: Explain Transformer via Attention by Reinforcement Learning
Niu, R., Wei, Z., Wang, Y., & Wang, Q. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/102 · Cited by 8
Do Deep Neural Nets Display Human-like Attention in Short Answer Scoring?
Zeng, Z., Li, X., Gašević, D., & Chen, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.14 · Cited by 8
“However, it remains disputable to use attention weights to measure the importance of input text (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Jain and Wallace, 2019).”
BackgroundTopics to Avoid: Demoting Latent Confounds in Text ClassificationPreprint
Kumar, S., Wintner, S., Smith, N. A., & Tsvetkov, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00453 · Cited by 8
“Recently, there has been a debate on whether attention can be used to explain model decisions (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), we thus present additional analysis of our proposed method based on saliency maps (Ding et al.”
MethodBrunila, M., Zhao, R., Mircea, A., Lumley, S., & Sieber, R. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11835 · Cited by 8
“…in neural networks, such as gradients, while Serrano and Smith (2019) showed that attention can “noisily” predict the importance of features for overall model performance and Wiegreffe and Pinter (2019) argued that attention can serve plausible, although not faithful explanations of models.”
BackgroundLocal Explanation of Dialogue Response GenerationPreprint
Tuan, Y.-L., Pryor, C., Chen, W., Getoor, L. C., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06528 · Cited by 8
“We compare our explanations LERG_L and LERG_S with attention [46], gradient [43], LIME [35] and Shapley value [42].”
ResultEntity Anchored ICD CodingPreprint
DeYoung, J., Shing, H.-C., Kong, L., Winestock, C., & Shivade, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.07444 · Cited by 8
“Additionally, interpretability is a secondary concern with most, and systems producing scores use derived metrics (e.g., model attention) now known to be misleading 27,28 for providing evidence for model results.”
BackgroundConsistent Multi-Granular Rationale Extraction for Explainable Multi-hop Fact VerificationPreprint
Si, J., Zhu, Y., & Zhou, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.09400 · Cited by 9
“While this method is incapable to guarantee the inattention of low-score features, drawing criticism recently (Wiegreffe and Pinter, 2019; Meister et al., 2021).”
MethodFaithful Interpretation for Graph Neural Networks
Hu, L., Huang, T., Yu, L., Lin, W., Zheng, T., & Wang, D. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2410.06950 · Cited by 8
“…is an essential property that explanation models should satisfy, which refers to the explanation should accurately reflect the true reasoning process of the model (Wiegreffe and Pinter 2019; Herman 2017; Jacovi and Goldberg 2020; Lyu, Apid-ianaki, and Callison-Burch 2022; Hu et al. 2024a,b,c).”
BackgroundMASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage SignalsPreprint
Liu, Y., Feng, H., Pu, J., & Chen, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.07509 · Cited by 8
“Since these weights indicate how later tokens attend to earlier ones, they can provide a signal for estimating the relevance between different events in the execution trace [19, 37].”
Guo, Z., Kamigaito, H., & Watanabe, T. (2024) · arXiv · doi:10.48550/arxiv.2406.12335 · Cited by 8
Emergent Structures and Training Dynamics in Large Language Models
Teehan, R., Clinciu, M., Serikov, O., Szczechla, E., Seelam, N., Mirkin, S., & Gokaslan, A. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.11 · Cited by 7
“However, these approaches have been criticized for their inability to produce meaningful and coherent interpretations (Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020; Serrano and Smith, 2019).”
BackgroundUnderstanding mental health discourse on Reddit with transformers and explainability
Rodríguez, I. S., Bianchi, J., Pinelli, F., Panizza, F., Ricciardi, E., Pietrini, P., & Petrocchi, M. (2026) · Scientific Reports · doi:10.1038/s41598-026-35918-3 · Cited by 7
SCOUT: Selective Coupling via Optimal Unbalanced Transport for Interpretable Text Classification
Jia, J., Zheng, H., Wu, Y., Chen, H., Wang, H., Bu, J., & Wu, L. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.290 · Cited by 7
Generalize Deep Neural Networks With Adaptive Regularization for Classifying
Guo, K., Tao, Z., Zhang, L., Hu, B., & Kui, X. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3264804 · Cited by 7
“Wiegreffe and Pinter [32] discovered that different models were able to provide different explanations of attention for the same output, and thus, the attention mechanism had limited applicability.”
BackgroundVisActive: Visual-concept-based Active Learning for Image Classification under Class Imbalance
Khaleel, M., Idris, A., Tavanapong, W., Pratt, J., Oh, J., & de Groen, P. C. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617999 · Cited by 7
“Our definition of faithfulness score is similar to those in [50,51] and represents whether the image patches closest to the visual concepts of the class are utilized in the class prediction.”
MethodTowards Understanding Neural Machine Translation with Attention Heads’ Importance
Zhou, Z., Zhu, J., & Li, W. (2024) · Applied Sciences · doi:10.3390/app14072798 · Cited by 7
Xie, S., Vosoughi, S., & Hassanpour, S. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.27 · Cited by 7
“The utility of attention for generating saliency explanations is widely recognized (Deng et al., 2017; Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Martins et al., 2020), notwithstanding initial doubts regarding the faithfulness and consistency of attention mechanisms (Serrano and Smith,…”
Luo, X., Wang, J. J., Yin, L., Zhao, K., Qian, K., Zhang, D., & Chen, K. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3491894 · Cited by 7
“In the field of natural language processing (NLP), there is a discussion about the feasibility of utilizing the attention mechanism to explain the degrees of importance that a model assigns to different segments or features within inputs [45].”
Fan, Y. B., Hu, Z., Li, R., Wang, Y. X., Zhang, G., Tan, H., & Liang, J. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121763 · Cited by 7
Ye, X., Zhang, Z., Lan, H., Zhang, C., Wang, L., Lin, J., Xu, X., Xu, Y., Du, L., & Tian, X. (2025) · Macromolecules · doi:10.1021/acs.macromol.5c00386 · Cited by 7
Comparison of explainability methods for hallucination analysis in LLMs
Papagiannopoulos, I., Koutalidis, H., Rempi, P., Ntanos, C., & Askounis, D. (2025) · Open Research Europe · doi:10.12688/openreseurope.20839.1 · Cited by 7
Interpret and Improve In-Context Learning via the Lens of Input-Label Mappings
Sun, C., Huang, Z., Zhang, Y., Lü, L., Li, H., Tian, X., Shen, X., & Ye, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.196 · Cited by 7
Tutorial on Human-Centered Explainability for Healthcare
Chakraborty, P., Kwon, B. C., Dey, S., Dhurandhar, A., Gruen, D. M., Ng, K., Sow, D., & Varshney, K. R. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3406470 · Cited by 7
DeepEAD: Explainable Anomaly Detection from System Logs
Wang, X., Kim, K. J., Wang, Y., Koike–Akino, T., & Parsons, K. (2023) · International Conference on Communications · doi:10.1109/icc45041.2023.10279563 · Cited by 7
“, hidden states, not each attribute token, whose explainability is controversial [16], [23].”
BackgroundGeneric Attention-model Explainability by Weighted Relevance Accumulation
Huang, Y., Jia, A., Zhang, X., & Zhang, J. (2023) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3595916.3626437 · Cited by 7
“Debates about whether attention can be explained [32] [39] have lasted for a long period.”
BackgroundFuzzy Neural Logic Reasoning for Robust Classification
Lin, G., & Zhang, Y. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3704728 · Cited by 7
“Moreover, reliance on attention weights alone has proven inadequate for comprehensive model explanation [69].”
Denoising Rationalization for Multi-hop Fact Verification via Multi-granular Explainer
Si, J., Zhu, Y., Lü, W., & Zhou, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.736 · Cited by 7
“While this method is incapable of guaranteeing the inattention of low-score features, drawing criticism recently (Wiegreffe and Pinter, 2019; Meister et al., 2021).”
Raieli, S., Jeanray, N., Gérart, S., Vachenc, S., & Altahhan, A. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00056-0 · Cited by 7
Zhao, L., Zhang, Y., & Zhang, C. (2022) · Aslib Journal of Information Management · doi:10.1108/ajim-12-2021-0385 · Cited by 7
Explaining video summarization based on the focus of attention
Apostolidis, E., Balaouras, G., Mezaris, V., & Patras, I. (2022) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism55400.2022.00029 · Cited by 7
“However, contrary to these approaches, we form explanations using several attention-based signals used in the NLP domain [11]–[16], and we examine the network’s input-output based on various replacement functions.”
MethodExplainable Video Summarization for Advancing Media Content Production
Apostolidis, E., Balaouras, G., Patras, I., & Mezaris, V. (2023) · Advances in information quality and management · doi:10.4018/978-1-6684-7366-5.ch065 · Cited by 7
Semantic Interpretation and Validation of Graph Attention-Based Explanations for GNN Models
Panagiotaki, E., Martini, D. D., & Kunze, L. (2023) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar58858.2023.10406370 · Cited by 7
“[ ˆ R | ˆ τ ] and ground-truth values [ R | τ ] , respectively.”
MethodMulti-Layer Attention-Based Explainability via Transformers for Tabular Data
Gavito, A. T., Klabjan, D., & Utke, J. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00010 · Cited by 7
The Reverse Turing Test for Evaluating Interpretability Methods on Unknown Tasks
González, A. V., & Søgaard, A. (2020) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 7
Explainability Via Causal Self-Talk
Roy, N., Kim, J., & Rabinowitz, N. C. (2022) · neural information processing systems · doi:10.52202/068431-0556 · Cited by 7
RECAST: Interactive Auditing of Automatic Toxicity Detection Models
Wright, A. P., Shaikh, O., Park, H., Epperson, W., Ahmed, M., Pinel, S., Yang, D., & Chau, D. H. (2020) · doi:10.1145/3403676.3403691 · Cited by 7
“tains 97% of the original model’s understanding capacity with substantially fewer parameters [14]. Furthermore, transformer models utilize self-attention, which can be used as a proxy for explanation [18]. Transformer models also utilize context through their use of self-attention, which makes these models far more ef…”
MethodExplainable Prediction of Adverse Outcomes Using Clinical NotesPreprint
Lovelace, J., Hurley, N. C., Haimovich, A. D., & Mortazavi, B. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14095 · Cited by 7
Lovelace, J., Hurley, N. C., Haimovich, A. D., & Mortazavi, B. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01197 · Cited by 7
“However, Wiegreffe and Pinter (2019) explored some of the problems with their underlying assumptions and experimental setup and demonstrated that their experiment failed to fully explore their premise, and thus failed to support their claim.”
BackgroundPredicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream TransformerPreprint
Fouladvand, S., Talbert, J., Dwoskin, L. P., Bush, H., Meadows, A. L., Peterson, L. E., Kavuluru, R., & Chen, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08800 · Cited by 7
“While it is still unclear whether attentions can be used to explain deep learning models 43,44 , attention weights have been used extensively to assess feature importance 24,42 .”
BackgroundValentino, M., Pratt-Hartmann, I. E., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01974 · Cited by 7
“Previous work on explainability evaluation has mainly focused on methods for assessing the quality and faithfulness of explanations generated by deep learning models (Camburu et al., 2020; Subramanian et al., 2020; Kumar and Talukdar, 2020; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundDuTrust: A Sentiment Analysis Dataset for Trustworthiness EvaluationPreprint
Wang, L., Liu, H., Peng, S., Tang, H., Xiao, X., Chen, Y., Wu, H. R., & Wang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13140 · Cited by 7
“When comparing different interpretation methods, we observe that ATT method outperforms other interpretation methods across all evaluated models, which also confirms the viewpoint in Wiegreffe and Pinter [2019].”
BackgroundMethodA Comparison of Code Embeddings and BeyondPreprint
Han, S., Wang, D., Li, W., & Lu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07173 · Cited by 7
“We attempt to answer the question by investigating what elements in the source code are mostly relevant to the model prediction, which is a widely studied problem in the field of explainable deep learning in recent years [28, 51, 69].”
BackgroundMethodDual Decomposition of Convex Optimization Layers for Consistent Attention in Medical ImagesPreprint
Ron, T., Weiler‐Sagie, M., & Hazan, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.02761 · Cited by 7
“In return, Wiegreffe & Pinter (2019) refined the discussion and considered two types of explanations: plausible and faithful.”
BackgroundZeldes, A., & Liu, Y. (2022) · Dialogue & Discourse · doi:10.5087/dad · Cited by 7
“We thank an anonymous reviewer for pointing out Wiegreffe and Pinter (2019), who offer experimental support for the variable utility of attention as an explanatory metric.”
MethodOn the Interpretability of Attention NetworksPreprint
Pandey, L. N., Vashisht, R., & Ramaswamy, H. G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.14776 · Cited by 7
Emergent Communication with AttentionPreprint
Ri, R., Ueda, R., & Naradowsky, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10920 · Cited by 7
“…may not always align with human intuitions (Alkhouli, Bretschner, & Ney, 2018; F. Liu et al., 2020) and there is an ongoing debate about the extent to which the attention weights can be used as explanations for model predictions (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Bibal et al., 2022).”
BackgroundGao, Y., Zhou, H.-Y., Wang, X., Zhang, T., Beets‐Tan, R. G. H., Han, L., Estacio, L., D'Angelo, A. R., Teuwen, J., Mann, R. M., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2924701/v1 · Cited by 7
Evaluating Long Range Dependency Handling in Code Generation LLMs
Assogba, Y., & Ren, D. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2407.21049 · Cited by 7
SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language ModelsPreprint
Kim, G., Thakur, S. S., Park, S. M., Wei, W., & Bao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.15021 · Cited by 7
“Identifying these important tokens allows the models to focus on the most relevant information [12, 13, 14].”
Towards Understanding Graphical Perception in Large Multimodal ModelsPreprint
Zhang, K., Yang, J.-W., Inala, J., Singh, C., Gao, J.-F., Su, Y., & Wang, C.-L. (2025) · arXiv.org · doi:10.48550/arxiv.2503.10857 · Cited by 6
Bi-Orthogonal Factor Decomposition for Vision TransformersPreprint
Doshi, F., Fel, T., Konkle, T., & Alvarez, G. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.05328 · Cited by 7
“However, these maps reflect only the surface structure of interactions: they indicate the amount of attention exchanged, not the content of that exchange or the reasons for it [2, 7, 84].”
Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejection
Kang, M., & Kim, J. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2603.06745 · Cited by 7
“The attention mechanism has long been a focal point for interpretability research (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Elhage et al., 2021), but its direct application to inference time steering remains a relatively under-explored area.”
Deep learning for Internet of Things (IoT) network security
Shahid, M. R. (2021) · theses.fr (ABES) · doi:10.70675/74fe10adz3ca2z463dz92f9zfd15d2f94f36 · Cited by 7
Neural Approaches to Historical Words Reconstruction
Fourrier, C. (2022) · theses.fr (ABES) · doi:10.70675/647b8a58zeea0z48cdz8229zf1d7ee8b0742 · Cited by 7
Large Vision-Language Models Get Lost in AttentionPreprint
Xi, G., Tian, Y., Yang, M., Yi, H., Lin, L., Hao, X., Wang, K., & Wang, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.05668 · Cited by 7
“For attention, foundational studies show that raw attention weights can be an unreliable attribution signal (Jain & Wallace, 2019; Serrano & Smith, 2019; Wiegreffe & Pinter, 2019).”
Mohammadi, S., Raff, E., Malekar, J., Palit, V., Ferraro, F., & Gaur, M. (2024) · Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2024.blackboxnlp-1.23 · Cited by 6
CGS-Mask: Making Time Series Predictions Intuitive for All
Lu, F., Li, W., Sun, Y., Song, C., Ren, Y., & Zomaya, A. Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29325 · Cited by 6
BoostedDim attention: A novel data-driven approach to improving LiDAR-based lane detection
Patil, O., Nair, B. B., Soni, R., Thayyilravi, A., & Manoj, C. R. (2024) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2024.102887 · Cited by 6
Salmi, S., Mérelle, S. Y. M., Gilissen, R., van der Mei, R., & Bhulai, S. (2024) · JMIR Mental Health · doi:10.2196/57362 · Cited by 6
“Jain and Wallace [16] argue attention weights do not provide explanations, while Wiegreffe and Pinter [17] in turn challenged their claims.”
BackgroundComparative Study of Explainability Methods for Legal Outcome Prediction
Staliūnaitė, I., Valvoda, J., & Satoh, K. (2024) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2024.nllp-1.20 · Cited by 6
“Researchers have compared the faithfulness and cost of various interpretability methods (Lip-ton, 2018; Wiegreffe and Pinter, 2019; Jain and Wallace, 2019; Wang et al., 2020).”
Zhai, W., Zubiaga, A., Liu, B., Sun, C., & Zhao, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1052 · Cited by 6
“Besides, this process does not guarantee that the extracted explanations will be faithful to the reasoning of the model (Wiegreffe and Pinter, 2019), leading to what we call spurious explanations(Zhao et al., 2023).”
BackgroundMethodeXING-IoT conceptual framework for explainability integration in next generation-IoT
Vultureanu‐Albişi, A., Bădică, C., & Ivanović, M. D. (2025) · Connection Science · doi:10.1080/09540091.2025.2507180 · Cited by 6
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.