Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 601–700 of 1,008
Towards interpretability in fingerprint based indoor positioning: May attention be with us
Brunello, A., Montanari, A., & Saccomanno, N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120679 · Cited by 12
Briakou, E., Goyal, N., & Carpuat, M. J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.690 · Cited by 12
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
Gao, C., Huang, S., Li, J., & Chen, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.868 · Cited by 12
Plausible Extractive Rationalization through Semi-Supervised Entailment Signal
Jie, Y. W., Satapathy, R., & Cambria, E. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.307 · Cited by 12
“In contrast, proponents suggest that the relevance of attention weights varies with the definition of faithfulness and that multiple weight combinations can yield the same output (Wiegreffe and Pinter, 2019).”
BackgroundAttention uncovers task-relevant semantics in emotional narrative understanding
Nguyen, T.-S., Wu, Z., & Ong, D. C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107162 · Cited by 12
“Recently, Wiegreffe and Pinter [69] also presented issues with analyzing explanabilities of attention weights using “counterfactual” weights, and argued that such a claim was ambiguous in its definition of explanation and may only work in certain parts of the neural network.”
BackgroundKW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification
Jang, H., Bang, S., Xiao, W., Carenini, G., Ng, R. T., & Lee, Y. J. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.10 · Cited by 12
“…ide as a(n) application, php as a(n) free_software” KW replacement “object-oriented_programming_language, application, free_software, swing, best, looking” We assume that attention can be used for prediction explanations based on (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019).”
MethodSupporting Complaints Investigation for Nursing and Midwifery Regulatory Agencies
Lertvittayakumjorn, P., Petej, I., Gao, Y., Krishnamurthy, Y., van der Gaag, A., Jago, R., & Stathis, K. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.10 · Cited by 12
“Our method provides explanations in all the aforementioned forms except attention distributions, as it remains unclear whether attention distributions can be reliably used as explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
MethodCorpus-level and Concept-based Explanations for Interpretable Document Classification
Shi, T. Z., Zhang, X., Wang, P., & Reddy, C. K. A. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3477539 · Cited by 12
“For example, Jain andWallace [20], Serrano and Smith [38] focused on studying if attention can be used to interpret a model, however, there are still problems in their experimental designs [48].”
BackgroundTü-CL at SIGMORPHON 2023: Straight-Through Gradient Estimation for Hard Attention
Girrbach, L. (2023) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2023.sigmorphon-1.19 · Cited by 12
Tü-CL at SIGMORPHON 2023: Straight-Through Gradient Estimation for Hard Attention
Girrbach, L. (2023) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2023.sigmorphon-1.17 · Cited by 12
“Modifying the attention mechanism is necessary, because for soft attention, which does not realise hard alignment decisions, the relation of model outputs and attention weights is not fully understood (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundLai, V., Cai, J. Z., & Tan, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08534 · Cited by 12
Unboxing Engagement in YouTube Influencer Videos: An Attention-Based ApproachPreprint
Rajaram, P., & Manchanda, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12311 · Cited by 12
“However, some of these correlations may potentially be spurious due to peculiarities (artifacts) in the design of deep learning models (Vashishth et al., 2019; Wiegreffe & Pinter, 2019).”
Exploring Self-Attention for Crop-type Classification ExplainabilityPreprint
Obadić, I., Roscher, R., Oliveira, D. A. B., & Zhu, X. X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.13167 · Cited by 12
“Conversely, the interpretation of the attention weights is always conditional to the classes in the dataset and focuses on events that help to disambiguate them.”
Breaking through the learning plateaus of in-context learning in TransformerPreprint
Fu, J., Yang, T., Wang, Y., Lü, Y., & Zheng, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.06054 · Cited by 12
“…the ReLU activation function in the feed-forward layers can achieve results comparable to the original model (Press et al., 2019), and 2) the softmax operation may not be essential for the functioning of the Transformer (Wiegreffe & Pinter, 2019; Brunner et al., 2019; Richter & Wattenhofer, 2020).”
ResultQuantifying the Plausibility of Context Reliance in Neural Machine TranslationPreprint
Sarti, G., Chrupała, G., Nissim, M., & Bisazza, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01188 · Cited by 12
“It is important to note that plausibility does not imply faithfulness , i.e. how accurately the rationale reflects the true reasoning process of the model (Wiegreffe & Pinter, 2019), since a good explanation of model behavior might not align with human intuition.”
BackgroundViT-ReciproCAM: Gradient and Attention-Free Visual Explanations for Vision TransformerPreprint
Byun, S.-Y., & Lee, W.-J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02588 · Cited by 12
Do Language Models Learn Semantics of Code? A Case Study in Vulnerability DetectionPreprint
Steenhoek, B. J., Rahman, M. M., Sharmin, S., & Le, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04109 · Cited by 12
Graph-Based Explainable AI: A Comprehensive SurveyPreprint
Bugueño, M., Biswas, R., & de Melo, G. (2024) · HAL · Cited by 12
POINT$^{2}$: A Polymer Informatics Training and Testing DatabasePreprint
Xu, J., Liu, G., Guo, R., Jiang, M., & Luo, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.23491 · Cited by 12
“Model-intrinsic methods, like linear models, tree-based models [70, 71], attention mechanisms [72–76], and rationalization [12, 77–79], offer interpretability directly embedded within the model’s structure, facilitating a more integrated understanding of prediction processes.”
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention HeadsPreprint
Wang, J., Wang, C., Fang, Y., Hu, H., Du, J., Li, A., Rong, S., Wu, H., Xu, J., Liu, L., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.17237 · Cited by 12
“The premise that attention weights encode relevance is empirically supported (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019), and mechanistic interpretability has revealed functional specialization among attention heads—with only a sparse subset being critical for specific tasks (Wu et al.,…”
Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability
Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.04709 · Cited by 11
“Informally, faithfulness has been defined as the degree to which an explanation accurately represents the ‘true reasoning process behind a model’s behavior’ (Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020; Lyu et al., 2022; Chan et al., 2022a).”
BackgroundMulti-view graph-based interview representation to improve depression level estimation
Agarwal, N., Dias, G. H., & Dollfus, S. (2024) · Brain Informatics · doi:10.1186/s40708-024-00227-w · Cited by 11
“Attention scores are widely used as a tool to justify model predictions, however validity of these explanations is debatable [33, 34].”
MethodTransformer Neural Networks for Behavior-Centric Production Forecasting in Unconventional Reservoir
Cornelio, J., Razak, S. M., Cho, Y., Liu, H., Vaidya, R. N., & Jafarpour, B. (2025) · SPE Journal · doi:10.2118/212953-pa · Cited by 12
Computational methods for binding site prediction on macromolecules
Kozlovskii, I., & Popov, P. A. (2025) · Quarterly Reviews of Biophysics · doi:10.1017/s003358352500006x · Cited by 12
XueXin, C., Cai, R., Huang, Z., Li, Z., Zheng, J., & Wu, M. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf142 · Cited by 11
“• Fidelity : the Infidelity of attention-based methods is relatively low, possibly due to the inherent instability and high-frequency biases of attention mechanisms [6, 7].”
Discursive Circuits: How Do Language Models Understand Discourse Relations?
Miao, Y., & Kan, M. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1657 · Cited by 11
Chen, C. L. P., Xia, H., Zhou, W., Xia, C., Liu, M., Hao, R., & Wang, T. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07102-7 · Cited by 11
“In contrast, Wiegreffe and Pinter [ 59] maintain that ”attention is not not explanation” , emphasizing that Table 8 Scoring criteria for human evaluation experiment Score Description 5 The explanation is highly consistent with human-intuitive toxic cues; the identified features and their ranking…”
Lin, T., Lau, R. Y. K., & Hu, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11565-y · Cited by 11
“Nevertheless, when properly validated, attention can serve as a meaningful explana - tory signal (Wiegreffe and Pinter 2019 ).”
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
Sun, X., Wu, D., Qin, S., Echizen, I., Ali, A. E., & Sugawara, S. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1495 · Cited by 11
Zhai, S., Tan, Y., Zhang, C., Hipolito, C. J., Song, L., Zhu, C., Zhang, Y., Duan, H., & Yin, Y. (2023) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.3c00627 · Cited by 11
Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling
Gunaratna, K., Srinivasan, V., Yerukola, A., & Jin, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.245 · Cited by 11
Music performance style transfer for learning expressive musical performance
Xiao, Z., Chen, X., & Zhou, L. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02788-5 · Cited by 11
“In particular, the attention mechanism [8] and transformer architecture greatly promote the artistic level of music generation.”
Dai, L., Yang, X., Li, H., Zhao, X., Lin, L., Jiang, Y., Wang, Y., Li, Z., & Shen, H. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102772 · Cited by 11
Explainable Authorship Identification in Cultural Heritage Applications
Setzu, M., Corbara, S., Monreale, A., Moreo, A., & Sebastiani, F. (2024) · Journal on Computing and Cultural Heritage · doi:10.1145/3654675 · Cited by 11
“…of the Epistle with different colours based on the classification scores obtained when important role in the prediction. the user the burden of understanding why the system has reached exactly that decision. consists of ranking the features used by the classifier by their importance (this is…”
Anisah, G., & Amreta, M. Y. (2024) · KEMBARA Journal of Scientific Language Literature and Teaching · doi:10.22219/kembara.v10i1.32974 · Cited by 11
Redundancy and Concept Analysis for Code-trained Language Models
Sharma, A., Hu, Z., Quinn, C., & Jannesari, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694931 · Cited by 11
“Attention-based code perturbation techniques are useful when extracting hard-coded features is not a viable option for dynamically typed languages or arbitrary code-snippets.”
AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning
Yang, T., Deng, J., Quan, X., Wang, Q., & Nie, S. (2022) · neural information processing systems · doi:10.52202/068431-0894 · Cited by 11
Deep Attentive End-to-End Continuous Breath Sensing from Speech
MacIntyre, A. D., Rizos, G., Batliner, A. M., Baird, A., Amiriparian, S., Hamilton, A. F. D. C., & Schuller, B. W. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2832 · Cited by 11
“More recently, the authors of [29] have addressed the criticisms by claiming that the existence of alternative explanations is not indicative of lack of explanatory power, as there may be multiple explanations for the predictions of a model, something also indicated by the success of models that utilise multi-head sel…”
BackgroundMethodNguyen, T. H., & Rudra, K. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583861 · Cited by 11
“Many previous works [14, 33, 42] have argued that attention is not explanation.”
BackgroundLearning to Select Prototypical Parts for Interpretable Sequential Data Modeling
Zhang, Y., Gao, N., & Ma, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25812 · Cited by 11
RORA: Robust Free-Text Rationale Evaluation
Jiang, Z., Lu, Y., Chen, H., Khashabi, D., Durme, B. V., & Liu, A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.60 · Cited by 11
Offensive text detection on English Twitter with deep learning models and rule-based systems
Gémes, K., Kovacs, A., Reichel, M., & Recski, G. (2021) · reposiTUm (TU Wien) · doi:10.34726/4342 · Cited by 11
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
A survey on bias in machine learning researchPreprint
Mikołajczyk, A., & Grochowski, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.11254 · Cited by 12
“In 2019, Jain et al. [49] explained why they think that ”Attention is not an explanation”, whereas the authors of [116] claimed that Attention is not not an explanation .”
Improving Interpretation Faithfulness for Vision TransformersPreprint
Hu, L., Liu, Y., Liu, N., Huai, M., Sun, L., & Wang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.17983 · Cited by 11
“Faithfulness is a crucial property that explanation models should satisfy, ensuring that the explanation accurately reflects the true reasoning process of the model (Wiegreffe & Pinter, 2019; Herman, 2017; Jacovi & Goldberg, 2020; Lyu et al., 2022).”
BackgroundHindsight PRIORs for Reward Learning from Human PreferencesPreprint
Verma, M., & Metcalf, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.08828 · Cited by 11
“Some popular methods include gradient / saliency based approaches (Greydanus et al., 2018; Selvaraju et al., 2017; Simonyan et al., 2013; Weitkamp et al., 2019) and self-attention based methods Ras et al. (2022); Wiegreffe & Pinter (2019); Vashishth et al. (2019).”
MethodSRViT: Vision Transformers for Estimating Radar Reflectivity from Satellite Observations at ScalePreprint
Stock, J., Hilburn, K., Ebert‐Uphoff, I., & Anderson, C. W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.16955 · Cited by 11
“…the interactions between all tokens, but faithful interpretations are challenged by not considering (a) the token vector magnitudes and high dimensionality and (b) the interplay between tokens across intermediate hidden layers (Brunner et al., 2019; Wiegreffe & Pinter, 2019; Serrano & Smith, 2019).”
BackgroundOn-the-Fly Attention Modulation for Neural Generation
Dong, Y., Bhagavatula, C., Lu, X., Hwang, J. D., Bosselut, A., Cheung, J. C. K., & Choi, Y. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.107 · Cited by 10
“Other works observed that attention patterns can provide explanations (Wiegreffe and Pinter, 2019) or coarse word alignments in machine translation (Zenkel et al.”
BackgroundLabadie, R., Slijepčević, D., Chen, X., Böck, A. J., Babic, A., Freimann, L., Atzmüller, C., & Zeppelzauer, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104309 · Cited by 10
There Is More to Attention: Statistical Filtering Enhances Explanations in Vision Transformers
Ayyar, M. P., Benois‐Pineau, J., & Zemmari, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31666-0_39 · Cited by 10
Rethinking explainability in the era of multimodal AI
Agarwal, C. (2026) · AI Magazine · doi:10.1002/aaai.70094 · Cited by 10
A Deep Learning Framework for Forecasting Medium‐Term Covariance in Multiasset Portfolios
Reis, P. M. N., Serra, A. P., & da Gama, J. M. P. (2026) · Journal of Forecasting · doi:10.1002/for.70110 · Cited by 10
Locally Aggregated Feature Attribution on Natural Language Model Understanding
Zhang, S., Wang, J., Jiang, H., & Song, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.159 · Cited by 10
Can Deep Learning Models Learn the Vulnerable Patterns for Vulnerability Detection?
Yan, G., Chen, S., Bail, Y., & Li, X. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00142 · Cited by 10
“However, Wiegreffe et al. [35] suggest that the attention mechanism is meaningful especially when it works in coordination with the entire DL model.”
BackgroundAttention-Based Interpretable Regression of Gene Expression in Histology
Graziani, M., Marini, N., Deutschmann, N., Janakarajan, N., Müller, H., & Martínez, M. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17976-1_5 · Cited by 10
Explainability of Speech Recognition Transformers via Gradient-Based Attention Visualization
Sun, T., Chen, H., Hu, G., He, L., & Zhao, C. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3282488 · Cited by 10
“Although there are some controversy researches debating on the validity of interpreting Transformer prediction with attention [8], [9], the plausible and meaningful interpretations offered by attention visualization have shown a great opportunity to understand Trans-formers [10], [11], [12].”
BackgroundREFER: An End-to-end Rationale Extraction Framework for Explanation Regularization
Madani, M. R. G., & Minervini, P. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.40 · Cited by 11
Waveformer for modeling dynamical systems
Navaneeth, N., & Chakraborty, S. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.111253 · Cited by 10
Enhancing Explainability in Mobility Data Science Through a Combination of Methods
Makridis, G., Koukos, V., Fatouros, G., Separdani, M. M., & Kyriazis, D. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-62269-4_4 · Cited by 10
Attention Please: What Transformer Models Really Learn for Process Prediction
Käppel, M., Ackermann, L., Jablonski, S., & Härtl, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70396-6_12 · Cited by 10
Focal Modulation Networks for Interpretable Sound Classification
Libera, L. D., Subakan, C., & Ravanelli, M. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10669912 · Cited by 10
Shin, Y.-M., Li, S., Cao, X., & Shin, W.-Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i19.34254 · Cited by 10
“There is even a significant body of research debating the validity of self-attention as explanations in natural language processing (NLP) (Jain and Wallace 2019; Wiegreffe and Pinter 2019; Bibal et al. 2022).”
Detecting Conceptual Abstraction in LLMs
Regneri, M., Abdelhalim, A., & Laue, S. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/47b34gfttpbe · Cited by 10
Attention-Based Explanation in a Deep Learning Model For Classifying Radiology Reports
Putelli, L., Gerevini, A., Lavelli, A., Maroldi, R., & Serina, I. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77211-6_42 · Cited by 10
On the Difficulty of Segmenting Words with Attention
Sanabria, R., Tang, H., & Goldwater, S. J. (2021) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2021.insights-1.11 · Cited by 10
Visualizing and Explaining Language Models
Brașoveanu, A., & Andonie, R. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-93119-3_8 · Cited by 10
Is Attention Interpretation? A Quantitative Assessment on Sets
Haab, J., Deutschmann, N., & Martínez, M. R. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23618-1_21 · Cited by 10
A Fine Line Between Irony and Sincerity: Identifying Bias in Transformer Models for Irony Detection
Maladry, A., Lefever, E., Hee, C. V., & Hoste, V. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.28 · Cited by 10
“Though popular, the use of attention weights as an explanation has been the topic of heavy discussion, with researchers opposing (Jain and Wallace, 2019) and supporting (Wiegreffe and Pinter, 2019) their reliability.”
BackgroundCan Embeddings Analysis Explain Large Language Model Ranking?
Lucchese, C., Minello, G., Nardini, F. M., Orlando, S., Perego, R., & Veneri, A. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615225 · Cited by 10
“There are works claiming that attention is not really useful to be used as an explanation [5, 16], while others claim that it might be helpful in some scenarios [21].”
BackgroundExplainability of Text Processing and Retrieval Methods: A Survey
Saha, S., Majumdar, D., & Mitra, M. (2026) · ACM Computing Surveys · doi:10.1145/3801957 · Cited by 10
“Further, Wiegreffe and Pinter (2019) [196] showed that attention weights frozen to a uniform distribution performed comparably to original weights on some datasets, suggesting attention is not essential for these tasks.”
ForceReader: a BERT-based Interactive Machine Reading Comprehension Model with Attention Separation
Chen, Z., & Wu, K. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.241 · Cited by 10
“Despite all the arguments about the interpretability of attention(Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019), we still start from the visual distribution of attention, propose an interactive reading comprehension model based on the idea of separate representation of the attention.”
Why Attentions May Not Be Interpretable?Preprint
Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05656 · Cited by 10
“Many researchers provide evidence to support or refute the interpretability of the attention mechanisms, and a debate on the effectiveness of attention-based interpretations has raised [12, 27, 37].”
BackgroundEvent Detection on Dynamic GraphsPreprint
Kosan, M., Silva, A., Medya, S., Uzzi, B., & Singh, A. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12148 · Cited by 10
“We notice that the use of attention weights for interpretability is a contentious topic in the literature (Jain and Wallace 2019; Pruthi et al. 2019; Wiegreffe and Pinter 2019).”
A Novel Counterfactual Data Augmentation Method for Aspect-Based Sentiment AnalysisPreprint
Wu, D., Wen, L., Chen, C., & Shi, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.11260 · Cited by 10
“To argue this, Wiegreffe and Pinter (2019) believed that attention can be used as explanation in some scenarios, and verified through experiments.”
BackgroundFaithfulness Measurable Masked Language ModelsPreprint
Madsen, A., Reddy, S., & Chandar, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07819 · Cited by 10
“Note, we do not cover the faithfulness metrics that are specific to attention (Moradi et al., 2021; Wiegreffe and Pinter, 2019; Vashishth et al., 2019), as this paper presents a general approach.”
BackgroundModel-Editing-Based Jailbreak against Safety-aligned Large Language ModelsPreprint
Li, Y., Zhang, Z., Wang, K., Shi, L., & Wang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.08201 · Cited by 10
LiDDA: Data Driven Attribution at LinkedInPreprint
Bencina, J., Aykutlug, E., Chen, Y., Zhang, Z., Sorenson, S., Tang, S., & Wei, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.09861 · Cited by 10
“The topic of attention interpretability has been the subject of extensive research [12, 22].”
Zhang, Z., Wang, Y., Huang, X., Fang, T., Zhang, H., Deng, C., Li, S., & Yu, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.485 · Cited by 9
A counterfactual explanation method based on modified group influence function for recommendation
Guo, Y., Cai, F., Pan, Z., Shao, T., Chen, H., & Zhang, X. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01547-4 · Cited by 9
“However, obtaining such information in practical scenarios is not only challenging but controversial [13, 21].”
Toward Faithful Neural Network Intrinsic Interpretation With Shapley Additive Self-Attribution
Sun, Y., Zhu, H., & Xiong, H. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3570692 · Cited by 9
“Moreover, the uncertain e ff ectiveness of attention in pinpointing significant features [10], [12], [13], [22] has led to the exploration of more straightforward nonattentional interpretations.”
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
Pitorro, H., & Treviso, M. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1194 · Cited by 9
Sarcasm Detection in Tamil and Malayalam YouTube Comments
Chakravarthi, B. R. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01486-z · Cited by 9
Predictive design of crystallographic chiral separation
Elijošius, R., King‐Smith, E., Faber, F. A., Bernier, L., Berritt, S., Farrell, W. P., Hou, X., Klug-McLeod, J. L., Mustakis, J. G., Sach, N. W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62825-4 · Cited by 9
“The interpretability of attention weights remains a subject of ongoing debate 54,55 .”
Deng, Q., Luz, S. F., & García, S. D. L. F. (2025) · Interspeech (USB) · doi:10.21437/interspeech.2025-1968 · Cited by 9
“In text-based depression detection (e.g., social media analysis), attention scores from transformer models have been used for model interpretation because they provide understandable weight distributions over input features [7].”
Fine-Tuning and Explaining FinBERT for Sector-Specific Financial News: A Reproducible Workflow
Cristescu, M. P., Brândaș, C., Mara, D. A., & Petrea, I. (2025) · Electronics · doi:10.3390/electronics14234680 · Cited by 9
Whisper Has an Internal Word Aligner
Yeh, S.-L., Meng, Y., & Tang, H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434635 · Cited by 9
“Following [40], we use the transcriptions from the TIMIT training set for evaluation.”
Kamahi, S., & Yaghoobzadeh, Y. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.28 · Cited by 9
Explainable Artificial Intelligence for Business and Economics: Methods, Applications and Challenges
Lyu, Q., & Wu, S. (2025) · Expert Systems · doi:10.1111/exsy.70017 · Cited by 10
Why and when should you pool? Analyzing Pooling in Recurrent Architectures
Maini, P., Kolluru, K., Pruthi, D., & Mausam (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.410 · Cited by 9
More Identifiable yet Equally Performant Transformers for Text Classification
Bhardwaj, R., Majumder, N., Poria, S., & Hovy, E. H. (2021) · doi:10.18653/v1/2021.acl-long.94 · Cited by 9
“observed that the weights generated in the model do not provide meaningful explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundZero-shot Sequence Labeling for Transformer-based Sentence Classifiers
Bujel, K., Yannakoudakis, H., & Rei, M. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.20 · Cited by 9
“While evaluating the faithfulness (Herman, 2017) of a model’s rationale is still an open research question and up for debate (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; DeYoung et al., 2020; Jacovi and Goldberg, 2020; Atanasova et al., 2020), the methods explored here allow for measuring”
MethodGHOST at SemEval-2021 Task 5: Is explanation all you need?
Pluciński, K., & Klimczak, H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.114 · Cited by 9
“However, Wiegreffe and Pinter (2019) challenges the aforementioned work, claiming that while the answers given by attention should not be uncontrol-lably trusted and the definition of explanation must be clearly stated, it could still be a useful tool for model understanding and should not be…”
BackgroundAttention-like feature explanation for tabular data
Konstantinov, A. V., & Utkin, L. V. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-022-00351-y · Cited by 9
“As a result, several studies discuss pros [25, 26, 27, 28, 29, 30] and cons [31, 32] of the attention application to problems of explanation.”
BackgroundOn Text Style Transfer via Style-Aware Masked Language Models
Narasimhan, S., Pooja, H., Dey, S., & Desarkar, M. S. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.25 · Cited by 9
Wang, Z., Zhang, L., Qiu, Q., & Kong, F. (2023) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss59052.2023.00016 · Cited by 9
Noun-based attention mechanism for Fine-grained Named Entity Recognition
Rodríguez, A. J. C., Castro, D. C., & Herold-García, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116406 · Cited by 9
Investigating human reading behavior during sentiment judgment
Chen, X., Mao, J., Liu, Y., Zhang, M., & Ma, S. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01523-9 · Cited by 9
“Since the attention mechanism was introduced [1], the investigation of whether the attention is interpretable becomes a hot but controversial topic [5, 13, 36, 40].”
BackgroundInteractively Providing Explanations for Transformer Language Models
Friedrich, F., Schramowski, P., Tauchmann, C., & Kersting, K. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220218 · Cited by 9
“Generally, there are methods that analyze word representations [14], the attention distribution throughout the model [15,16], and the (attention and classification) model heads [17,18].”
MethodMultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations
Lee, D. W., Kim, Y., Picard, R. W., Breazeal, C., & Park, H. W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/433 · Cited by 9
Towards Musically Meaningful Explanations Using Source SeparationPreprint
Haunschmid, V., Manilow, E., & Widmer, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02051 · Cited by 9
“However, whether attention mechanisms are explanations is currently critically debated in the machine learning community [6, 23].”
Background
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.