Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 501–600 of 1,008
Exploring the Translation Mechanism of Large Language Models
Zhang, H., Chen, K., Bai, X., Li, X., Xiang, Y., & zhang, M. (2025) · neural information processing systems · doi:10.52202/085713-3554 · Cited by 16
“Interpretive analyses of model components risk misleading or non-rigorous (Bolukbasi et al., 2021; Wiegreffe and Pinter, 2019).”
LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
Askari, H., Gupta, S., Wang, F., Chhabra, A., & Chen, M. (2025) · neural information processing systems · doi:10.52202/085713-4195 · Cited by 16
Kibria, M. R., Lafond, S., & Arslan, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104405 · Cited by 16
Neural Relation Inference for Multi-dimensional Temporal Point Processes via Message Passing Graph
Zhang, Y., & Yan, J. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/469 · Cited by 16
“The attention model in [Xiao et al. , 2019; Wang et al. , 2017] may be used to infer the relation, while the interpretability is still controversial [Jain and Wallace, 2019; Wiegreffe and Pinter, 2019].”
BackgroundMethodMultilingual multi-aspect explainability analyses on machine reading comprehension models
Cui, Y., Zhang, W.-N., Che, W., Liu, T., Chen, Z., & Wang, S. (2022) · iScience · doi:10.1016/j.isci.2022.104176 · Cited by 16
“However, on the contrary, some works hold positive attitudes about this topic, and they believe that the attention mechanism is a source of explainability (Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020).”
BackgroundLiu, D., Greene, D., & Dong, R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.178 · Cited by 16
“Some researchers have supported this viewpoint (Serrano and Smith, 2019), but treated with skepticism by others (Wiegreffe and Pinter, 2019).”
Interpretable Low-Resource Legal Decision Making
Bhambhoria, R. V., Liu, H., Dahan, S., & Zhu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21438 · Cited by 16
A self-interpretable module for deep image classification on small data
Rosa, B. L., Capobianco, R., & Nardi, D. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03886-6 · Cited by 16
Convolution-Enhanced Evolving Attention Networks
Wang, Y., Yang, Y., Li, Z., Bai, J., Zhang, M., Li, X., Yu, J., Zhang, C., Huang, G. M., & Tong, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3236725 · Cited by 16
“nism helps explain the predictions of the model [8], [21].”
BackgroundRobust Natural Language Understanding with Residual Attention Debiasing
Wang, F., Huang, J. Y., Yan, T., Zhou, W., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.32 · Cited by 16
Empirically revisiting and enhancing automatic classification of bug and non-bug issues
Li, Z., Pan, M., Pei, Y., Zhang, T., Wang, L., & Li, X. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2771-z · Cited by 16
Socialized Learning: A Survey of the Paradigm Shift for Edge Intelligence in Networked Systems
Wang, X., Zhao, Y., Qiu, C., Hu, Q., & Leung, V. C. M. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3482978 · Cited by 17
“However, it’s essential to recognize that while attention provides significant insights, it’s not an exhaustive explanation of model behavior, much like adhering to a norm doesn’t fully capture society’s complexity [126].”
Attention Head Masking for Inference Time Content Selection in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.397 · Cited by 16
MEME: Generating RNN Model Explanations via Model ExtractionPreprint
Kazhdan, D., Dimanov, B., Jamnik, M., & Lió, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06954 · Cited by 17
“Furthermore, it has recently been argued that “attention is not explanation”, emphasising that in practice, it is often unclear what relationship exists between attention weights and model outputs [26].”
BackgroundOn Robust Prefix-Tuning for Text ClassificationPreprint
Yang, Z., & Liu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.10378 · Cited by 16
“2019; Wiegreffe & Pinter, 2019; Abnar & Zuidema, 2020).”
Should We Attend More or Less? Modulating Attention for FairnessPreprint
Zayed, A., Mordido, G., Shabanian, S., & Chandar, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13088 · Cited by 16
“There has been an ongoing debate in the literature regarding the interpretability of the attention mechanism (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Moradi et al., 2019; Mohankumar et al., 2020).”
BackgroundEvaluating the Utilities of Large Language Models in Single-cell Data AnalysisPreprint
Zhao, H., Liu, T., Li, K., Wang, Y., & Li, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3376641/v1 · Cited by 16
Attention Meets Post-hoc Interpretability: A Mathematical PerspectivePreprint
Lopardo, G., Precioso, F., & Garreau, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.03485 · Cited by 16
“More recently, Bibal et al. (2022) provide an overview of the debate on whether attention serves as an explanation, focusing on literature that builds on the works of Jain & Wallace (2019) and Wiegreffe & Pinter (2019).”
InfluentialNeural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Tong, A., Nguyen-Tang, T., Lee, D., Nguyen, D., Tran, T. M., Hall, D. W., Kang, C., & Choi, J. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2503.01329 · Cited by 16
Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformers
Kim, J., Jung, K., Na, D., Jang, S., Park, E., & Choi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.95 · Cited by 15
i-Align: an interpretable knowledge graph alignment model
Trisedya, B. D., Salim, F. D., Chan, J., Spina, D., Scholer, F., & Sanderson, M. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00963-3 · Cited by 15
“The proposed model is built on top of a Transformer model [19] to exploit its self-attention mechanism to rank the importance of the attributes and neighbors [20] as the model’s explanation.”
MethodInterpreting Stress Detection Models Using SHAP and Attention for MuSe-Stress 2022
Park, H.-M., Kim, G., Oh, J., Messem, A. V., & Neve, W. D. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3488112 · Cited by 15
“For example, several publications argue that attention is an intermediate result and has little correlation with model interpretation [43], [44], while other papers refute this claim [45], [46].”
Jiao, Z., & Tao, R. (2025) · Transactions in GIS · doi:10.1111/tgis.70029 · Cited by 15
“Second, although GeoGAT-E's interpretability has a theoretical foundation and is supported by previous research (Sui et al. 2022; Zhao and Zhang 2024; Vaswani 2017; Wiegreffe and Pinter 2019), the precise accuracy of its interpretability still requires further validation.”
Learning Task Representations from In-Context Learning
Sağlam, B., Hu, X., Yang, Z., Kalogerias, D. S., & Karbasi, A. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.345 · Cited by 15
Tokenization and deep learning architectures in genomics: A comprehensive review
Testagrose, C., & Boucher, C. A. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.038 · Cited by 15
Review of Explainable Graph-Based Recommender Systems
Markchom, T., Liang, H., & Ferryman, J. M. (2025) · ACM Computing Surveys · doi:10.1145/3772273 · Cited by 15
“…annotations - Visualization-based evaluation is subjective, varying based on individual observers’ inter-pretations-Moresuited for preliminary assessments that broadly evaluate overall model performance; additional evaluation techniques are required for a comprehensive understanding -”
Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety
Lee, S., Cho, A., Kim, G., Peng, S., Phute, M., & Chau, D. H. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1091 · Cited by 15
Mining Actuarial Risk Predictors in Accident Descriptions Using Recurrent Neural Networks
Baillargeon, J.-T., Lamontagne, L., & Marceau, É. (2020) · Risks · doi:10.3390/risks9010007 · Cited by 15
“Even if the interpretability of the attention vectors is challenged, we believe, as Wiegreffe and Pinter (2019), that our application benefits from using attention weighs from multiple attention mechanisms.”
BackgroundTradformer: A Transformer Model of Traditional Music Transcriptions
Casini, L., & Sturm, B. L. T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/681 · Cited by 15
An Empirical Study of User Engagement in Influencer Marketing on Weibo and WeChat
Wang, J., Yu, Y., Liu, Q., Fang, Z., Shujuan, S., & Xu, Y. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3204177 · Cited by 15
“Yet, since it is not originally proposed for the purpose of interpretation, it is still under debate whether attention distributions can reflect on the reasons during decisions of one model [46].”
BackgroundPost-hoc Saliency Methods Fail to Capture Latent Feature Importance in Time Series Data
Schröder, M., Zamanian, A., & Ahmidi, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39539-0_10 · Cited by 15
Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes
Xie, K., Wiegreffe, S., & Riedl, M. O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.209 · Cited by 15
Self-citationAn Attribution Method for Siamese Encoders
Moeller, L., Nikolaev, D., & Padó, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.980 · Cited by 15
Self-Supervised Forecasting in Electronic Health Records With Attention-Free Models
Kumar, Y., Ilin, A., Salo, H., Kulathinal, S., Leinonen, M. K., & Marttinen, P. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3353164 · Cited by 15
“While attention maps, from the self-attention mechanism, provide a degree of insight into model explainability, the clarity and reliability of such interpretability are subject of ongoing debate within the research community [29], [68].”
BackgroundTowards a Framework for Evaluating Explanations in Automated Fact Verification
Kotonya, N., & Toni, F. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3kcoqz4uix8g · Cited by 16
Relation Extraction with Explanation
Shahbazi, H., Fern, X. Z., Ghaeini, R., & Tadepalli, P. V. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.579 · Cited by 15
Examining the rhetorical capacities of neural language models
Zhu, Z., Pan, C., Abdalla, M., & Rudzicz, F. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.3 · Cited by 15
Minun: evaluating counterfactual explanations for entity matching
Wang, J., & Li, Y. (2022) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3533028.3533304 · Cited by 15
“Some studies in the NLP field tried to explain the results of attention modules [33, 41] and Transformer-based language models [6, 37] with the distribution of attention weights.”
BackgroundMultimodal Continuous Visual Attention Mechanisms
Farinhas, A., Martins, A. F. T., & Aguiar, P. M. Q. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00122 · Cited by 15
“In addition to boosting the performance of a model, attention mechanisms can provide insights into the model’s decision process, being suitable for interpretability purposes [25, 5].”
BackgroundLink Prediction using Graph Neural Networks for Master Data ManagementPreprint
Ganesan, B., Parkala, S., Singh, N. R., Bhatia, S., Mishra, G., Pasha, M. A., Patel, H., & Naganna, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04732 · Cited by 15
Latent Template Induction with Gumbel-CRFsPreprint
Fu, Y., Tan, C., Bi, B., Chen, M., Feng, Y., & Rush, A. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14244 · Cited by 15
“Non-probabilistic approaches include using attention weights as sources of interpretability [26, 61], or building specialized network architectures like entity modeling [49] or copy mechanism [22].”
MethodSalKG: Learning From Knowledge Graph Explanations for Commonsense ReasoningPreprint
Chan, A., Xu, J., Long, B., Sanyal, S., Gupta, T., & Ren, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08793 · Cited by 15
“How feature attribution methods should be chosen remains an open question and the subject of much recent debate (Bastings and Filippova, 2020; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Jain and Wallace, 2019).”
BackgroundNukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy DomainsPreprint
Burke, L. H., Pazdernik, K., Fortin, D., Wilson, B., Goychayev, R., & Mattingly, J. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12192 · Cited by 15
“…is not a clear consensus on how useful these embeddings can be in providing explanations, with arguments from both sides (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), there is undoubtedly some information contained within these transformer-based language models because their predictive…”
BackgroundAttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
Liu, F., Kandpal, N., & Raffel, C. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2411.15102 · Cited by 15
“Attention weights have therefore also been a popular tool to identify important information to the model (Pruthi et al., 2020a; Serrano & Smith, 2019; Wiegreffe & Pinter, 2019) and we therefore use attention weight as one of our baselines.”
Rationales for Sequential Predictions
Vafa, K., Deng, Y., Blei, D., & Rush, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.807 · Cited by 14
“An explanation is faithful if it accurately depicts how a model makes a decision (Jacovi and Goldberg, 2020); an explanation is plausible if it can be understood and interpreted by humans (Wiegreffe and Pinter, 2019).”
BackgroundDebiasing pretrained text encoders by paying attention to paying attention
Gaci, Y., Benatallah, B., Casati, F., & Benabdeslem, K. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.651 · Cited by 14
“In this work, we disagree with these anti-attention studies, and following the arguments of Wiegreffe and Pinter (2019), we emphasize that their limitations include: (1) the experimental setup was particularly limited to recurrent architectures (RNNs).”
Viehmann, C., Beck, T., Maurer, M., Quiring, O., & Gurevych, I. (2022) · Communication Methods and Measures · doi:10.1080/19312458.2022.2151579 · Cited by 14
“For transformer models (e.g., BERT), the attention mechanism (Ethayarajh & Jurafsky, 2021; Wiegreffe & Pinter, 2019) is utilized to gain insights about the influence of input words on the decision process of the classifier.”
MethodExplainable Vision Transformers for Vein Biometric Recognition
Albano, R., Giusti, L., Maiorana, E., & Campisi, P. (2024) · IEEE Access · doi:10.1109/access.2024.3393558 · Cited by 14
“Although there is an ongoing debate about the effectiveness of generic attention mechanisms as explainability tools [48], [49], there is also a consensus on the efficacy of ViT attention maps in providing relevant insights on models explainability for visual tasks [50], [51].”
BackgroundExplaining Neural News Recommendation with Attributions onto Reading Histories
Möller, L., & Padó, S. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3673233 · Cited by 14
“Apart from exclusively optimizing for click-prediction accuracy, an increasing number of publications is also concerned with the diversity of news recommendations [20, 26, 54, 67, 77], debiasing them [62, 75], privacy-preservation [52] and fairness [53, 80].”
Hierarchical attention network for short-term runoff forecasting
Wang, H. E., Qin, H., Liu, G., Huang, S., Qu, Y., Qi, X., & Zhang, Y. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131549 · Cited by 14
Kim, J., Woo, J., Park, J. Y., Kim, D., & Kim, D. (2024) · Metabolic Engineering · doi:10.1016/j.ymben.2024.11.007 · Cited by 14
Abundant Modalities Offer More Nutrients: Multi-Modal-Based Function-Level Vulnerability Detection
Ni, C., Yin, X., Li, X., Xu, X., & Yu, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731557 · Cited by 14
“…line within the function is Line 20: bytes + = ASSEGMENT_SIZE (written, use32bit); To better explain why MVulD + can detect this vulnerability, we follow prior works [15, 26, 58] and employ attention mechanisms to investigate the content that the model focuses on during the recognition process.”
Ge, X., Lee, Y.-T., & Yeh, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1564339 · Cited by 14
BERT for Sequence-to-Sequence Multi-label Text Classification
Yarullin, R., & Serdyukov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72610-2_14 · Cited by 14
Sentiment Lexical Strength Enhanced Self-supervised Attention Learning for sentiment analysis
Wang, X., Fan, M., Kong, M., & Pei, Z. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109335 · Cited by 14
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
Saha, S., Hase, P., Rajani, N. F., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.137 · Cited by 14
ExpMRC: explainability evaluation for machine reading comprehension
Cui, Y., Liu, T., Che, W., Chen, Z., & Wang, S. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09290 · Cited by 14
“as whether the attention could be explanations (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020).”
BackgroundTeaching Machines to Read, Answer and Explain
Cui, Y., Liu, T., Che, W., Chen, Z., & Wang, S. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3156789 · Cited by 14
Model Distillation for Faithful Explanations of Medical Code Predictions
Wood-Doughty, Z., Cachola, I. A., & Dredze, M. H. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.41 · Cited by 14
“Jain and Wallace (2019) showed that attention mechanisms can provide mis-leading explanations, whereas Wiegreffe and Pinter (2019) argued that attention-based explanations are often plausible, even when unfaithful.”
BackgroundAttention Analysis and Calibration for Transformer in Natural Language Generation
Lu, Y., Zhang, J., Zeng, J., Wu, S., & Zong, C. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3180678 · Cited by 14
“Unlike criticizing attention weights as an explanation, Wiegreffe and Pinter [7] claim that trained attention mechanisms do learn something meaningful about the relationship between inputs and outputs, such as syntactic information [46], [47].”
BackgroundFollow-Up Attention: An Empirical Study of Developer and Neural Model Code Exploration
Paltenghi, M., Pandita, R., Henley, A. Z., & Ziegler, A. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3445338 · Cited by 15
An Interpretability Evaluation Benchmark for Pre-trained Language ModelsPreprint
Shen, Y., Wang, L., Chen, Y., Xiao, X., Liu, J., & Wu, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.13948 · Cited by 14
“Attention based methods use attention weights as the importance scores (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Cui et al., 2022).”
MethodSummarization Programs: Interpretable Abstractive Summarization with Neural Modular TreesPreprint
Saha, S., Zhang, S., Hase, P., & Bansal, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.10492 · Cited by 14
“Typically, the modules in an NMN provide attention-based explanations whose interpretability has been debated (Serrano & Smith, 2019; Wiegreffe & Pinter, 2019).”
BackgroundVisual Explanation for Open-Domain Question Answering With BERT
Shao, Z., Sun, S., Zhao, Y., Wang, S., Wei, Z., Gui, T., Turkay, Ç., & Chen, S. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3243676 · Cited by 13
“Althoughcontroversyremainsabout whetherattentionisinterpretable[29],[88]ornot,aprevious study[59]claimedthatthedistributionofattentionmapsoften representslocalaggregation,andtheattributionscorerepresents globalaggregation.”
Feature-Enhanced Neural Collaborative Reasoning for Explainable Recommendation
Zhang, X., Shi, S., Li, Y., Ma, W., Sun, P., & Zhang, M. (2024) · ACM Transactions on Information Systems · doi:10.1145/3690381 · Cited by 13
“However, it has been controversial whether attention mechanisms can serve as explanations [18, 49].”
Carrasco-Ribelles, L. A., Cabrera‐Bean, M., Khalid, S., Roso‐Llorach, A., & Violán, C. (2025) · Journal of Medical Systems · doi:10.1007/s10916-024-02138-z · Cited by 13
“While attention maps can be useful in increasing model trans-parency [32, 33], their role in increasing interpretability may be limited, according to recent discussions [33–35].”
Xiang, Y., Li, X., Gao, Q., Xia, J., & Yue, Z. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3550353 · Cited by 14
TransCORALNet: A two-stream transformer CORAL networks for supply chain credit assessment cold start
Shi, J., Siebes, A. P. J. M., & Mehrkanoon, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127581 · Cited by 13
Verification is All You Need: Prompting Large Language Models for Zero-Shot Clinical Coding
Li, S., Zheng, C., Wu, J., Xu, Q., Xu, X., Wang, H., Sun, Y., Bai, Z., Xu, Y., Zhu, L., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3593028 · Cited by 13
“As a result, automated clinical coding has been proposed [4], [5], [6], [7], [8], [9], [10], [11], [12].”
Direct Visual Grounding by Directing Attention of Visual Tokens
Esmaeilkhani, P., & Latecki, L. J. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00561 · Cited by 13
“Inspired by attention–rationale alignment in NLP [53] and Grad-CAM in vision [44], recent VQA and grounding works add auxiliary losses that guide tokens toward annotated or automatically generated maps [36, 59].”
Sardar, S., Dash, S., Roychowdhury, P., Solanki, J., Sikdar, P., Thangappan, J., & Dutta, S. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag269 · Cited by 13
“While Wiegreffe and Pinter [ 91] partially rebutted this claim, arguing that interpretability is condition-dependent rather than categorically absent, and proposing calibration protocols; including variance analysis across random seeds and adversarial attention training to empirically determine…”
Hu, Z., Liu, H., Xiong, Y., Wang, L., Wu, R., Guan, K., Hu, Y., Lyu, T., & Fan, C. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15361-6 · Cited by 13
“Methods have developed for different kinds of neural network structures and applications [15, 24, 52].”
Unsupervised Token-level Hallucination Detection from Summary Generation By-products
Marfurt, A., & Henderson, J. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.21 · Cited by 13
“It is debated whether model attentions can be used to explain model decisions (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019) and how much a Transformer encoder’s output representation still represents the token at its position in the input (Brunner et al., 2020).”
BackgroundVision DiffMask: Faithful Interpretation of Vision Transformers with Differentiable Patch Masking
Nalmpantis, A., Panagiotopoulos, A., Gkountouras, J., Papakostas, K., & Aziz, W. F. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00388 · Cited by 13
“However, a consensus has not been reached on whether attention can be considered as a faithful interpretation of the model’s decision-making process, with advocates both in favor [24] and against it [12].”
BackgroundLi, Z., Jin, B., & Fang, J. (2023) · Genomics · doi:10.1016/j.ygeno.2023.110749 · Cited by 13
A Generic and Model-Agnostic Exemplar Synthetization Framework for Explainable AI
Bărbălau, A., Cosma, A., Ionescu, R. T., & Popescu, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67661-2_12 · Cited by 13
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models
Geva, M., Katz, U., Ben-Arie, A., & Berant, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.646 · Cited by 13
“Our findings also relate to explainability methods that highlight parts from the input via the model’s attention (Wiegreffe and Pinter, 2019),”
BackgroundMethodLaughing Heads: Can Transformers Detect What Makes a Sentence Funny?
Peyrard, M., Borges, B., Gligorić, K., & West, R. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/537 · Cited by 13
Pair the Dots: Jointly Examining Training History and Test Stimuli for Model InterpretabilityPreprint
Meng, Y., Fan, C. C., Sun, Z., Hovy, E. H., Wu, F., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06943 · Cited by 13
“The attention mechanism has been used for model interpretation (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Vashishth et al., 2019).”
MethodChallenges in Explanation Quality EvaluationPreprint
Schuff, H., Adel, H., Qi, P., & Vu, N. T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07126 · Cited by 13
“It is frequently implemented as a heat map over the input that visualizes an importance measure, such as integrated gradients (Sundararajan et al., 2017) or attention weights (Wiegreffe and Pinter, 2019).”
BackgroundClass-Discriminative Attention Maps for Vision TransformersPreprint
Brocki, L., Binda, J., & Chung, N. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2312.02364 · Cited by 13
“There is also an ongoing discussion about whether attention provides meaningful explanations [8, 24, 25].”
BackgroundInternalInspector $I^2$: Robust Confidence Estimation in LLMs through Internal StatesPreprint
Beigi, M., Shen, Y., Yang, R., Lin, Z., Wang, Q., Mohan, A., He, J., Jin, M., Lu, C., & Huang, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12053 · Cited by 13
“…workings of transformers indicate that while attention should not be directly equated with explanation (Pruthi et al., 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), it provides significant insights into the model’s operational behavior and helps in error diagnosis and hypothesis…”
BackgroundA survey on bias in machine learning researchPreprint
Mikołajczyk, A., & Grochowski, M. (2024) · Research Square · doi:10.21203/rs.3.rs-5019524/v1 · Cited by 13
When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
Xiao, Y., Tonekaboni, S., Gerych, W., Suriyakumar, V. M., & Ghassemi, M. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.07452 · Cited by 13
“We also acknowledge the ongoing debate around using attention weights to interpret model behavior (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Bibal et al., 2022).”
Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language ModelsPreprint
Gan, Z., Ren, R., Yao, W., Hu, X., Xu, G., Qian, C., Tang, H., Gong, Z., Yao, X., Tang, P., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.02907 · Cited by 13
Kendiukhov, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17532 · Cited by 13
Unit Testing for Concepts in Neural Networks
Lovering, C., & Pavlick, E. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00514 · Cited by 12
“Many techniques have been proposed in order to determine which input features are “important” to model decisions (Ribeiro et al., 2016; Sundararajan et al., 2017; Kim et al., 2018; Wiegreffe and Pinter, 2019).”
BackgroundAmino acid sequence encodes protein abundance shaped by protein stability at reduced synthesis cost
(2024) · doi:10.1002/pro.5239 · Cited by 12
Pande, M., Budhraja, A., Nema, P., Kumar, P., & Khapra, M. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17605 · Cited by 12
From outputs to insights: a survey of rationalization approaches for explainable text classification
Guzman, E. M., Schlegel, V., & Batista-Navarro, R. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1363531 · Cited by 13
Zhu, J., Miao, S., Ying, R., & Li, P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00998-9 · Cited by 13
Muhammad, D., Salman, M., & Bendechache, M. (2025) · Computational and Structural Biotechnology Reports · doi:10.1016/j.csbr.2025.100066 · Cited by 13
Rizzo, M., Veneri, A., Marcuzzo, M., Zangari, A., Albarelli, A., Lucchese, C., Nobile, M. S., & Conati, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02703-1 · Cited by 13
A Survey on Bias in Machine Learning Research
Mikołajczyk, A., & Grochowski, M. (2025) · IEEE Access · doi:10.1109/access.2025.3649998 · Cited by 12
“In 2019, Jain et al. [49] explained why they think that ”Attention is not an explanation”, whereas the authors of [116] claimed that Attention is not not an explanation .”
Explainable AI for Hate Speech Moderation: A Stakeholder‐Centered and Sociotechnical Review
Qureshi, M. D. M., Qureshi, M. A., & Rashwan, W. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70076 · Cited by 12
A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
Nguyen, D. H., Gravier, G., & Sébillot, P. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00259 · Cited by 12
“To explain this phenomenon, [11] has introduced the concept of faithfulness when trying to evaluate an attention mechanism, which refers to the capability that an attention map can reflect the model reasoning.”
Shi, Y., & Shi, H. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/8446281 · Cited by 12
“Wiegreffe and Pinter [15] chose test questions similar to the wrong ones for practice.”
BackgroundInfluentialRetractedTang, Z. S., & Surdeanu, M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00463 · Cited by 12
“Followup work debated the validity of this strategy (Jain and Wallace 2019; Wiegreffe and Pinter 2019b; Kobayashi et al. 2020).”
BackgroundAn Attention-Based Architecture for Hierarchical Classification With CNNs
Pizarro, I., Ñanculef, R., & Valle, C. (2023) · IEEE Access · doi:10.1109/access.2023.3263472 · Cited by 12
“…some recent studies have suggested that attention scores can be inconsistent with other feature importance measures [51], [52], subsequent experiments have shown that attention weights can still help analyze the decision and learning mechanisms of otherwise black-box models [47], [53], [54].”
ResultSchröder, M., Zamanian, A., & Ahmidi, N. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5020032 · Cited by 12
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.