Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 801–900 of 1,008
Entropy-Guided KV Caching for Efficient LLM Inference
Kim, H.-K., & Jung, Y. (2025) · Mathematics · doi:10.3390/math13152366 · Cited by 7
Explainability in carbon price forecasting systems: A systematic literature review
Altiary, M., & Hussain, F. K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114612 · Cited by 6
Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions
Mohammadi, S., Vedula, B. H., Lamba, H., Raff, E., Kumaraguru, P., Ferraro, F., & Gaur, M. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1648 · Cited by 6
ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models
Li, H., Wen, Z., Jiang, Q., Li, C., Wu, Y., Yang, Y., Wang, Y., Huang, X., Zhu, M., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3634806 · Cited by 6
“However, the academic community remains divided on the reliability of attention weights as a source of explanation [3,12,64].”
Multinational Address Parsing: A Zero-Shot Evaluation
Yassine, M., Beauchemin, D., Laviolette, F., & Lamontagne, L. (2021) · International Journal of Information Science and Technology · doi:10.48550/arxiv.2112.04008 · Cited by 6
“However, other work has suggested that attention mechanisms provide a certain degree of interpretability depending on the task at hand [25] [26].”
BackgroundAre Prompt-based Models Clueless?
Kavumba, P., Takahashi, R., & Oda, Y. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.991 · Cited by 6
Zero-Shot Translation of Attention Patterns in VQA Models to Natural Language
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54605-1_25 · Cited by 6
An Explainable Approach to Understanding Gender Stereotype Text
Jeyaraj, M. N., & Delany, S. J. (2024) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2024.gebnlp-1.4 · Cited by 6
“Nevertheless, there are works that strongly challenge this claim of the attention not being an explanation of feature importance (Wiegreffe and Pinter, 2019).”
BackgroundMill, E., Garn, W., & Turner, C. J. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2430867 · Cited by 6
“This approach acknowledges the fact that a strictly faithful explanation is unlikely to exist since counter-examples can always be found (Wiegreffe and Pinter 2019).”
Language Modelling Techniques for Analysing the Impact of Human Genetic Variation
Hegde, M., Nebel, J., & Rahman, F. (2025) · Bioinformatics and Biology Insights · doi:10.1177/11779322251358314 · Cited by 7
“Alternatively, in attention-based models such as transformers, the attention weights confer the relative importance given to certain tokens 259,260 and hence can be used to infer the impact of certain tokens in the decision of the model.”
DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving
Lai, S., Xue, T., Xiao, H., Hu, L., Wu, J., Feng, N., Guan, R., Liao, H., Li, Z., & Yue, Y. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127884 · Cited by 6
Bayes-enhanced Lifelong Attention Networks for Sentiment Classification
Wang, H., Wang, S., Mazumder, S., Liu, B., Yang, Y. T., & Li, T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.50 · Cited by 6
“In addition, whether attention is explainable (or interpretable) was recently discussed in (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
BackgroundIs “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases
Lyu, Q., Zheng, H., Li, D., Zhang, L., Apidianaki, M., & Callison-Burch, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.388 · Cited by 6
Assessing the use of attention weights to interpret BERT-based stance classification
Sáenz, C. A. C., & Becker, K. E. (2021) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3486622.3493966 · Cited by 6
“Other works have discussed the value of attention weights to provide some level of interpretability to BERT-based models [7, 17].”
BackgroundStyLEx: Explaining Style Using Human Lexical Annotations
Hayati, S. A., Park, K., Rajagopal, D., Ungar, L., & Kang, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.208 · Cited by 6
“Explainable NLP Heat maps generated from attention values from the models (Bahdanau et al., 2014) are widely used as an interpretability tool, but these attention maps are often unfaithful and unreliable (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2020).”
MethodLearning the Visualness of Text Using Large Vision-Language Models
Verma, G., Rossi, R. A., Tensmeyer, C., Gu, J., & Nenkova, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.147 · Cited by 6
Enhancing Tabular Reasoning with Pattern Exploiting Training
Shankarampeta, A., Gupta, V., & Zhang, S. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.54 · Cited by 6
Controlling the Focus of Pretrained Language Generation Models
Ji, J., Kim, Y., Glass, J. R., & He, T. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.260 · Cited by 6
Corro, L. D., & Hoffart, J. (2021) · Workshop on Economics and Natural Language Processing (ECONLP) · doi:10.18653/v1/2021.econlp-1.6 · Cited by 6
“Multiple approaches explored alternatives to extract price signals from news such as sentiment analysis, semantic parsing, etc (Gidófalvi, 2001; Schumaker and Chen, 2009; Xie et al., 2013; Li et al., 2014; Peng and Jiang, 2016).”
BackgroundLabel-Based Convolutional Neural Network for Text Classification
Wang, C., & Tan, C. (2021) · International Conference on Control Engineering and Artificial Intelligence (CCEAI) · doi:10.1145/3448218.3448235 · Cited by 6
“Besides, in [10–12], we find that the interpretability of attention is controversial.”
BackgroundResultAttending via both Fine-tuning and Compressing
Zhou, J., Wu, Y., Chen, Q., Huang, X., & He, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.189 · Cited by 6
“Existing research (Jain and Wallace, 2019) also shows that attention is not explicable, and there are a lot of controversy regarding to the result explanations (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019).”
BackgroundAre Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc InterpretabilityPreprint
Liu, N., Meng, Y., Hu, X., Tie, W., & Long, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07494 · Cited by 6
“This follows previous work (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019) where clas-sification is set as the major task scenario.”
BackgroundExplaining Natural Language Processing Classifiers with Occlusion and Language ModelingPreprint
Harbecke, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11889 · Cited by 6
“Wiegreffe and Pinter (2019) disagree and show that under some circumstances these distortion lead to a significant decrease in performance.”
BackgroundFaithful and Plausible Explanations of Medical Code PredictionsPreprint
Wood-Doughty, Z., Cachola, I. A., & Dredze, M. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07894 · Cited by 6
“Wiegreffe and Pinter (2019) has argued that while the explanations produced by attention may not always be faithful, they”
BackgroundEvaluating Explanations for Reading Comprehension with Realistic Counterfactuals.Preprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · Cited by 6
Rationale production to support clinical decision-makingPreprint
Taylor, N., Sha, L., Joyce, D. W., Lukasiewicz, T., Nevado‐Holgado, A. J., & Kormilitzin, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07611 · Cited by 6
Automatic Rule Induction for Interpretable Semi-Supervised LearningPreprint
Pryzant, R., Yang, Z., Yi‐chong, X., Zhu, C., & Zeng, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09067 · Cited by 6
“…Zhang et al. (2022) we subsample each validation set so that it roughly matches the size of the training set den layers are not valid explanations (Wiegreffe and Pinter, 2019), however in our case the attention scores are part of the model’s output layer, i.e. used in a linear combination to…”
MethodInput-length-shortening and text generation via attention valuesPreprint
Tan, N. Ö., Peng, A. Y., Bensemann, J., Bao, Q., Hartill, T., GAHEGAN, M. N., & Witbrock, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07585 · Cited by 6
“Although attention values cannot be considered as the “exclusive” explanation for the model’s predictions, [35] argues that attention values are still “plausible” explanation of the model’s predictions.”
BackgroundMethodData augmentation and explainability for bias discovery and mitigation in deep learningPreprint
Mikołajczyk, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.09464 · Cited by 7
TaCo: Targeted Concept Erasure Prevents Non-Linear Classifiers From Detecting Protected AttributesPreprint
Jourdan, F., Béthune, L., Picard, A. M., Risser, L., & Asher, N. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06499 · Cited by 6
“…attribution matrix as in (Fel et al., 2023)), (iv) W as the possible parameter settings for the final layer of an attention model as suggested in (Wiegreffe and Pinter, 2019), or (v) sets of factors or dimensions of the final layer embedding matrix from the entire model (Jourdan et al., 2023).”
BackgroundPSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language ModelsPreprint
Sadanandan, B., & Behzadan, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21428 · Cited by 6
“Attention as explanation has known limitations (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019); we use it as one signal among several.”
Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM AgentsPreprint
Chen, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.16364 · Cited by 6
“The 4D-mask intervention needs a model that honors a passed mask (Phi-3 out of scope) and white-box per-layer attention (closed frontier models out of reach); causality of raw attention is contested (Jain and Wallace 2019; Wiegreffe and Pinter 2019), hence our head-line is interventional .”
In the eyes of a language model: A comprehensive examination through eye-tracking data
Dini, L., Moroni, L., Brunato⋄, D., & Dell’Orletta⋄, F. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130617 · Cited by 5
Explainable Seismic Event Discrimination: Improved Explainability With Vision Transformers
Kasburg, V., Zehner, M., van Laaten, M., Müller, J., & Kukowski, N. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2025jh000802 · Cited by 5
Sparse Autoencoders Make Audio Foundation Models More Explainable
Mariotte, T., Lebourdais, M., Almudévar, A., Tahon, M., Ortega, A., & Dugué, N. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463660 · Cited by 5
Rangel-Martínez, D., & Ricardez‐Sandoval, L. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109469 · Cited by 5
Quantifying Modality Contributions via Disentangling Multimodal Representations
Amit, P., Kashyap, O. M., Rayasam, N., Shekhar, N., & Narayan, S. (2026) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.39.1.141869 · Cited by 5
Arous, I., Chehbouni, K., Cheng, Z., & Dossou, B. F. P. (2025) · Springer eBooks · doi:10.1007/978-981-97-8440-0_85-1 · Cited by 6
Aligning What LLMs Do and Say: Towards Self-Consistent Explanations
Admoni, S., Amir, O., Hallak, A., & Ziser, Y. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.49 · Cited by 5
“Attention-based methods are simple but often unreliable (Jain and Wallace, 2019; Ser-rano and Smith, 2019; Wiegreffe and Pinter, 2019), whereas gradient-and perturbation-based methods provide stronger signals, including Integrated Gra-dients (Sundararajan et al., 2017), LRP (Bach et al., 2015;…”
Analyzing Interpretability of Summarization Model with Eye-gaze Information
Ikhwantri, F., Yamada, H., & Tokunaga, T. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2zvrzmob5jcm · Cited by 5
“However, other studies still claimed the effectiveness of the attention layer for an explanation of the model performance (Wiegreffe and Pinter, 2019) and the model behavior similar to human eye movement, especially for Transformer-based models (Eberle et al., 2022; Ikhwantri et al., 2023).”
MethodPerturbation-Driven Visual Analytics for Probing InternalRepresentations in Neural Language Models
Pasula, E. (2026) · International Journal of Intelligent Systems and Data Science · doi:10.67231/jpr65e97 · Cited by 5
“Recent work in interpretability has produced tools for visualizing attention weights and saliency maps [2, 3], but these typically present static views that do not support interactive exploration or causal reasoning about model behavior.”
Variable Instance-Level Explainability for Text Classification.
Chrysostomou, G., & Αλέτρας, Ν. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 5
Zhao, Y., Zhang, Y., Jiang, Z., Ju, Y., Zhao, J., & Liu, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_18 · Cited by 5
“Faithfulness refers to how accurately the explanation reflects the true reasoning process of the model (Herman, 2017; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020).”
BackgroundAn introduction to neural networks and deep learning
Mulyadi, A. W., Yoon, J. S., Jeon, E., Ko, W., & Suk, H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-32-385124-4.00009-x · Cited by 5
MuLX-QA: Classifying Multi-Labels and Extracting Rationale Spans in Social Media Posts
Poddar, S., Mukherjee, R., Samad, A. M., Ganguly, N., & Ghosh, S. (2024) · ACM Transactions on the Web · doi:10.1145/3653303 · Cited by 5
“Prior studies have also deliberated the possibility of explaining model predictions using intermediate representations or attention weights [28, 44, 83].”
Unravelling the semantic mysteries of transformers layer by layer
Zhang, C., Lv, J., Cao, J., Sheng, J., Song, D., & Zhang, T. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf034 · Cited by 5
Explainable AI for Software Engineering: A Systematic Review and an Empirical Study
Haji Mohammadkhani, A. (2023) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/40697 · Cited by 5
Investigating Self-Rationalizing Models for Commonsense Reasoning
Rancourt, F., Vondrlik, P., Maupomé, D., & Meurs, M. (2023) · Stats · doi:10.3390/stats6030056 · Cited by 5
Synchronization-Inspired Interpretable Neural Networks
Han, W., Qin, Z., Liu, J., Böhm, C., & Shao, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3297672 · Cited by 5
“Stability [52], [53], [54] under perturbation of interpreted parts is also a side basis for interpretation assessment.”
BackgroundBallout, M., Krumnack, U., Heidemann, G., & Kühnberger, K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_1 · Cited by 5
Speeding up LIME using Attention Weights
Santhiappan, S., Reghu, M. K., Saminathan, M., & Veerappan, A. (2024) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3632410.3632450 · Cited by 5
Pay Attention to Attention for Sequential Recommendation
Liu, Y. B., Liu, M., & Liu, X. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688164 · Cited by 5
“We therefore only present the results from different datasets in the context of SASRec.”
Lee, J., Ma, S., Serban, N., & Yang, S. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf032 · Cited by 5
Linearly-interpretable concept embedding models for text analysis
Santis, F. D., Bich, P., Ciravegna, G., Barbiero, P., Giordano, D., & Cerquitelli, T. (2025) · Machine Learning · doi:10.1007/s10994-025-06839-5 · Cited by 5
“…explainability methods rely mainly on the post-hoc analysis of the attention mechanisms [6, 7] and the output gradients [8], both of which have shown limited interpretability as they are often unreliable [9, 10] and only show where the model looks, but not what it sees in a given input [11, 12].”
PromptExplainer: Explaining Language Models through Prompt-based Learning
Feng, Z., Zhou, H., Zhu, Z., & Mao, K. (2024) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2024.findings-eacl.60 · Cited by 5
“However, this method’s reliability and effectiveness diminish when applied to Transformer architectures (Wiegreffe and Pinter, 2019), commonly used in language models (De-vlin et al., 2019; Liu et al., 2019; OpenAI, 2022; Touvron et al., 2023).”
MethodTowards Providing Clinical Insights on Long Covid from Twitter Data
Bhambhoria, R. V., Saab, J., Uppal, S., Li, X. S., Yakimovich, A., Bhatti, J. A., Valdamudi, N. K., Moyano, D., Bales, M., Dolatabadi, E., et al. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-14771-5_19 · Cited by 5
Robust Representation Learning via Sparse Attention Mechanism for Similarity Models
Ermilova, A., Baramiia, N., Kornilov, V., Petrakov, S., & Zaytsev, A. (2024) · IEEE Access · doi:10.1109/access.2024.3418779 · Cited by 5
“The obtained results prove the statement that ‘‘Attention is not not Explanation’’ [58].”
ResultSemantic Matching for Sequence-to-Sequence Learning
Zhang, R., Chen, C., Zhang, X., Bai, K., & Carin, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.21 · Cited by 5
Prototypical Convolutional Neural Network for a Phrase-Based Explanation of Sentiment Classification
Pluciński, K., Lango, M., & Stefanowski, J. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93736-2_35 · Cited by 5
Explainable Text Classification for Legal Document Review in Construction Delay Disputes
Huber-Fliflet, N., Zhang, J., Gronvall, P., Wei, F., Spinelli, P., Dąbrowski, A. G., & Yang, J. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386240 · Cited by 5
“On the other side, Wiegreffe and Pinter [7] challenged Jain and Wallace’s argument and claimed that attention weights can indeed be used to produce the explanation for a prediction based on the experiments they conducted.”
BackgroundOn effects of Knowledge Distillation on Transfer LearningPreprint
Thapa, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.09668 · Cited by 5
“We also do not explore the attention-based method, even if we use a transformer-based model, due to the debate about its validity as an explanation and its vulnerability to manipulation [103, 104].”
MethodTowards eXplainable AI for Mobility Data SciencePreprint
Jalali, A., Graser, A., & Heistracher, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08461 · Cited by 5
Decoding Layer Saliency in Language TransformersPreprint
Hou, E., & Castanon, G. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.05219 · Cited by 5
“However, this new category did not come without its share of controversy, with many papers questioning or defending their explanatory power (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Serrano & Smith, 2019; Pruthi et al., 2019; Vashishth et al., 2019; Bastings & Filippova, 2020).”
BackgroundDeng, Q., Luz, S. F., & García, S. D. L. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13476 · Cited by 5
“Attention scores have been suggested as a means of model interpretation because they provide un-derstandable weight distributions over input features [7].”
Does Faithfulness Conflict with Plausibility? An Empirical Study in Explainable AI across NLP TasksPreprint
Lu, X., & Ma, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.00140 · Cited by 5
“…attention-based, gradient-based and perturbation-based feature attribution methods have been extensively studied to explore the internal mechanisms of black-box deep models [3, 4, 5], which improves transparency in AI, particularly in sensitive applications like clinical decision-making systems.”
BackgroundMethodLompo, B. A., Le, T.-D., Jouvet, P. A., & Noumeir, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.10171 · Cited by 5
Explainable Risk Classification in Financial ReportsPreprint
Tan, X. W., & Kok, S. (2024) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2405.01881 · Cited by 5
“This interpretability aspect of the attention mechanism makes it an attractive choice for generating explanations (Wiegreffe & Pinter, 2019).”
BackgroundSelf-Explaining Hypergraph Neural Networks for Diagnosis PredictionPreprint
Yu, L., Cai, Y., Zhang, M., & Hu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.10689 · Cited by 5
“The attention mechanism has been employed to achieve intrinsic interpretability by us-ing attention weights as explanations (Mohankumar et al., 2020), but its reliability is arguable (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Jain and Wallace, 2019).”
Quantifying Cross-Attention Interaction in Transformers for Interpreting TCR-pMHC BindingPreprint
Li, J., Yin, Z., Smith, H., Ding, Z., Landry, S. J., & Mettu, R. R. (2025) · PubMed Central · doi:10.48550/arxiv.2507.03197 · Cited by 5
“We compare our QCAI method against several existing post-hoc interpretability techniques, including AttnLRP [2], TokenTM [67], AttCAT [52], Rollout [1], GradCAM [56], LRP [6], and RawAttn [66].”
InfluentialFrom Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMsPreprint
Yu, K. S., Bulusu, V., Yasunaga, O., Lau, C., Blondin, C., O’Brien, S., Zhu, K., & Sharma, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.21800 · Cited by 5
Position: We Need An Algorithmic Understanding of Generative AIPreprint
Eberle, O., McGee, T., Giaffar, H., Webb, T. W., & Momennejad, I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07544 · Cited by 5
“…feature attribution methods compute feature importance scores, addressing the limits of attention analysis in providing faithful explanations (Wiegreffe & Pinter, 2019), with techniques like saliency (Chefer et al., 2021) and modified gradient methods (Ali et al., 2022; Achtibat et al.,…”
Päivärinta, M. (2022) · UTUPub (University of Turku) · Cited by 5
Interpreting Transformers Through Attention Head InterventionPreprint
Kadem, M., & Zheng, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.04398 · Cited by 5
“Even if many heads are removable, the remaining critical heads may still provide faithful explanations of model behaviour (Wiegreffe and Pinter, 2019).”
InfluentialWhen Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent ReasoningPreprint
Sahoo, S., Chadha, A. R., Jain, V., & Chaudhary, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.03475 · Cited by 5
Metheniti, E. (2023) · theses.fr (ABES) · doi:10.70675/2f81071bz4724z44ecz999dz1acee85d8951 · Cited by 5
Advancing fairness in natural language processing : from traditional methods to explainability
Jourdan, F. (2024) · theses.fr (ABES) · doi:10.70675/b8535a05z82f1z46f7z9688z3cd35cea8381 · Cited by 5
Experience-guided multi-agent interpretable framework for radiology report summarization
Li, J., Zhou, T., Zhou, Z., Wei, X., Song, H., Wang, Z., Chen, Y., & Lv, H. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.109078 · Cited by 4
Pittala, R. B., Kiran, M. A., Sharma, N., Thaile, M., Singamaneni, K. K., Byrapuneni, L. P., Krishna Bhargavi, Y., & Nagamani, P. (2026) · Journal of Intelligent & Fuzzy Systems · doi:10.1177/18758967261420571 · Cited by 4
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Tian, Y., Mo, F., Zhang, W., Qi, Y., & Nie, J. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.66 · Cited by 4
“Despite ongoing debate regarding their faithfulness as explanations of internal reasoning (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhang et al., 2024, 2025a,b; Mo et al., 2026), attention patterns have demonstrated considerable utility in practice.”
GRITHopper: Decomposition-Free Multi-Hop Dense Retrieval
Erker, J.-J., Reimers, N., & Gurevych, I. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.5 · Cited by 4
Darnoto, B. R. P., & Firmawan, D. B. (2026) · Journal of Computing Theories and Applications · doi:10.62411/jcta.15975 · Cited by 4
“While Wiegreffe and Pinter [38] partially challenged this conclusion, the broader explainability literature [39] generally supports the use of model-agnostic perturbation-based approaches for more reliable post-hoc interpretation.”
A survey on multimodal explainable Artificial Intelligence
Giovannoni, C., Metta, C., Monreale, A., & Rinzivillo, S. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200671 · Cited by 4
Arık, K. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10702-8 · Cited by 3
“While attention maps provide a seemingly transparent view into model behavior, they do not always reflect causal decision-making processes and may be sensitive to input perturbations [37, 38].”
A machine learning framework for off-ball defensive evaluation applied to corner kicks
Groom, S., Belo, F., Rice, A., Anderson, L., & Wang, S. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00138-7 · Cited by 4
Głuchowska, A., Khan, M. H. U., & Pawełkowicz, M. E. (2026) · Applied Sciences · doi:10.3390/app16147275 · Cited by 4
Yeşil, Ö. F. (2026) · Foods · doi:10.3390/foods15162860 · Cited by 4
Silver Data for Coreference Resolution in Ukrainian: Translation, Alignment, and Projection
Kuchmiichuk, P. (2023) · Ukrainian Natural Language Processing Workshop (UNLP) · doi:10.18653/v1/2023.unlp-1.8 · Cited by 4
“…subject of various experiments, and while the saliency methods have been proven to work better, cross-attention can still provide important information about the functionality of the models (Vashishth et al., 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020).”
BackgroundA Trustworthy Approach to Classify and Analyze Epidemic-Related Information From Microblogs
Nguyen, T. H., Fisichella, M., & Rudra, K. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3391395 · Cited by 4
Research on attention mechanism optimization of deep learning algorithms
Huang, C. (2024) · AIP conference proceedings · doi:10.1063/5.0222691 · Cited by 4
Pandey, A., Chen, W., & Keten, S. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00205 · Cited by 4
Pure component property estimation framework using explainable machine learning methods
Jiao, J., Gao, X. A., & Li, J. (2025) · Chinese Journal of Chemical Engineering · doi:10.1016/j.cjche.2025.05.011 · Cited by 4
Zheng, Y., Li, H., Yu, K., & Chen, L. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.522 · Cited by 4
Order-Level Attention Similarity Across Language Models: A Latent Commonality
Liang, J., Zhong, J., Huang, S., Hu, Y., Zhang, H., Li, H., Fan, L., & Gu, H. (2025) · neural information processing systems · doi:10.52202/085713-4044 · Cited by 4
Epistemic limits of local interpretability in self-modulating cognitive architectures
Mahrouk, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1677528 · Cited by 4
Modeling the Centaur: Human-Machine Synergy in Sequential Decision Making
Shoresh, D., & Loewenstein, Y. (2025) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/uvho8208 · Cited by 4
Sadeghsalehi, H. (2026) · Array · doi:10.1016/j.array.2026.100778 · Cited by 4
Investigating alignment interpretability for low-resource NMT
Boito, M. Z., Villavicencio, A., & Besacier, L. (2020) · Machine Translation · doi:10.1007/s10590-020-09254-w · Cited by 4
“However, recent studies focused on the interpretability of attention mechanisms in NLP tasks failed to find a strong correlation between the attention layer’s weights and the systems’ output (Jain and Wallace 2019; Wiegreffe and Pinter 2019; Vashishth et al. 2019; Serrano and Smith 2019).”
Background
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.