Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 201–300 of 1,008
Explainable machine learning in cybersecurity: A survey
Yan, F., Wen, S., Nepal, ., Paris, C. L., & Xiang, Y. (2022) · International Journal of Intelligent Systems · doi:10.1002/int.23088 · Cited by 59
“Attention is not explanation : In 2019, there were two papers bring a discussion 72,73,112,113 of the explanatory attention mechanism.”
BackgroundA Prioritization Model for Suicidality Risk Assessment
Shing, H.-C., Resnik, P., & Oard, D. W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.723 · Cited by 59
“Although concerns about using attention for interpretation exist (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Wallace, 2019), Shing et al.”
BackgroundHuman Interpretation of Saliency-based Explanation Over Text
Schuff, H., Jacovi, A., Adel, H., Goldberg, Y., & Vu, N. T. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533127 · Cited by 59
“2This question is distinct from the question of whether the explanation faithfully communicates information about the model [27, 53]: even if the feature-attribution information is entirely faithful, discrepancies may still arise in how humans comprehend this information.”
BackgroundRethinking Attention-Model Explainability through Faithfulness Violation TestPreprint
Liu, Y., Li, H., Guo, Y., Kong, C., Li, J., & Wang, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.12114 · Cited by 59
“…puzzles in inherent attention explanations, such as whether attention is directly explainable (Jain & Wallace, 2019; Serrano & Smith, 2019; Wiegreffe & Pinter, 2019; Brunner et al., 2020), or how attention behaves (Clark et al., 2019; Bai et al., 2021; Michel et al., 2019; Mohankumar et…”
BackgroundFeature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58
“…networks have been noted as producing incoherent results when subjected to perturbation-style analysis (Feng et al. 2018), while certain types of attention mechanism have been subject to a pro-tracted debate over their utility and informativeness (Jain and Wallace 2019; Wiegreffe and Pinter 2019).”
BackgroundJin, J., Xing, S., Ji, E., & Liu, W. (2025) · Sensors · doi:10.3390/s25072183 · Cited by 58
MambaLRP: Explaining Selective State Space Sequence Models
Jafari, F. R., Montavon, G., Müller, K., & Eberle, O. (2024) · neural information processing systems · doi:10.52202/079017-3764 · Cited by 58
Dodrio: Exploring Transformer Models with Interactive Visualization
Wang, Z. J., Turko, R., & Chau, D. H. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.16 · Cited by 57
“In NLP, there has been a growing body of research on attention used as a tool for interpretability across many language tasks (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Kobayashi et al., 2020).”
MethodExplain and improve: LRP-inference fine-tuning for image captioning models
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.008 · Cited by 57
“The question to what extent attention is suitable as an explanation has been raised in the natural language processing context [79], [80], [81].”
BackgroundUnderstanding How Encoder-Decoder Architectures AttendPreprint
Aitken, K., Ramasesh, V., Cao, Y., & Maheswaranathan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15253 · Cited by 57
“Related Work As mentioned in the introduction, a common technique to gain some understanding is to visualize learned attention matrices, though the degree to which such visualization can explain model predictions is disputed Wiegreffe & Pinter (2019); Jain & Wallace (2019); Serrano & Smith (2019).”
Entropy-based attention regularization frees unintended bias mitigation from lists
Attanasio, G., Nozza, D., Hovy, D., & Baralis, E. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.88 · Cited by 56
Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?
Eberle, O., Brandl, S., Pilot, J., & Søgaard, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.296 · Cited by 56
Attention Mechanisms in Deep Learning : Towards Explainable Artificial Intelligence
Dehimi, N. E. H., & Tolba, Z. (2024) · International Conference on Pattern Analysis and Intelligent Systems (PAIS) · doi:10.1109/pais62114.2024.10541203 · Cited by 55
“• Wiegreffe and Pinter [6] introduced counterfactual attention explanations, analyzing how changes to the input based on attention weights would influence the model’s output, offering insights into the model’s sensitivity to specific features .”
BackgroundAttention Flows are Shapley Value Explanations
Ethayarajh, K., & Jurafsky, D. (2021) · doi:10.18653/v1/2021.acl-short.8 · Cited by 55
“Others have countered that they have some utility (Wiegreffe and Pinter, 2019).”
BackgroundMadsen, A., Meade, N., Adlakha, V., & Reddy, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.125 · Cited by 56
Information-Transport-based Policy for Simultaneous Translation
Zhang, S., & Feng, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.65 · Cited by 54
Obtaining Faithful Interpretations from Compositional Neural Networks
Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., & Gardner, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.495 · Cited by 37
Offensive Language Detection Explained
Risch, J., Ruff, R., & Krestel, R. (2020) · Cited by 54
Explaining and Improving Model Behavior with k Nearest Neighbor RepresentationsPreprint
Rajani, N. F., Krause, B., Yin, W., Niu, T., Socher, R., & Xiong, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09030 · Cited by 54
“Interpreting attention as a form of explanation for neural NLP models has been a topic of debate (Jain and Wal-lace 2019; Serrano and Smith 2019; Wiegreffe and Pinter 2019).”
BackgroundOrder in the Court: Explainable AI Methods Prone to DisagreementPreprint
Neely, M., Schouten, S. F., Bleeker, M., & Lučić, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03287 · Cited by 55
Artificial Intelligence for Colonoscopy: Past, Present, and Future
Tavanapong, W., Oh, J., Riegler, M. A., Khaleel, M., Mittal, B., & de Groen, P. C. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3160098 · Cited by 53
“However, the effectiveness of the attention-based interpretation has not yet been proven [182], [183].”
BackgroundReconstruction of lossless molecular representations from fingerprints
Ucak, U. V., Ashyrmamatov, I., & Lee, J. (2023) · Journal of Cheminformatics · doi:10.1186/s13321-023-00693-0 · Cited by 53
A Deformable Attention Network for High-Resolution Remote Sensing Images Semantic Segmentation
Zuo, R., Zhang, G., Zhang, R., & Jia, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3119537 · Cited by 52
Li, Z., Fang, J., Wang, S., Zhang, L., Chen, Y., & Pian, C. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac037 · Cited by 52
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
Wu, S., Chen, H., Quan, X., Wang, Q., & Wang, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.471 · Cited by 53
On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs
Calderon, N., & Reichart, R. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.29 · Cited by 52
DISSECT: Disentangled Simultaneous Explanations via Concept TraversalsPreprint
Ghandeharioun, A., Kim, B., Li, C., Jou, B., Eoff, B. D., & Picard, R. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15164 · Cited by 52
“[1, 30, 61, 70, 84]) and determining when attention can be used as an explanation [84].”
BackgroundDesigning accessible, explainable AI (XAI) experiences
Wolf, C. T., & Ringland, K. E. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386302 · Cited by 51
Open Source Intelligence for Malicious Behavior Discovery and Interpretation
Huang, Y.-T., Lin, C., Guo, Y.-R., Lo, K.-C., Sun, Y. S., & Chen, M. C. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3119008 · Cited by 51
Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based GamesPreprint
Xu, Y., Fang, M., Chen, L., Du, Y., Zhou, J. T., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11655 · Cited by 51
“Although there are controversial points about whether attention values are explainable when being applied to text [25, 51], we believe our attention mechanism is beneficial for interpretability.”
BackgroundLiu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00474-9 · Cited by 50
Ferrer Florensa, A., Almagro Armenteros, J. J., Kaas, R. S., Clausen, P. T. L. C., Nielsen, H., Rost, B., & Aarestrup, F. M. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag129 · Cited by 50
Rethinking Interpretability in the Era of Large Language Models
Singh, C., Inala, J. P., Galley, M., Caruana, R., & Gao, J. (2024) · arXiv · doi:10.48550/arxiv.2402.01761 · Cited by 50
“Besides feature attributions, attention mechanisms within an LLM offer another avenue for visualizing token contributions to an LLM generation 61 , though their faithfulness/effectiveness remains unclear 62 .”
BackgroundDale, D. C., Voita, E., Lam, J., Hansanti, P., Ropers, C., Kalbassi, E., Gao, C., Barrault, L., & Costa‐jussà, M. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.42 · Cited by 50
When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
Petrov, A., Torr, P. H. S., & Bibi, A. (2023) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2310.19698 · Cited by 49
“An open question for language model interpretability is whether attention is sufficient for explainability (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundOn the Faithfulness of Vision Transformer Explanations
Wu, J., Kang, W., Tang, H., Hong, Y., & Yan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01040 · Cited by 49
“However, whether these interpretation maps are faithful to the model’s actual behavior remains a matter of debate [16, 24, 51].”
BackgroundMethodWright, A. P., Shaikh, O., Park, H., Epperson, W., Ahmed, M., Pinel, S., Chau, D. H., & Yang, D. (2021) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3449280 · Cited by 35
“However, the capacity for these methods to explain model predictions may differ across tasks [58].”
BackgroundAuto-Parsing Network for Image Captioning and Visual Question Answering
Yang, X., Gao, C., Zhang, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00220 · Cited by 49
“Specifically, when setting Q, K, V to the same value, such mechanism is called self-attention or non-local convolution [62, 64].”
BackgroundLearning to Estimate Shapley Values with Vision TransformersPreprint
Covert, I., Kim, C., & Lee, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.05282 · Cited by 49
“Recent work has disputed the role of attention as an indicator of feature importance [51, 29, 63, 10], and we find in our experiments that attention is a poor proxy for the effect of removing features from a model.”
BackgroundXAI in the Context of Predictive Process Monitoring: An Empirical Analysis Framework
Elkhawaga, G., Abu-Elkheir, M., & Reichert, M. (2022) · Algorithms · doi:10.3390/a15060199 · Cited by 48
Evolving Attention with Residual ConvolutionsPreprint
Wang, Y., Yang, Y., Bai, J., Zhang, M., Bai, J., Yu, J., Zhang, C., Huang, G. M., & Tong, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12895 · Cited by 48
“Moreover, there is a debate on whether or not the intermediate representations offered by attention mechanisms are useful to explain the reasons for a model’s prediction (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).”
BackgroundApproximating How Single Head Attention LearnsPreprint
Snell, C., Zhong, R., Klein, D., & Steinhardt, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07601 · Cited by 48
“For example, in text classification, while standard models typically attend to salient (high gradient influence) words (Serrano and Smith, 2019a), recent work constructs accurate models that attend to irrelevant words instead (Wiegreffe and Pinter, 2019a; Pruthi et al., 2020).”
Rajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., & Wang, F. (2024) · Patterns · doi:10.1016/j.patter.2023.100913 · Cited by 47
“114 In general, these methods suffer from only working due to careful model design and are limited to providing explanations only on specific modalities.”
BackgroundReinforcement Learning Based Path Exploration for Sequential Explainable Recommendation
Li, Y., Chen, H., Li, Y., Li, L., Yu, P. S., & Xu, G. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3237741 · Cited by 47
“[33], [34], [35], [36] However, there are different opinions on whether the attention mechanism could be used as a way to explain data [37], [38], [39].”
BackgroundNeuro-Symbolic Interpretable Collaborative Filtering for Attribute-based Recommendation
Zhang, W., Yan, J., Wang, Z., & Wang, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512042 · Cited by 47
“Therefore, the explainability of attention mechanism is somewhat questionable [12, 33].”
BackgroundExplainable depression symptom detection in social media
Bao, E., Pérez, A., & Parapar, J. (2024) · Health Information Science and Systems · doi:10.1007/s13755-024-00303-9 · Cited by 46
“There is an ongoing discussion in the community about the utility of attention weights for interpretability purposes, with some research advocating its benefits [60, 61], while others point out its constraints [62, 63].”
BackgroundOrmerod, M., del Rincón, J. M., & Devereux, B. J. (2021) · JMIR Medical Informatics · doi:10.2196/23099 · Cited by 46
“While this approach is intuitive, there is still an ongoing debate about the extent to which the attention mechanism can be used to interpret a model’s decision-making process [28,29].”
BackgroundToward Practical Usage of the Attention Mechanism as a Tool for Interpretability
Tutek, M., & Šnajder, J. (2022) · IEEE Access · doi:10.1109/access.2022.3169772 · Cited by 46
An Explainable Data-Driven Framework for Fuel Cell Aging Prediction Under Dynamic Condition
Xie, R., Li, C., Ma, R., Xu, L., & Zhou, X. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3199935 · Cited by 46
“Besides, it is proved that attention can provide certain explanations to the NN model [24].”
BackgroundOmics Data and Data Representations for Deep Learning-Based Predictive Modeling
Tsimenidis, S., Vrochidou, Ε., & Papakostas, G. A. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232012272 · Cited by 46
“The degree to which the attention mechanisms provide real explainability is contested [164], but in any case, identifying the regions of a sequence to which the model assigned the greatest amount of importance, can be helpful, perhaps with an expert evaluating the findings, or with a literature…”
BackgroundAmpel, B., Samtani, S., Zhu, H., & Chen, H. (2024) · MIS Quarterly · doi:10.25300/misq/2023/17316 · Cited by 46
Zhao, Y., & Bethard, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.429 · Cited by 47
“Serrano and Smith (2019), Jain and Wallace (2019) and Wiegreffe and Pinter (2019) found many redundancies in the attention of sequence-to-sequence models, suggesting that attention may encode knowledge in many ways.”
BackgroundStructured Attention for Unsupervised Dialogue Structure Induction
Qiu, L., Zhao, Y., Shi, W., Liang, Y., Shi, F., Yuan, T., Yu, Z., & Zhu, S. C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.148 · Cited by 46
The Explanation Game: Towards Prediction Explainability through Sparse Communication
Treviso, M. V., & Martins, A. F. T. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.10 · Cited by 46
“As in Jain and Wallace (2019) and Wiegreffe and Pinter (2019), our classifier C is an RNN with attention.”
BackgroundMethodInfluentialLearning to Explain: Generating Stable Explanations Fast
Situ, X., Zukerman, I., Paris, C. L., Maruf, S., & Haffari, G. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.415 · Cited by 46
“Attention-based methods (Wiegreffe and Pinter, 2019) can only be applied to Transformer-based models (Vaswani et al., 2017), and their effectiveness is questionable (Jain and Wallace, 2019; Ser-rano and Smith, 2019).”
MethodThe Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
Fomicheva, M., Lertvittayakumjorn, P., Zhao, W., Eger, S., & Gao, Y. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.17 · Cited by 46
Deep Transfer Learning With Self-Attention for Industry Sensor Fusion Tasks
Zhang, Z., Farnsworth, M., Song, B., Tiwari, D., & Tiwari, A. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3186505 · Cited by 46
From the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer
Pavlopoulos, J., Laugier, L., Xenos, A., Sorensen, J. S., & Androutsopoulos, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.259 · Cited by 46
Leveraging redundancy in attention with Reuse TransformersPreprint
Bhojanapalli, S., Chakrabarti, A., Veit, A., Łukasik, M., Jain, H., Liu, F., Chang, Y.-W., & Kumar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06821 · Cited by 46
“…has focused on analyzing and interpreting the attention scores computed by trained Transformer models (Serrano & Smith, 2019; Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Clark et al., 2019; Rogers et al., 2020) to gain insight into how attention is useful for inference in specific…”
MethodAbstract, Rationale, Stance: A Joint Model for Scientific Claim Verification
Zhang, Z., Li, J., Fukumoto, F., & Ye, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.290 · Cited by 46
“Rationale Regularization (RR): The attention scores have been used for interpretability in NLP tasks (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Sun and Lu, 2020).”
MethodYang, K., Xu, Y., Zou, P., Ding, H., Zhao, J., Wang, Y., & Xie, B. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25667 · Cited by 45
“For the sake of rigor in the medical area, we also add the KerPrinta− permuting the weights of time-aware attention and global knowledge integration attention (Wiegreffe and Pinter 2019).”
Multi-task learning for toxic comment classification and rationale extraction
Nelatoori, K. B., & Bindu, K. H. (2022) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00726-4 · Cited by 45
“However, Wiegreffe and Pinter (2019) recently called explainability effectiveness into question by pointing out that attention distributions are inconsistent with the significance of input units as measured by gradient-based methods.”
BackgroundExplainability in Graph Neural Networks: An Experimental SurveyPreprint
Li, P., Yang, Y., Pagnucco, M., & Song, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.09258 · Cited by 46
“For more ob-jective evaluation, Faithfullness was proposed and widely used [Lakkaraju et al. , 2019; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020; Schlichtkrull et al. , 2021].”
MethodResultInfluentialKabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44
“…as attention visualization [146], prompt attribution [147], explanation with exemplar-based reasoning (e.g., in-context learning) [147], chain-of-thought prompting [148], and self-explaining models [149] are being investigated to provide a transparent view of the model’s internal logic [150,151].”
Relative Importance in Sentence Processing
Hollenstein, N., & Beinborn, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.19 · Cited by 44
“Recent research indicates that complementary attention distributions can lead to the same model prediction (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019) and that the removal of input tokens with large attention weights often does not lead to a change in the model’s prediction (Serrano and…”
BackgroundOn Exploring Attention-based Explanation for Transformer Models in Text Classification
Liu, S., Le, F., Chakraborty, S., & Abdelzaher, T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671639 · Cited by 44
“Also, recent studies have looked at directly utilizing attention weights as explanation methods [6]– [8], but with mixed results.”
ResultRationalizing Text Matching: Learning Sparse Alignments via Optimal Transport
Swanson, K., Yu, L., & Lei, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.496 · Cited by 43
Attention Can Reflect Syntactic Structure (If You Let It)
Ravishankar, V., Kulmizev, A., Abdou, M., Søgaard, A., & Nivre, J. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.264 · Cited by 43
Introducing NBEATSx to realized volatility forecasting
Souto, H. G., & Moradi, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122802 · Cited by 43
Shen, H., Huang, C.-Y., Wu, T., & Huang, T.-H. K. (2023) · Computer Supported Cooperative Work and Social Computing · doi:10.1145/3584931.3607492 · Cited by 42
“The majority of the proposed XAI approaches are commonly validated effectively using the “functionally-grounded” evaluation methods [28, 33, 79], which seek for automatic metrics (e.”
MethodPDiscoNet: Semantically consistent part discovery for fine-grained recognition
van der Klis, R., Alaniz, S., Mancini, M., Dantas, C. F., Ienco, D., Akata, Z., & Marcos, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00179 · Cited by 42
“Attention rollout [1, 9] is a popular way to understand whether attention modules can provide such explanations [19, 34].”
BackgroundToken Transformation Matters: Towards Faithful Post-Hoc Explanation for Vision Transformer
Wu, J., Duan, B., Kang, W., Tang, H., & Yan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01039 · Cited by 42
“Such a lack of transparency undermines the trustworthiness of decision-making processes [2, 16, 26, 38, 49].”
BackgroundMethodA Comprehensive Survey on Self-Interpretable Neural Networks
Ji, Y., Sun, Y., Zhang, Y., Wang, Z., Zhuang, Y., Gong, Z., Shen, D., Qin, C., Zhu, H., & Xiong, H. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3635153 · Cited by 43
“Studies raise concerns about attention weights as explanatory mechanisms, highlighting their misalignment with actual feature importance [120], [121], [122], [123].”
Model-agnostic vs. Model-intrinsic Interpretability for Explainable Product Search
Ai, Q., & Narayanan.R, L. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482276 · Cited by 42
“Examples of the first group including the examination of network neurons and layers [7, 22, 46, 53, 71], the use of attention networks [32, 63, 69], and the design of disentangled model structure and information representations [13, 17, 30, 40].”
BackgroundHassid, M., Peng, H. M., Rotem, D., Kasai, J., Montero, I., Smith, N. A., & Schwartz, R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.101 · Cited by 43
A graph multi-attention network for predicting airport delays
Zheng, H., Wang, Z., Zheng, C., Wang, Y., Fan, X., Cong, W., & Hu, M. (2023) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2023.103375 · Cited by 41
Applications of interpretable deep learning in neuroimaging: A comprehensive review
Munroe, L., da Silva, M., Heidari, F., Grigorescu, I., Dahan, S., Robinson, E. C., Deprez, M., & So, P. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00214 · Cited by 41
Chen, S., Liu, J., Yuan, H., Tao, Y., Xu, F., & Yang, L. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.125116 · Cited by 41
Predicting Future Laboratory Fault Friction Through Deep Learning Transformer Models
Wang, K., Johnson, C. W., Bennett, K. C., & Johnson, P. A. (2022) · Geophysical Research Letters · doi:10.1029/2022gl098233 · Cited by 43
Towards Explainable Evaluation Metrics for Natural Language GenerationPreprint
Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.11131 · Cited by 41
We need to talk about random seedsPreprint
Bethard, S. J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.13393 · Cited by 41
Interpretation of Time-Series Deep Models: A SurveyPreprint
Zhao, Z., Shi, Y., Wu, S., Yang, F., Song, W., & Liu, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14582 · Cited by 42
“Meanwhile, the proponents advocates that attention weights are capable of demonstrating meaningful interpretation, and the adversarial aspects of attention mechanisms does not necessarily hold due to the ambiguous definition on model explanation [108].”
BackgroundAttention in Large Language Models Yields Efficient Zero-Shot Re-Rankers
Chen, S. H., Gutiérrez, B. J., & Su, Y. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.02642 · Cited by 41
“While there exist controversial opinions on how reliable attention weights are for interpretability (Serrano & Smith, 2019; Wiegreffe & Pinter, 2019), previous work has shown the effectiveness of distilling cross-attention between documents and a reader model for training better retrievers (Izacard…”
Staying True to Your Word: (How) Can Attention Become Explanation?
Tutek, M., & Šnajder, J. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.17 · Cited by 40
Towards Interpreting BERT for Reading Comprehension Based QA
Ramnath, S., Nema, P., Sahni, D., & Khapra, M. M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.261 · Cited by 40
Fine-grained Classification of Political Bias in German News: A Data Set and Initial Experiments
Aksenov, D., Bourgonje, P., Zaczynska, K., Ostendorff, M., Schneider, J. M., & Rehm, G. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.13 · Cited by 40
“The question whether or not attention can be used for explaining a model’s prediction is still under discussion (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundInterpretation of Black Box NLP Models: A SurveyPreprint
Choudhary, S., Chatterjee, N., & Saha, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.17081 · Cited by 40
“However, these arguments were countered by (Wiegreffe and Pinter, 2019) and argued that model’s weight are learned in a unified manner with their parameters.”
BackgroundA neural network transformer model for composite microstructure homogenization
Pitz, E. J., & POCHIRAJU, K. V. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108622 · Cited by 39
“In contrast, in [41] the authors argue that attention can provide a plausible explanation for a model’s predictions, however, not “ one true, faithful interpretation of the link . . . between inputs and outputs ”.”
BackgroundResultOpen-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models
Marcos-Manchón, P., Alcover‐Couso, R., SanMiguel, J. C., & Martínez, J. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00883 · Cited by 39
“The use of attention matrices to relate semantic information to spatial layout draws inspiration from natural language processing, where word-specific attention has been shown to correlate with lexical attribution [5, 54].”
BackgroundThinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code
Paltenghi, M., & Pradel, M. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678712 · Cited by 39
“, by measuring the effectiveness of attention layers against other explanation techniques [32], [66].”
BackgroundMethodGDPR and Large Language Models: Technical and Legal Obstacles
Feretzakis, G., Vagena, E., Kalodanis, K., Peristera, P., Kalles, D., & Anastasiou, A. (2025) · Future Internet · doi:10.3390/fi17040151 · Cited by 38
On Measuring Faithfulness or Self-consistency of Natural Language Explanations
Parcalabescu, L., & Frank, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.329 · Cited by 38
“…methods correlate high attention weights with high feature importance, which is debated (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). iii) Perturbation methods like RISE (Petsiuk et al., 2018) and SHAP (Lundberg and Lee, 2017) compute importance scores…”
Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
Esslinger, K., Platt, R., & Amato, C. (2022) · arXiv · doi:10.48550/arxiv.2206.01078 · Cited by 38
“While the use of attention weights as a tool for explainability is still being studied (Jain & Wal-lace, 2019; Wiegreffe & Pinter, 2019), it does allow us to observe which observations the agent finds most valuable in its history.”
BackgroundMethodNatural Language Processing Methods for the Study of Protein–Ligand Interactions
Michels, J., Bandarupalli, R., Akbari, A. A., Le, T., Xiao, H. Y., Li, J., & Hom, E. F. Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01907 · Cited by 38
Wu, X., Li, L., Tao, X., Yuan, J., & Xie, H. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3537918 · Cited by 38
“The attention mechanism supervises any models to focus on specific parts of the input when making predictions or generating outputs and has boosted performance on factor explanation in various tasks [34], [35].”
Harnessing the linguistic signal to predict scalar inferences
Schuster, S., Chen, Y., & Degen, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.479 · Cited by 38
“Attention weight analyses have been successfully used for inspecting and debugging model decisions (e.g., Lee et al., 2017; Ding et al., 2017; Wiegreffe and Pinter, 2019; Vashishth et al., 2019; but see Serrano and Smith, 2019, and Jain and Wallace, 2019, for critical discussions of this approach).”
MethodHan, X., & Tsvetkov, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.374 · Cited by 38
A novel feature attention mechanism for improving the accuracy and robustness of runoff forecasting
Wang, H. E., Qin, H., Liu, G., Liu, S., Qu, Y., Wang, K., & Zhou, J. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.129200 · Cited by 38
Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars
Wen, K., Li, Y., Liu, B., & Risteski, A. (2023) · neural information processing systems · doi:10.52202/075280-1681 · Cited by 38
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.