Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
“Although numerous works have explained the use of transformers in NLP [193], [239], it remains a challenging subject to clearly explain why transformer works well on visual tasks.”
BackgroundA review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995
Explainable Recommendation: A Survey and New Perspectives
Zhang, Y., & Chen, X. (2020) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000066 · Cited by 755
A General Survey on Attention Mechanisms in Deep Learning
Brauwers, G., & Frasincar, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3126456 · Cited by 695
“Yet, in another paper titled “Attention is not not Explanation” [121], the claim that attention is not explanation is questioned by challenging the assumptions of the previous”
BackgroundResultInfluentialExplainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
“We first examine works that study vector representations to measure embedded knowledge.”
BackgroundAttention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 635
“Wiegreffe and Pinter [9] pushed the discussion further, providing experiments that demonstrate that creating an effective global adversarial attention model is much more difficult than creating a local one and that attention weights may contain information regarding feature importance.”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“However, this claim continues to be debated, because attention weights representing different explanations can produce similar final predictions [170, 355].”
BackgroundExplainability in Graph Neural Networks: A Taxonomic Survey
Yuan, H., Yu, H., Gui, S., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204236 · Cited by 583
“Good metrics should evaluate the results from the model’s perspective, such as whether the explanations are faithful to the model [84], [85].”
BackgroundERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
MERLOT: Multimodal Neural Script Knowledge Models
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 53
“While we do not claim that these attention weights provide a full explanation of the model behavior [51, 98], they do play some role in the model’s decision [116], and we find that our masking strategy improves performance on downstream tasks by around 1% (Table 4), versus a SpanBERT baseline [54].”
BackgroundExplainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
LineVul: a transformer-based line-level vulnerability prediction
Fu, M., & Tantithamthavorn, C. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528452 · Cited by 384
“…argued that the learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, while Wiegreffe et al. [61] argued that the accuracy/reliability of such attention weights could provide meaningful explanations, depending on the definition and the rigor…”
BackgroundResultAttention is Not Only a Weight: Analyzing Transformers with Vector Norms
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.574 · Cited by 374
“the models (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Pruthi et al., 2020; Vashishth et al., 2019).”
Explanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355
“Consequently, Wiegreff and Pinter [93] disproved this claim and argued that such a claim depends on one’s definition of explanation and that prior work against the effectiveness of attention maps for explanations does not disprove the usefulness of attention mechanisms for explainability.”
BackgroundInfluentialTransformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.446 · Cited by 353
“Also related are interpretability methods that explain predictions (Han et al., 2020; Wiegreffe and Pinter, 2019), however, our focus is entirely different: we do not interpret individual predictions, but aim to understand the mechanism of transformers.”
BackgroundTransformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2020) · arXiv · doi:10.48550/arxiv.2012.14913 · Cited by 353
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
“Interpreting attention on language sequences is a well-established area of research [12, 86, 89].”
BackgroundAttention in Psychology, Neuroscience, and Machine Learning
Lindsay, G. W. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00029 · Cited by 309
“However, caution should be applied when interpreting the outputs of attention as they may not always explain the behavior of the model as expected (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundAre NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.168 · Cited by 307
Function Vectors in Large Language Models
Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B., & Bau, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.15213 · Cited by 305
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
“Mainly, hybrid models based on Transformer, GATs, and Memory Networks have emerged for multimodal learning and several other application domains.”
BackgroundInfluentialComparative Layer-Wise Analysis of Self-Supervised Speech Models
Pasad, A., Shi, B., & Livescu, K. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10096149 · Cited by 105
“The relative unreliability of layer weights as an analysis tool is not too surprising and is similar to findings for attention weights in NLP models [25, 26].”
ResultExtraction of organic chemistry grammar from unsupervised learning of chemical reactions
Schwaller, P., Hoover, B., Reymond, J.-L., Strobelt, H., & Laino, T. (2021) · Science Advances · doi:10.1126/sciadv.abe4166 · Cited by 296
Tank, A., Covert, I., Foti, N., Shojaie, A., & Fox, E. B. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3065601 · Cited by 282
“Our finding that the IMV-LSTM performs poorly at this task is consistent with recent work suggesting that attention mechanisms are not indicative of feature importance [59].”
ResultThe role of machine learning in clinical research: transforming the future of evidence generation
Weissler, E. H., Naumann, T., Andersson, T., Ranganath, R., Elemento, O., Luo, Y., Freitag, D. F., Benoit, J., Hughes, M. C., Khan, F., et al. (2021) · Trials · doi:10.1186/s13063-021-05489-x · Cited by 280
Bastings, J., & Filippova, K. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.14 · Cited by 150
Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review
Kuznietsov, A., Gyevnar, B., Wang, C., Peters, S., & Albrecht, S. V. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3474469 · Cited by 144
“To better understand stake-holder requirements and to adapt explanations to the varied socio-technical interactions of the real world, it is crucial to develop methods that are rooted in actual problems and not merely motivated by a vague sense of need for safety and trustworthiness.”
Lv, H., Chen, J., Pan, T., Zhang, T., Feng, Y., & Liu, S. (2022) · Measurement · doi:10.1016/j.measurement.2022.111594 · Cited by 261
Attention Interpretability Across NLP Tasks
Vashishth, S., Upadhyay, S., Tomar, G. S., & Faruqui, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11218 · Cited by 63
“For single sequence tasks, we adopt the model architecture from Jain & Wallace (2019); Wiegreffe & Pinter (2019).”
MethodExplaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Han, X., Wallace, B. C., & Tsvetkov, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.492 · Cited by 104
TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
Bento, J., Saleiro, P., Cruz, A. F., Figueiredo, M. A. T., & Bizarro, P. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467166 · Cited by 243
“However, whether attention can in fact explain a model’s behavior is debatable and a known source of controversy in the ML community [31, 32, 33].”
BackgroundExploring the role of large language models in the scientific method: from hypothesis to discovery
Zhang, Y., Khan, S. A., Mahmud, A., Yang, H., Lavin, A., Levin, M. G., Frey, J. G., Dunnmon, J., Evans, J. A., Bundy, A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00019-5 · Cited by 242
“Althoughthefaithfulnessofattentionweightinterpretabilityhas been questioned in various cases 182,183 , it remains widely used in many AI-for-science applications 184 .”
A survey on interpretable reinforcement learning
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., & Liu, W. (2024) · Machine Learning · doi:10.1007/s10994-024-06543-w · Cited by 243
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
Wu, Z., Chen, Y., Kao, B., & Liu, Q. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.383 · Cited by 152
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 236
Learning to Faithfully Rationalize by Construction
Jain, S., Wiegreffe, S., Pinter, Y., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.409 · Cited by 136
Self-citationMambaOut: Do We Really Need Mamba for Vision?
Yu, W., & Wang, X. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00423 · Cited by 228
“It is noteworthy that papers such as “Attention is All You Need” [82], “Attention is NOT All You Need” [25], “At-tention is NOT Explanation” [43], and “Attention is NOT NOT Explanation” [87] have all been accepted to top-tier AI conferences, despite their differing claims.”
The black box problem revisited. Real and imaginary challenges for automated legal decision making
Brożek, B., Furman, M., Jakubiec, M., & Kucharzyk, B. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09356-9 · Cited by 115
“…an AI system (e.g., a machine learning model) may be con - sidered explainable if it can be understood by humans through an external, simplified representation (though explanatory value of internal components of the models is also discussed – see Jain and Wallace 2019 ; Wiegreffe and Pinter 2019 ).”
Is Attention Explanation? An Introduction to the Debate
Bibal, A., Cardon, R., Alfter, D., Wilkens, R., Wang, X., François, T., & Watrin, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.269 · Cited by 225
“In addition, they also object to the exclusive explanation argument that attention is " an explanation, not the explanation" (Wiegreffe and Pinter, 2019, p. 13).”
BackgroundInvestigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
“a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
BackgroundWhen BERT Plays the Lottery, All Tickets Are Winning
Prasanna, S., Rogers, A., & Rumshisky, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.259 · Cited by 131
Holzinger, A., Dehmer, M., Emmert-Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurisica, I., & Díaz-Rodríguez, N. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 213
“Even the best current machine learning models do not generalize well, have difficulty with small training datasets [10], and are sensitive to even small perturbations [11–13].”
BackgroundAI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Wortman Vaughan, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.8036d03b · Cited by 210
Post-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207
“This is more commonly known as faithfulness [39, 53, 94, 129] or sometimes idelity [97].”
BackgroundMethodResultInfluentialA Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“Attention’s explanatory power is questionable [180] as they lack an association with the attention weights and gradients mapping for generating faithful explanations.”
BackgroundLearning to Deceive with Attention-Based Explanations
Pruthi, D., Gupta, M., Dhingra, B., Neubig, G., & Lipton, Z. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.432 · Cited by 203
A Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing · doi:10.18653/v1/2020.aacl-main.46 · Cited by 203
“…work has widely used attention layers (Luo et al., 2018; Xie et al., 2017; Mullenbach et al., 2018) to enable feature importance explainability, the jury is still out as to how much explainability attention provides (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
BackgroundMethodBiomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
“Though controversial [78, 206], the attention mechanism [11] that is ubiquitously used in modern BQA systems provides at least some level of explainability.”
BackgroundExplainable Recommendation: A Survey and New Perspectives
Zhang, Y., & Chen, X. (2020) · doi:10.1561/9781680836592 · Cited by 199
A survey on the interpretability of deep learning in medical diagnosis
Teng, Q., Liu, Z., Song, Y., Han, K., & Lu, Y. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00960-4 · Cited by 198
“Recently, there have been some arguments about whether the attention mechanism is interpretable [47–49].”
BackgroundIntelligent financial fraud detection practices in post-pandemic era
Zhu, X., Ao, X., Qin, Z., Chang, Y., Liu, Y., He, Q., & Li, J. (2021) · The Innovation · doi:10.1016/j.xinn.2021.100176 · Cited by 197
Explainable Automated Fact-Checking: A Survey
Kotonya, N., & Toni, F. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.474 · Cited by 98
Do Feature Attribution Methods Correctly Attribute Features?
Zhou, Y., Booth, S., Ribeiro, M. T., & Shah, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i9.21196 · Cited by 103
CausaLM: Causal Model Explanation Through Counterfactual Language Models
Feder, A., Oved, N., Shalit, U., & Reichart, R. (2021) · Computational Linguistics · doi:10.1162/coli_a_00404 · Cited by 39
Is Attention Better Than Matrix Decomposition?
Geng, Z., Guo, M.-H., Chen, H., Li, X., Wei, K., & Lin, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04553 · Cited by 47
“Some work tries to interpret the attention mechanism by visualizing or attacking attention weights (Serrano & Smith, 2019; Jain & Wallace, 2019; Wiegreffe & Pinter, 2019), while others formulate attention into non-local operation (Wang et al., 2018) or diffusion models (Tao et al., 2018; Lu et al.,…”
BackgroundSelf-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 189
An emotion and cognitive based analysis of mental health disorders from social media data
Uban, A. S., Chulvi, B., & Rosso, P. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.05.032 · Cited by 187
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Ross, A., Marasović, A., & Peters, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.336 · Cited by 86
Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Mrini, K., Dernoncourt, F., Tran, Q., Bui, T., Chang, W. H., & Nakashole, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.65 · Cited by 180
“However, Wiegreffe and Pinter (2019) show through alternative tests that prior work does not discredit the usefulness of attention for interpretability.”
BackgroundExplanation in AI and law: Past, present and future
Atkinson, K., Bench-Capon, T., & Bollegala, D. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103387 · Cited by 178
“However, it has been shown that attention alone is inadequate as a form of explanation especially when the number of layers in a deep neural network increases and multiple nonlinear activation functions are used after an attention layer [116, 134].”
BackgroundFastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
Guo, H., Rajani, N., Hase, P., Bansal, M., & Xiong, C. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.808 · Cited by 55
“For example, previous work has considered interpreting predictions by constructing importance scores over the input tokens (Belinkov and Glass, 2019), attentionbased methods (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019), gradientbased methods (Simonyan et al.”
BackgroundBaur, L., Ditschuneit, K., Schambach, M., Kaymakci, C., Wollmann, T., & Sauer, A. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100358 · Cited by 175
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 172
“However, it is not clear if such mechanisms provide meaningful explanations of a model’s decisions (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Grimsley, Mayfield, & Bursten, 2020).”
BackgroundSelva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 169
“Restrictions apply. explanations on what the model deems relevant [199], [200], [201].”
BackgroundFruit ripeness classification: A survey
Rizzo, M., Marcuzzo, M., Zangari, A., Gasparetto, A., & Albarelli, A. (2023) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2023.02.004 · Cited by 167
A survey on aspect-based sentiment classification
Brauwers, G., & Frasincar, F. (2023) · ACM Computing Surveys · doi:10.1145/3503044 · Cited by 165
“Although, the usefulness of the interpretation of attention weights is controversial [97, 138, 214].”
BackgroundOn Identifiability in TransformersPreprint
Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., & Wattenhofer, R. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04211 · Cited by 170
“However, their identifiability affects the interpretability of the output, i.e., whether attention weights can provide the basis for explanatory insights on the model’s predictions (cf. Jain & Wallace (2019) and Wiegreffe & Pinter (2019)).”
BackgroundChen, V., Yang, M., Cui, W., Kim, J. S., Talwalkar, A., & Ma, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02359-7 · Cited by 115
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.523 · Cited by 161
“A popular way of explaining NLP models is to attribute importance weights to the input tokens, either using attention scores (Wiegreffe and Pinter, 2019) or by summarizing the model behavior on perturbed instances (e.”
BackgroundA Survey of Explainable Artificial Intelligence (XAI) in Financial Time Series Forecasting
Arsenault, P.-D., Wang, S., & Patenaude, J.-M. (2025) · ACM Computing Surveys · doi:10.1145/3729531 · Cited by 158
“If we extend the logic of the deinition of the "explanation" in [111], the attention weights could be reliable, depending on the deinition of the reliability.”
Rethinking pooling in graph neural networksPreprint
Mesquita, D. P. P., Junior, A. H. D. S., & Kaski, S. (2020) · Aaltodoc (Aalto University) · doi:10.48550/arxiv.2010.11418 · Cited by 156
“Regarding attention-based models, Wiegreffe and Pinter [42] show that attention weights usually do not provide meaningful explanations for predictions.”
BackgroundCausal Discovery from Temporal Data: An Overview and New Perspectives
Gong, C., Zhang, C., Yao, D., Bi, J., Li, W., & Xu, Y. (2024) · ACM Computing Surveys · doi:10.1145/3705297 · Cited by 154
“Due to space limitations, we leave details of statistical learning-based methods, including model-free and model-based methods, in supplementary materials.”
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.22 · Cited by 151
“However, interpreting attention patterns without understanding the attended-to embeddings, or relying on attention alone as a faithful explanation, can lead to faulty interpretations (Brunner et al., 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Li et al., 2019).”
BackgroundOperA: Attention-Regularized Transformers for Surgical Phase Recognition
Czempiel, T., Paschali, M., Ostler, D., Kim, S. T., Busam, B., & Navab, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87202-1_58 · Cited by 150
“However, this assumption has been challenged [24] suggesting that each work should define their notion of explanation since it could be dependent on the task at hand.”
BackgroundTeach Me to Explain: A Review of Datasets for Explainable Natural Language ProcessingPreprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12060 · Cited by 149
“Highlights are usually evaluated following two criteria: (i) plausibility, according to humans, how 138 well a highlight supports a predicted label [130, 28], and (ii) faithfulness or fidelity, how accurately a 139 highlight represents the model’s decision process [5, 124].”
BackgroundSelf-citationAttnLRP: Attention-Aware Layer-Wise Relevance Propagation for TransformersPreprint
Achtibat, R., Hatefi, S. M. V., Dreyer, M., Jain, A., Wiegand, T., Lapuschkin, S., & Samek, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05602 · Cited by 147
“Although attention maps capture parts of the data distribution, they lack class specificity and do not provide a meaningful interpretation of the final prediction (Wiegreffe and Pinter, 2019).”
BackgroundA Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145
“For instance, evidence supporting both the self-attention interpretability [82, 83] and noninterpretability [84] have been published.”
BackgroundExplainability in AI-based behavioral malware detection systems
Galli, A., Gatta, V. L., Moscato, V., Postiglione, M., & Sperlí, G. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103842 · Cited by 141
Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation
Dankers, V., Lucas, C., & Titov, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.252 · Cited by 139
Aligning Faithful Interpretations with their Social Attribution
Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00367 · Cited by 73
“When formalizing the desired properties of a quality interpretation of a model or a decision, the NLP community has settled on the key property of faithfulness (Lipton, 2018; Herman, 2017; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020), or how ‘‘accurately’’ the interpretation represents the true reasoning pro…”
BackgroundMultistage attention network for multivariate time series prediction
Hu, J., & Zheng, W. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.11.060 · Cited by 135
The state of human-centered NLP technology for fact-checking
Das, A., Liu, H., Kovatchev, V., & Lease, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103219 · Cited by 136
“Attention-based intelligibility Despite the debate about attention being a reliable intelligibility method (Bibal et al., 2022; Jain & Wallace, 2019; Serrano & Smith, 2019; Wiegreffe & Pinter, 2019), it remains a popular method in existing deep neural network approaches in fact-checking.”
Learning Transactional Behavioral Representations for Credit Card Fraud Detection
Xie, Y., Liu, G., Yan, C., Jiang, C., Zhou, M., & Li, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3208967 · Cited by 135
“Overall, since the researchers give different opinions on the interpretability of an attention module [69], [70], [71], and our purpose is to learn effective transactional behavioral representations to better detect frauds, we mainly focus on the following aspects.”
BackgroundGuo, R., Liu, H., Xie, G., Zhang, Y. M., & Liu, D. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3181692 · Cited by 132
“The weight coefficients calculated by the attention mechanism are used as the explanations to identify the importance of the input information [26].”
MethodAdaptive fusion of multi-modal remote sensing data for optimal sub-field crop yield prediction
Mena, F., Pathak, D., Najjar, H., Sanchez, C., Helber, P., Bischke, B., Habelitz, P., Miranda, M., Siddamsetty, J., Nuske, M., et al. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114547 · Cited by 132
Visualizing Transformers for NLP: A Brief Survey
Brasoveanu, A. M. P., & Andonie, R. (2020) · 2020 24th International Conference Information Visualisation (IV) · doi:10.1109/iv51561.2020.00051 · Cited by 84
“Wiegreffe [47] explaining attention permutation performance permutation scatterplots multiple line charts”
BackgroundEXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124
“Although novel learning architectures are constantly developed toward better performances in most different domains [7], [8], [100], their understanding has been primarily ignored [101].”
BackgroundThe Role of Explainability in Assuring Safety of Machine Learning in Healthcare
Jia, Y., McDermid, J. A., Lawton, T., & Habli, I. (2022) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3171314 · Cited by 123
“Although the attention-mechanism could render an attention map to provide some insights or intuitive feeling of the model, it is important to note that the interpretation of attention as explanation is currently the subject of debate [25], [26].”
BackgroundConformal Prediction for Natural Language Processing: A Survey
Campos, M., Farinhas, A., Zerva, C., Figueiredo, M. A. T., & Martins, A. F. T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00715 · Cited by 125
“…2019; Desai and Durrett, 2020), evaluation is limited and sometimes affected by data contamination (Sainz et al., 2023; Golchin and Surdeanu, 2024), explanations are often unreliable (Zhao et al., 2024; Wiegreffe and Pinter, 2019), and models often exhibit undesired biases (Gallegos et al., 2024).”
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
“may not always suggest faithful explanations, and an overreliance on them for interpretation can be hazardous [138], [261], [300].”
BackgroundRodis, N., Sardianos, C., Grammatikis, P. I. R., Sarigiannidis, P. G., Varlamis, I., & Papadopoulos, G. T. (2024) · IEEE Access · doi:10.1109/access.2024.3467062 · Cited by 123
“6 illustrates indicative examples of representative literature approaches.”
BackgroundTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 120
Measuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.804 · Cited by 119
“Analysis of NLP Models Structural tests for analyzing models’ internals include probing (Tenney et al., 2019) and attention analysis (Jain and Wal-lace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Tutek and Snajder, 2020).”
MethodSelf-citation
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.