Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 401–500 of 1,008
Pafla, M., Larson, K. S., & Hancock, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642934 · Cited by 24
Flexible Instance-Specific Rationalization of NLP Models
Chrysostomou, G., & Αλέτρας, Ν. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21298 · Cited by 24
“…from Y , we consider four divergence measures ( ∆ ), also previously used in literature (Robnik-ˇSikonja and Kononenko 2008; Jain and Wallace 2019; Wiegreffe and Pinter 2019): Kullback Leibler (KL) : A non-symmetric divergence measure of how a particular distribution divergences from a…”
MethodA survey of feature attribution techniques in explainable AI: taxonomy, analysis and comparison
Nazir, A., Evangelista, E. D. L., Bukhari, S. M. S., & Sharma, R. (2025) · Annals of Mathematics and Computer Science · doi:10.56947/amcs.v28.547 · Cited by 23
Uban, A. S., Chulvi, B., & Rosso, P. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-04431-1_13 · Cited by 23
Parameter-Efficient Tuning Makes a Good Classification Head
Yang, Z., Ding, M., Guo, Y., Lv, Q., & Tang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.514 · Cited by 23
Sang, Y., & Bao, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.117 · Cited by 23
“Since model interpretability is important in finance and economics, we follow (Wiegreffe and Pinter, 2019) to validate the usefulness of our model’s contextual attention layer for model interpretability.”
Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers
Held, W., & Yang, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.177 · Cited by 23
Effective Attention Sheds Light On Interpretability
Sun, K., & Marasović, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.361 · Cited by 23
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention
Ophir, Y., Tikochinski, R., Klomek, A. B., & Reichart, R. (2021) · Clinical Psychological Science · doi:10.1177/21677026211022013 · Cited by 23
“Researchers can therefore use these different weights and offer explanations regarding the specific features that drove the model in its decision-making process (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).”
BackgroundCounterfactual Evaluation for Explainable AIPreprint
Ge, Y., Liu, S., Li, Z., Xu, S., Geng, S., Li, Y., Tan, J., Sun, F., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01962 · Cited by 23
Tailoring Self-Rationalizers with Multi-Reward Distillation
Ramnath, S., Joshi, B., Hallinan, S., Lu, X., Li, L. H., Chan, A., Hessel, J., Choi, Y., & Ren, X. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.02805 · Cited by 23
MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Dong, S., Hu, J., Zhang, M., Yin, M., Fu, Y., & Qian, Q. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2508.18264 · Cited by 23
“For example, VisionZip (Yang et al., 2025a) and FastV (Chen et al., 2024a) These methods rely on vision information and may miss query-related semantics (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).”
Powell, C., & Riccardi, A. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00940-w · Cited by 23
The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
Kahardipraja, P., Achtibat, R., Wiegand, T., Samek, W., & Lapuschkin, S. (2025) · neural information processing systems · doi:10.52202/085713-3940 · Cited by 22
Enhancing Attention Models via Multi-head Collaboration
Wang, H., & Tu, M. (2020) · International Conference on Asian Language Processing · doi:10.1109/ialp51396.2020.9310460 · Cited by 22
“[12] and [13] provide insights about the interpretability of attention mechanism.”
BackgroundContext-Aware and Time-Aware Attention-Based Model for Disease Risk Prediction With Interpretability
Zhang, X., Qian, B., Li, Y., Cao, S., & Davidson, I. A. K. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130171 · Cited by 22
“In contrast, Wiegreffe et al. [39] challenged many of the assumptions underlying the prior work, arguing that such a claim depends on one’s definition of explanation: plausible or faithful explanations (or both).”
ResultSupervised Contrastive Learning for Interpretable Long-Form Document Matching
Jha, A., Rakesh, V., Chandrashekar, J., Samavedhi, A., & Reddy, C. K. A. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3542822 · Cited by 22
“While some works do demonstrate how attention scores can be used to provide insights into attention-based models [13, 39, 42], we deine interpretability as the weighted similarity scores between diferent sections and diferent chunks of text in a document.”
BackgroundThrough the looking glass: Deep interpretable dynamic directed connectivity in resting fMRI
Mahmood, U., Fu, Z., Ghosh, S., Calhoun, V. D., & Plis, S. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119737 · Cited by 22
“As our experiments show, our attention module is indeed eliable per the definitions and potential issues discussed by Jain and allace (2019) and Wiegreffe and Pinter (2019) .”
MethodER-Test: Evaluating Explanation Regularization Methods for Language Models
Joshi, B., Chan, A., Liu, Z., Nie, S., Sanjabi, M., Firooz, H., & Ren, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.242 · Cited by 22
“, 2017); attention-based, which computes rationales via F(xi)’s attention weights (Ding and Koehn, 2021; Wiegreffe and Pinter, 2019); and learned, which trains a model to compute rationales w.”
BackgroundA multi-facet analysis of BERT-based entity matching models
Paganelli, M., Tiano, D., & Guerra, F. (2023) · The VLDB Journal · doi:10.1007/s00778-023-00824-x · Cited by 22
“Never-theless, this is a controversial point since other papers, as [47, 49], demonstrated that low correlations happen only in limited conditions.”
Topic-Centric Explanations for News Recommendation
Liu, D., Greene, D., Li, I., Jiang, X., & Dong, R. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3680295 · Cited by 23
“However, Wiegreffe and Pinter [46] claim that, despite the fact that explanations provided by attention mechanisms are not always faithful, in practice, this does not invalidate the plausibility of using attention as an explanation.”
Attention Is All You Need for LLM-Based Code Vulnerability Localization
Li, Y., Li, X., Wu, H., Zhang, Y., Cheng, X., Liu, Y., Xu, F., & Zhong, S. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3725564 · Cited by 22
“There has been significant research [9], [17], [26], [37], [44], [48] on leveraging attention for the interpretability of language models.”
Telling BERT’s Full Story: from Local Attention to Global Aggregation
Pascual, D., Brunner, G., & Wattenhofer, R. P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.9 · Cited by 22
Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modeling
Harbecke, D., & Alt, C. (2020) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2020.acl-srw.16 · Cited by 22
Understanding Patterns of Anorexia Manifestations in Social Media Data with Deep Learning
Uban, A. S., Chulvi, B., & Rosso, P. (2021) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/2021.clpsych-1.24 · Cited by 22
“Recent studies (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019) have questioned whether attention mechanisms necessarily help with the interpretability of neural network predictions.”
BackgroundSelf-Verification Improves Few-Shot Clinical Information Extraction
Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., & Poon, H. (2023) · arXiv · doi:10.48550/arxiv.2306.00024 · Cited by 21
“In-terpretability has taken many forms in NLP, including posthoc feature importance (Lundberg & Lee, 2017; Ribeiro et al., 2016), intrinsically interpretable models (Rudin, 2019; Singh et al., 2022a), and visualizing model intermediates, e.g . attention (Wiegreffe & Pinter, 2019).”
BackgroundUnifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI
Dembinsky, D., Lucieri, A., Frolov, S., Najjar, H., Watanabe, K., & Dengel, A. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2506.15408 · Cited by 21
McLevey, J., Crick, T., Browne, P., & Durant, D. (2022) · Canadian Review of Sociology/Revue canadienne de sociologie · doi:10.1111/cars.12378 · Cited by 21
Wang, M., Wang, D., Xiang, Y., Liang, Y., Xia, R., Yang, J., Xu, F., & Huang, X. (2023) · Frontiers in Marine Science · doi:10.3389/fmars.2023.1112065 · Cited by 21
“Many studies have discussed the model interpretation ability of the attention weight distribution for neural network models based on attention mechanisms (Pruthi et al., 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
BackgroundNatural Language Processing and Text Mining (Turning Unstructured Data into Structured)
Bagheri, A., Giachanou, A., Mosteiro, P. J., & Verberne, S. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_5 · Cited by 21
Explainability Approach-Based Series Arc Fault Detection Method for Photovoltaic Systems
Wang, Y., Zhou, J., Paul, K. C., Zhao, T., & Sheng, D. (2024) · IEEE Access · doi:10.1109/access.2024.3381530 · Cited by 21
“The interpretability of the attention mechanism enables one to grasp the decision-making behavior of the network model [37], [38]: the key feature frequency bands of PV arc faults are extracted by visualizing feature weights, which improves the reliability of the network.”
MethodLocal interpretation of deep learning models for Aspect-Based Sentiment Analysis
Lam, S., Liu, Y., Broers, M., van der Vos, J., Frăsincar, F., Boekestijn, D., & van der Knaap, F. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109947 · Cited by 21
Measuring and Improving Faithfulness of Attention in Neural Machine Translation
Moradi, P., Kambhatla, N., & Sarkar, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.243 · Cited by 22
“…the model’s prediction at the time step t (Tu et al., 2016; Mi et al., 2016; Liu et al., 2016; Wang et al., 2016; Lee et al., 2017; Ding et al., 2017; Ghaeini et al., 2018 (Jain and Wal-lace, 2019; Wiegreffe and Pinter, 2019) have used a binary measure: either attention is faithful or it is not.”
BackgroundMethodOn the Explainability of Automatic Predictions of Mental Disorders from Social Media Data
Uban, A. S., Chulvi, B., & Rosso, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80599-9_27 · Cited by 21
Gupta, V., Bhat, R. A., Ghosal, A., Shrivastava, M., Singh, M., & Srikumar, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00482 · Cited by 22
“More effort is needed to either make the model more interpretable or at least generate explanations for the predictions (Feng et al., 2018; Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; DeYoung et al., 2020; Paranjape et al., 2020).”
BackgroundEnjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience
Chrysostomou, G., & Αλέτρας, Ν. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.645 · Cited by 21
“typically been used as a tool for evaluating explanation faithfulness generated by attention (Kennedy et al., 2020; Wiegreffe and Pinter, 2019; Pruthi et al., 2020; Ghorbani et al., 2019) while others used auxiliary objectives to improve the faithful-”
BackgroundMethodGeneric Temporal Reasoning with Differential Analysis and Explanation
Feng, Y., Zhou, B. Z., Wang, H., Jin, H., & Roth, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.671 · Cited by 21
Liu, J., Chen, Y., Liu, K., Jia, Y., & Sheng, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.229 · Cited by 21
“Considering that unsupervised attention may not always learn a good pattern (Wiegreffe and Pinter, 2019), we devise a “trial-and-error” approach to guide the learning.”
MethodFrom Kepler to Newton: Explainable AI for SciencePreprint
Li, Z., Ji, J., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12210 · Cited by 22
“Notable examples of model-intrinsic methods include linear regression [18], decision tree [19, 20] and attention mechanism [13, 21, 22, 23], whose explanations are regression coefficients, decision paths and attention weights, respectively.”
MethodEvaluating Human Alignment and Model Faithfulness of LLM RationalePreprint
Fayyaz, M., Yin, F., Sun, J., & Peng, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.00219 · Cited by 21
“Rationales 3 , i.e., tokens of the input text that are most influential to the models’ predictions, are widely studied in the NLP community prior to the era of LLMs to interpret model predictions (Lei et al., 2016; DeYoung et al., 2019; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020).”
BackgroundMethodLearning to Attribute with AttentionPreprint
Cohen-Wang, B., Chuang, Y.-S., & Mądry, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.13752 · Cited by 21
“However, prior work has cast doubt on the reliability of attention weights as explanations [JW19; WP19; SS19].”
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20
“…mechanisms, it may eventually be possible to use these models to formally quantify aspects of the summarization process in unprecedented ways, though currently there is debate about whether these mechanisms can be relied on for interpretability (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundReaders’ affect: predicting and understanding readers’ emotions with deep learning
Anoop, K. R., Deepak, P., Abraham, S. S., Lajish, V. L., & Gangan, M. P. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00614-2 · Cited by 20
“Ablation study over attention layer—uniform attention as the adversary There is no point in analyzing model behavior to study the impact of emotion words and named entities if attention does not have a reasonable influence on prediction [56].”
BackgroundMethodInfluentialA novel physically interpretable end-to-end network for stress monitoring in laser shock peening
Qin, R., Zhang, Z., Huang, J., Du, Z., Xiang, X., Wang, J., Wen, G., & He, W. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.104060 · Cited by 20
Prototype Learning for Medical Time Series Classification via Human–Machine Collaboration
Xie, J., Wang, Z., Yu, Z., Ding, Y., & Guo, B. (2024) · Sensors · doi:10.3390/s24082655 · Cited by 20
Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
Chen, H., Feng, S., Ganhotra, J., Wan, H., Gunasekara, C., Joshi, S., & Ji, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.306 · Cited by 20
Rebane, J. T., Samsten, I., Pantelidis, P., & Papapetrou, P. (2021) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms52027.2021.00025 · Cited by 20
Combining deep ensemble learning and explanation for intelligent ticket management
Zicari, P., Folino, G., Guarascio, M., & Pontieri, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117815 · Cited by 20
ToxCCIn: Toxic Content Classification with InterpretabilityPreprint
Xiang, T., MacAvaney, S., Yang, E., & Goharian, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01328 · Cited by 20
“On the one hand, they usually need postprocessing to perform interpretation, which is less straightforward and intuitive compared to LR; on the other hand, there are still debates about whether attention mechanisms can produce meaningful explanations (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019).”
BackgroundTowards Explainable Evaluation Metrics for Machine TranslationPreprint
Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.13041 · Cited by 20
“Natural language explanations are explanations in free text form (e.g. Wiegreffe and Maraso-vic 2021).”
BackgroundSCAR: Shapley Credit Assignment for More Efficient RLHF
Cao, M., Zhang, S., Chang, X.-W., & Precup, D. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1795 · Cited by 19
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
Xu, J., & Durrett, G. (2021) · doi:10.18653/v1/2021.acl-long.539 · Cited by 19
“Although attention has been questioned (Jain and Wallace, 2019), it still has some value as an explanation technique (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019).”
BackgroundExplainable Neural Rule Learning
Shi, S., Xie, Y., Wang, Z., Ding, B., Li, Y., & Zhang, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512023 · Cited by 19
“However, attention may not be sufficient as an explanation [48].”
BackgroundComputing and evaluating saliency maps for image classification: a tutorial
Gomez, T., & Mouchère, H. (2023) · Journal of Electronic Imaging · doi:10.1117/1.jei.32.2.020801 · Cited by 19
iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summaries
Coscia, A. J., Holmes, L., Morris, W., Choi, J. S., Crossley, S. A., & Endert, A. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645142 · Cited by 19
“6.1, there is on-going debate whether attention weights in transformers can be used as a reliable source of interpretation for model performance [1, 11, 26, 61].”
BackgroundInfluentialHong, J., Park, K. H., & Pavlic, T. P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00481 · Cited by 20
“In particular, the debate over what degree of interpretability can be ascribed to attention weights over input tokens still needs to be settled to help meet the need for interpretability of attention mechanisms [1, 9, 40, 71, 86].”
BackgroundStructurizing Misinformation Stories via Rationalizing Fact-Checks
Jiang, S., & Wilson, C. (2021) · doi:10.18653/v1/2021.acl-long.51 · Cited by 19
“Although there have been debates on the properties achieved by attention-based explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019), rationales extracted by straightforward rules on attention weights were demonstrated as comparable to human-generated rationales…”
BackgroundKDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation
Xue, M., Song, J., Wang, X., Chen, Y., Wang, X., & Song, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/444 · Cited by 19
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning
Gupta, V., Zhang, S., Vempala, A., He, Y., Choji, T., & Srikumar, V. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.231 · Cited by 19
Interpretability of BERT Latent Space through Knowledge Graphs
Anelli, V. W., Biancofiore, G. M., Bellis, A., Noia, T. D., & Sciascio, E. D. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557617 · Cited by 19
“Many works inspect the model through attention heads [26, 4, 20, 10, 30], fine-tune it to solve interpretable tasks [11], or even modify the pre-training procedure for the same goals [22, 32].”
BackgroundInterpretability Illusions in the Generalization of Simplified ModelsPreprint
Friedman, D., Lampinen, A. K., Dixon, L., Chen, D., & Ghandeharioun, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.03656 · Cited by 19
“Others have proposed tests for scenarios where attention can serve as a valid explanation (Wiegreffe & Pinter, 2019).”
BackgroundAdvancing Explainable AI Toward Human-Like Intelligence: Forging the Path to Artificial BrainPreprint
Zhou, Y., & Jiang, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06673 · Cited by 19
“Real-world XAI applications demand not only technically accurate but also comprehensible explanations, particularly in decision-making contexts (Wang et al., 2022; Jacovi & Goldberg, 2020; Wiegreffe & Pinter, 2019; Pope et al., 2019; Yuan et al., 2022).”
BackgroundSource Attribution in Retrieval-Augmented Generation
Nematov, I., Kalai, T., Kuzmenko, E., Fugagnoli, G., Sacharidis, D., Hose, K., & Sagi, T. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19099-4_23 · Cited by 18
Customer Acquisition via Explainable Deep Reinforcement Learning
Song, Y., Wang, W., & Yao, S. (2024) · Information Systems Research · doi:10.1287/isre.2022.0529 · Cited by 18
Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods
Yu, H., Atanasova, P., & Augenstein, I. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.444 · Cited by 18
“This finding aligns with Wiegreffe and Pinter (2019)’s argu-ment, that attention weights pose a meaningful impact on the model’s prediction and are important for understanding a model’s inner workings.”
ResultTowards Stable and Explainable Attention Mechanisms
Hu, L., Wang, X., Liu, Y., Liu, N., Huai, M., Sun, L., & Wang, D. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3538583 · Cited by 18
Conversational Explanations: Discussing Explainable AI with Non-AI Experts
Zhang, T., Zhang, M., Low, W. Y., Yang, X. J., & Li, B. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712143 · Cited by 18
“Specifically, we select Grad-CAM, Integrated Gradients, and SHAP from gradient-based methods, as well as LIME from surrogate methods, to evaluate the effectiveness of different conversational evaluation systems.”
A Comprehensive Guide to Interpretable AI-Powered Discoveries in Astronomy
Lieu, M. (2025) · Universe · doi:10.3390/universe11060187 · Cited by 18
“…guarantee causality, the complex non-linear transformations after the attention layer can influence the final prediction, and the attention weights can sometimes be manipulated without significantly changing the model’s output, whilst equally significant are challenges against those claims [103].”
What Makes a Scientific Paper be Accepted for Publication?
Fytas, P., Rizos, G., & Specia, L. (2021) · doi:10.18653/v1/2021.cinlp-1.4 · Cited by 18
Automatic Rule Induction for Efficient Semi-Supervised Learning
Pryzant, R., Yang, Z., Yi‐chong, X., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.3 · Cited by 18
Bridging Self-Attention and Time Series Decomposition for Periodic Forecasting
Jiang, S., Syed, T., Zhu, X., Levy, J., Aronchik, B., & Sun, Y. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557077 · Cited by 18
“However, such learned inherent scores are difficult to examine [18] and sometimes even suspicious [3, 30].”
BackgroundVisFIS: Visual Feature Importance Supervision with Right-For-The-Right-Reason Objectives
Ying, Z., Hase, P., & Bansal, M. (2022) · neural information processing systems · doi:10.52202/068431-1241 · Cited by 18
Ni, J., Chen, Z., Cheng, W. D., Zong, B., Song, D., Liu, Y., Zhang, X., & Chen, H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482355 · Cited by 18
“Although these methods can highlight some pertinent parts to the models’ predictions, attention cannot represent class-wise patterns that explicitly link inputs to outputs, and its usefulness depends on the types of explanations to seek [49].”
BackgroundTowards Ethics by Design in Online Abusive Content DetectionPreprint
Kiritchenko, S., & Nejadgholi, I. (2020) · arXiv (Cornell University) · doi:10.4224/40002027 · Cited by 18
Systematic Generalization and Emergent Structures in Transformers Trained on Structured TasksPreprint
Li, Y., & McClelland, J. L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00400 · Cited by 18
“However, it is often debated to what extent attention weights afford model interpretability in these settings (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Vashishth et al., 2019), especially considering head redundancy and the difficulty in correctly attributing relevance over…”
BackgroundWord Importance Explains How Prompts Affect Language Model OutputsPreprint
Hackmann, S., Mahmoudian, H., Steadman, M., & Schmidt, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03028 · Cited by 18
“There have been multiple works in the area of explainability in NLP, see Danilevsky et al. [2021], Danilevsky et al. [2020a], and Wiegreffe and Pinter [2019].”
BackgroundDeep learning for automatic ICD coding: Review, opportunities and challenges
Li, X., Zhang, Y., Hou, X., Wang, S., & Lin, H. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103187 · Cited by 17
Explainable time-series forecasting with sampling-free SHAP for Transformers
Hertel, M., Pütz, S., Mikut, R., Hagenmeyer, V., & Schäfer, B. (2026) · Nature Communications · doi:10.1038/s41467-026-73243-5 · Cited by 17
“We introduce SHAPformer, a time-series forecasting model that combines the strong predictive performance of Transformers with the explainability of SHAP.”
AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruption
Wang, Y., Geng, R., Chen, Y., & Jia, J. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00198 · Cited by 17
“By calculating the average attention weight of X i across all tokens in Y , A TTN ( X || Y ; X i , Y ) can be used to measure the overall contribution of X i to the generation of Y [57], [58].”
Fouladvand, S., Talbert, J., Dwoskin, L. P., Bush, H., Meadows, A. L., Peterson, L. E., Mishra, Y. R., Roggenkamp, S. K., Wang, F., Kavuluru, R., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3265920 · Cited by 17
“While attention weights cannot be considered as causal inferences and it is still unclear whether attentions can be used to explain deep learning models [35], [36], [37],”
BackgroundEvaluating Self-attention Interpretability Through Human-Grounded Experimental Protocol
Bhan, M., Achache, N., Legrand, V., Blangero, A., & Chesneau, N. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_2 · Cited by 17
Towards a neuro-symbolic cycle for human-centered explainability
Mileo, A. (2025) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240740 · Cited by 17
Wang, Z., Lin, Z., Lin, W., Yang, M., Zeng, M., & Tan, K. C. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3664098 · Cited by 17
““Plausibility” refers to how the interpretation convinces humans [42], [43], [44].”
What-if I ask you to explain: Explaining the effects of perturbations in procedural text
Rajagopal, D., Tandon, N., Clark, P., Dalvi, B. B., & Hovy, E. H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.300 · Cited by 17
IST-Unbabel 2021 Submission for the Explainable Quality Estimation Shared Task
Treviso, M., Guerreiro, N. M., Rei, R., & Martins, A. F. T. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.14 · Cited by 17
“Attention weights provide plausible, but not always faithful, explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundInvestigating the Learning Behaviour of In-Context Learning: A Comparison with Supervised Learning
Wang, X., Wang, Y., Xu, C., Geng, X., Zhang, B., Tao, C., Rudzicz, F., Mercer, R. E., & Jiang, D. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230559 · Cited by 17
Multi-Aspect Transfer Learning for Detecting Low Resource Mental Disorders on Social Media
Uban, A. S., Chulvi, B., & Rosso, P. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4xkujrraiz4o · Cited by 17
“Nevertheless, recent studies have shown the limitations of using attention analysis for interpretability (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019).”
BackgroundSonkar, S., Waters, A. E., & Baraniuk, R. G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.608 · Cited by 17
Huo, Y., Wong, D. F., Ni, L. M.-S., Chao, L. S., & Zhang, J. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106389 · Cited by 17
A Cross-lingual Comparison of Human and Model Relative Word Importance
Morger, F., Brandl, S., Beinborn, L., & Hollenstein, N. (2022) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 17
Revisiting Attention Weights as Explanations from an Information Theoretic PerspectivePreprint
Wen, B., Subbalakshmi, K. P., & Yang, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.07714 · Cited by 18
“As mentioned earlier, previous works [5, 24] assess the explainability of attention mechanisms by linking the attention scores to the importance of each token in the input sequence.”
InfluentialPoser: Unmasking Alignment Faking LLMs by Manipulating Their InternalsPreprint
Clymer, J., Juang, C., & Field, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.05466 · Cited by 17
“[Wiegreffe and Pinter, 2019] find that transformer attention scores are sometimes interpretable.”
BackgroundTowards reliability and interactive debugging for large language models
Paranjape, B. (2024) · ResearchWorks at the University of Washington (University of Washington) · Cited by 17
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision
Brahman, F., Shwartz, V., Rudinger, R., & Choi, Y. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i14.17492 · Cited by 31
Memory for prediction: A Transformer-based theory of sentence processing
Ryu, S. H., & Lewis, R. L. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104670 · Cited by 16
Measuring Faithful and Plausible Visual Grounding in VQA
Reich, D., Putze, F., & Schultz, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.206 · Cited by 16
“Additionally, attention mechanisms have been a topic of debate regarding the faithfulness of their explanation (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundKim, J., Chung, S., & Chi, S. (2024) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-14273 · Cited by 16
Gaze-infused BERT: Do human gaze signals help pre-trained language models?
Wang, B., Liang, B., Zhou, L., & Xu, R. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09725-8 · Cited by 16
“Additionally, there is ongoing debate regarding whether the intermediate representations provided by attention mechanisms contribute to explaining the reasons behind model predictions, as indicated by studies such as those conducted by [6] and [7].”
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.