Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 301–400 of 1,008
Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosis
Gimeno-Gómez, D., Botelho, C., Pompili, A., Abad, A., & Martínez-Hinarejos, C.-D. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3539845 · Cited by 37
Interpretable Machine Learning: A Comprehensive Review of Foundations, Methods, and the Path Forward
Fridkin, S., & Bendersky, M. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70075 · Cited by 38
Sycophancy Hides Linearly in the Attention Heads
Genadi, R. A., Nwadike, M. S., Mukhituly, N., Hiraoka, T., AlQuabeh, H., & Inui, K. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.324 · Cited by 37
Explainable AI for Mental Disorder Detection on Social Media: A Survey and Outlook
Ibrahimov, Y., Anwar, T., & Yuan, T. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3688443 · Cited by 37
Multi-Task Attentive Residual Networks for Argument Mining
Galassi, A., Lippi, M., & Torroni, P. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3275040 · Cited by 37
“mechanism is really helpful for improving the explainability of neural models is still a matter of discussion [28], [81], [82].”
BackgroundAttention Calibration for Transformer in Neural Machine Translation
Lu, Y., Zeng, J., Zhang, J., Wu, S., & Li, M. (2021) · doi:10.18653/v1/2021.acl-long.103 · Cited by 37
Explainable Clinical Decision Support from Text
Feng, J., Shaib, C., & Rudzicz, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.115 · Cited by 37
“The wide application of attention mechanisms has led to an ongoing debate over whether attention can be used as explanation (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundMethodInfluentialWang, N., Chen, M., & Subbalakshmi, K. P. (2020) · medRxiv · doi:10.1101/2020.06.24.20139592 · Cited by 38
Sparse Attention with Linear UnitsPreprint
Zhang, B., Titov, I., & Sennrich, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07012 · Cited by 37
“The explanatory power of standard attention weights is hotly debated (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019).”
BackgroundInterpreting and Improving Large Language Models in Arithmetic CalculationPreprint
Zhang, W., Wan, C., Zhang, Y., Cheung, Y., Tian, X., Shen, X., & Ye, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.01659 · Cited by 37
“Explanations for model behavior can easily be misleading or non-rigorous (Bolukbasi et al., 2021; Wiegreffe & Pinter, 2019).”
BackgroundMindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata PoisoningPreprint
Wang, Z., Du, H., Shi, G., Zhang, J., Cheng, H., Yao, Y., Guo, K., & Li, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.20412 · Cited by 37
“While a single attention head can be manipulated without affecting the model’s output [58], analyzing aggregate attention signals across layers and heads still provides a stable indicator of decision-shaping influence [59], [60].”
Zimmermann, J., Demartsev, V., Averly, B., Dhanjal‐Adams, K. L., Duteil, M., Gall, G. E. C., Faiß, M., Johnson-Ulrich, L., Stowell, D., Manser, M. B., et al. (2025) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70218 · Cited by 35
Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection
Attanasio, G., Nozza, D., Pastor, E., & Hovy, D. (2022) · doi:10.18653/v1/2022.nlppower-1.11 · Cited by 36
“Attention There is an open debate of whether attention is explanation or not (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020).”
BackgroundSEAT: Stable and Explainable Attention
Hu, L., Liu, Y., Liu, N., Huai, M., Sun, L., & Wang, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26517 · Cited by 36
Feldhus, N., Hennig, L., Nasert, M. D., Ebert, C., Schwarzenberg, R., & Mller, S. (2023) · doi:10.18653/v1/2023.nlrse-1.4 · Cited by 36
“Natural language explanations (NLEs), on the other hand, exceed other explainability methods in plausibility (Lei et al., 2016; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020), accessibil-”
BackgroundSIGMORPHON–UniMorph 2023 Shared Task 0: Typologically Diverse Morphological Inflection
Goldman, O., Batsuren, K., Khalifa, S., Arora, A., Nicolai, G., Tsarfaty, R., & Vylomova, E. (2023) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2023.sigmorphon-1.13 · Cited by 36
“They did not get into debate on whether soft attention between model’s hidden states is a good explanation (Jain and Wal-lace, 2019; Wiegreffe and Pinter, 2019), but rather applied a hard attention mechanism directly over static character representations.”
BackgroundIntelligent Video Analytics for Human Action Recognition: The State of Knowledge
Kulbacki, M., Segen, J., Chaczko, Z., Rozenblit, J. W., Kulbacki, M., Klempous, R., & Wojciechowski, K. W. (2023) · Sensors · doi:10.3390/s23094258 · Cited by 35
“> 6,000,000 400 Kinetics 700 [256] 650,000 700”
Topics to Avoid: Demoting Latent Confounds in Text Classification
Kumar, S., Wintner, S., Smith, N. A., & Tsvetkov, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1425 · Cited by 35
“Recently, there has been a debate on whether attention can be used to explain model decisions (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), we thus present additional analysis of our proposed method based on saliency maps (Ding et al.”
MethodMulti-Document Summarization with Determinantal Point Process Attention
Perez-Beltrachini, L., & Lapata, M. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12522 · Cited by 35
“Although model predictions cannot be attributed to attention weights (Jain & Wallace, 2019), Wiegreffe and Pinter (2019) empirically show that these capture some notion of token importance.”
BackgroundMethodAttributing Fair Decisions with Attention Interventions
Mehrabi, N., Gupta, U., Morstatter, F., Steeg, G. V., & Galstyan, A. (2022) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2022.trustnlp-1.2 · Cited by 35
A Survey on Explainable Fake News Detection
MISHIMA, K., & Yamana, H. (2022) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2021edr0003 · Cited by 36
ProtoTEx: Explaining Model Decisions with Prototype Tensors
Das, A., Gupta, C., Kovatchev, V., Lease, M., & Li, J. J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.213 · Cited by 35
Explainability in Practice: A Survey of Explainable NLP Across Various DomainsPreprint
Mohammadi, H., Bagheri, R. A., Giachanou, A., & Oberski, D. L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.00837 · Cited by 35
“A further caveat is that attention-weight visualization can mislead: in sentiment analysis, weights could highlight neutral tokens (e.g., “the”), downplaying salient words like “hate” [63, 64].”
Ikhwantri, F., Putra, J. W. G., Yamada, H., & Tokunaga, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103195 · Cited by 34
Nguyen, B., & Chioua, M. (2024) · Control Engineering Practice · doi:10.1016/j.conengprac.2024.105988 · Cited by 34
Yu, L., Tang, L., & Mu, L. (2025) · Sensors · doi:10.3390/s25133952 · Cited by 34
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Hense, J., Idaji, M. J., Eberle, O., Schnake, T., Dippel, J., Ciernik, L., Buchstab, O., Möck, A., Klauschen, F., & Müller, K. (2024) · neural information processing systems · doi:10.52202/079017-0266 · Cited by 34
Schrödinger's tree—On syntax and neural language models
Kulmizev, A., & Nivre, J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796788 · Cited by 34
“…recent years, due to (1) a lack of input token identifiability in deep Transformer models (Brunner et al. 2019; Abnar and Zuidema 2020) and (2) an unclear correspondence between a model’s attention patterns and its outputs (Jain and Wallace 2019; Wiegreffe and Pinter 2019; Serrano and Smith 2019).”
BackgroundHuman Rationales as Attribution Priors for Explainable Stance Detection
Jayaram, S., & Allaway, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.450 · Cited by 33
“However, recent studies have argued for a more nuanced understanding of faithfulness that prioritizes ‘explainable enough’ (Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020) and allows for the ‘best’ technique to vary by model, task, or input (Ghorbani et al., 2018).”
BackgroundM2: Mixed Models With Preferences, Popularities and Transitions for Next-Basket Recommendation
Peng, B., Ren, Z., Parthasarathy, S., & Ning, X. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3142773 · Cited by 34
“used to learn user embeddings as some other work also noticed [18], [36], [37].”
BackgroundMethodHow to Dissect a Muppet: The Structure of Transformer Embedding Spaces
Mickus, T., Paperno, D., & Constant, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00501 · Cited by 34
“Lastly, our approach provides some quantitative argument for the validity of attention-based studies (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Pruthi et al., 2020) and expands on earlier works looking beyond attention weights (Kobayashi et al.”
ResultExtractive Explanations for Interpretable Text Ranking
Leonhardt, J., Rudra, K., & Anand, A. (2022) · ACM Transactions on Information Systems · doi:10.1145/3576924 · Cited by 35
“However, recent studies showed that attention weights are not explanations [34, 82] and models are able to maintain the same prediction accuracy even in the absence of those tokens.”
BackgroundZhang, B., Vakanski, A., & Xian, M. (2023) · IEEE Access · doi:10.1109/access.2023.3298569 · Cited by 34
“However, there are still some limitations to the clinical meaning of these explanations and their acceptance [16], [17], [39].”
BackgroundChalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Αλέτρας, Ν., Androutsopoulos, I., & Malakasiotis, P. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2103.13084 · Cited by 34
“From a model-centric point of view, the main focus is to demystify a model’s inner workings, for example targeting selfattention mechanisms (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), and more recently Transformer-based language models (Clark et al.”
BackgroundShapTime: A General XAI Approach for Explainable Time Series Forecasting
Zhang, Y., Sun, Q., Qi, D., Liu, J., Ma, R., & Petrosian, O. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47721-8_45 · Cited by 33
KE-MLLM: A Knowledge-Enhanced Multi-Sensor Learning Framework for Explainable Fake Review Detection
Chen, J., Liu, J., Liang, Y., & Zhou, M. (2026) · Applied Sciences · doi:10.3390/app16062909 · Cited by 34
Optimizing Attention for Sequence Modeling via Reinforcement Learning
Fei, H., Zhang, Y., Ren, Y., & Ji, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3053633 · Cited by 33
“The attention interpretability can be expressed as follows: if the learned attention weights agree with natural measures of features importance, the alternative of counterfactual attention distribution will correspond-ingly change the model output distribution most [34]–[36].”
BackgroundNeely, M., Schouten, S. F., Bleeker, M., & Lučić, A. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220190 · Cited by 33
Representation learning for neural population activity with Neural Data TransformersPreprint
Ye, J., & Pandarinath, C. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.16.426955 · Cited by 33
Learned feature representations are biased by complexity, learning order, position, and more
Lampinen, A. K., Chan, S. C. Y., & Hermann, K. L. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.05847 · Cited by 33
“…(Adebayo et al., 2018, 2021; Bolukbasi et al., 2021; Friedman et al., 2023a,b; Ghorbani et al., 2019; Jain and Wallace, 2019; Wen et al., 2023; Wiegreffe and Pinter, 2019; Zhou et al., 2022), often suggesting that saliency or activations can be misleading about what a model is “doing”…”
BackgroundEmbedding deep networks into visual explanations
Qi, Z., Khorram, S., & Li, F. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2020.103435 · Cited by 32
T3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP
Li, R., Xiao, W., Wang, L., Jang, H., & Carenini, G. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.26 · Cited by 32
“Similarly, despite some limitations regarding the explanatory capabilities of the attention mechanism (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), the visualization of its weights has also been shown to”
BackgroundLogic Traps in Evaluating Attribution Scores
Ju, Y., Zhang, Y., Zhao, Y., Jiang, Z., Liu, K., & Zhao, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.407 · Cited by 32
Xu, Z., Zeng, H., Tan, J., Fu, Z., Zhang, Y., & Ai, Q. (2023) · ACM Transactions on Information Systems · doi:10.1145/3605357 · Cited by 32
“Works in CV and NLP [35, 36, 42, 51, 62, 83] also propose to measure faithfulness by the overlap of important inputs determined by the two models, e.g., important input text tokens and pixels.”
BackgroundInfluentialMulti-step-ahead and interval carbon price forecasting using transformer-based hybrid model
Wang, Y., Zhong, W., Xiaoyi, W., & Kang, X. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-29196-z · Cited by 32
“In multi-head attention, the Softmax activation function is used to calculate the attention score equation (12), and Softmax cannot assign exactly zero scores because all scores sum to one, which reduces the attention allocation in the relevant time steps (Wiegreffe and Pinter 2019) and extracts information from the i…”
BackgroundMethodMayya, V., Kamath, S. S., Krishnan, G. S., & Gangavarapu, T. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.01.013 · Cited by 32
Dai, T., Maher, K., & Perzan, Z. (2025) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.132606 · Cited by 31
“Furthermore, research on transformers like those employed by the ViT3d has demonstrated that attention mechanisms within these models may also learn hierarchical information [Jain and Wallace, 2019, Jawahar et al., 2019, Wiegreffe and Pinter, 2019].”
Detecting mental disorder on social media: A ChatGPT-augmented explainable approach
Belcastro, L., Cantini, R., Marozzo, F., Talia, D., & Trunfio, P. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100321 · Cited by 31
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
Ji, T., Jain, S., Ferdman, M., Milder, P. A., Schwartz, H. A., & Balasubramanian, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.363 · Cited by 32
Are Prompt-based Models Clueless?
Kavumba, P., Ryo, T., & Oda, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.166 · Cited by 31
Video Influencers: Unboxing the MystiquePreprint
Rajaram, P., & Manchanda, P. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3752107 · Cited by 31
MATE: Multi-view Attention for Table Transformer EfficiencyPreprint
Eisenschlos, J. M., Gor, M., Müller, T., & Cohen, W. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04312 · Cited by 32
“It is important to note that in this context we are not measuring attention as an explanation method (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundFooling Explanations in Text ClassifiersPreprint
Ivankay, A., Girardi, I., Marchiori, C., & Frossard, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.03178 · Cited by 31
“However recent work has questioned their faithfulness and plausibility (Serrano & Smith, 2019; Jain & Wallace, 2019; Wiegr-effe & Pinter, 2019) by proving that they often do not highlight input components that are most important to a DNN decision.”
Interpretable Text Embeddings and Text Similarity Explanation: A Survey
Opitz, J., Moeller, L., Michail, A., Padó, S., & Clematide, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1135 · Cited by 30
A little goes a long way: Improving toxic language classification despite data scarcity
Juuti, M., Gröndahl, T., Flanagan, A., & Asokan, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.269 · Cited by 29
Diagnosing AI Explanation Methods with Folk Concepts of Behavior
Jacovi, A., Bastings, J., Gehrmann, S., Goldberg, Y., & Filippova, K. (2023) · Journal of Artificial Intelligence Research · doi:10.1145/3593013.3593993 · Cited by 30
A Fine-grained Interpretability Evaluation Benchmark for Neural NLP
Wang, L., Shen, Y., Peng, S., Zhang, S., Xiao, X., Liu, H., Tang, H., Chen, Y., Wu, H., & Wang, H. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.6 · Cited by 30
A Theoretical Framework for AI Models Explainability with Application in Biomedicine
Rizzo, M., Veneri, A., Albarelli, A., Lucchese, C., Nobile, M. S., & Conati, C. (2023) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb56990.2023.10264877 · Cited by 30
“While formal proofs are still missing in current literature, a number of tests for faithfulness have been recently proposed [26; 9; 10; 11].”
BackgroundResultSaranti, A., Pfeifer, B., Gollob, C., Stampfer, K., & Holzinger, A. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1554 · Cited by 30
“…(like PointTransformer Zhao et al., 2021) that go beyond the well-known differences between explanation and attention (Jain & Wallace, 2019; Serrano & Smith, 2019; Wiegreffe & Pinter, 2019) that showed the necessity for more sophisticated explanation methodologies like the LRP (Ali et al., 2022).”
Hierarchical Interpretation of Neural Text Classification
Yan, H., Gui, L., & He, Y. (2022) · Computational Linguistics · doi:10.1162/coli_a_00459 · Cited by 29
“Wiegreffe and Pinter (2019) argued the validity of the claim in prior work (Jain and Wallace 2019) and proposed alternative experimental design to test when/whether attention can be used as explanation.”
BackgroundInterpretation-based Code Summarization
Geng, M., Wang, S., Dong, D., Wang, H., Cao, S., Zhang, K., & Jin, Z. G. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00026 · Cited by 29
“Our results are also consistent with the previous studies [66], [67] which show that merely relying on the attention score may not obtain promising interpretation results in some cases.”
ResultUnderstanding Interlocking Dynamics of Cooperative RationalizationPreprint
Yu, M., Zhang, Y., Chang, S., & Jaakkola, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13880 · Cited by 29
“In addition, recent works [7, 20, 35, 38, 44] also find that the same prediction on an input could be generated by totally different attentions, which limits its applicability to explaining neural predictions.”
BackgroundSelfExplain: A Self-Explaining Architecture for Neural Text ClassifiersPreprint
Rajagopal, D., Balachandran, V., Hovy, E. H., & Tsvetkov, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12279 · Cited by 29
“In natural language applications, feature attribution based on attention scores (Xu et al., 2015) has been the predominant method for developing inherently interpretable neural classifiers.”
MethodBrain and Cognitive Science Inspired Deep Learning: A Comprehensive Survey
Zhang, Z., Ding, X., Liang, X., Zhou, Y., Qin, B., & Liu, T. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3527551 · Cited by 28
“Transformer and Pre-trained Models: In the realm of brain-inspired interpretive studies on Transformers, considerable debate persists around whether attention weights exhibit the same patterns as human attention behavior [196], [197].”
Leveraging explainable AI for reliable prediction of nuclear power plant severe accident progression
Joo, S., Lee, Y., Song, S.-H., Song, K., Seo, M. R., Kim, S. J., & Lee, J. I. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111307 · Cited by 28
Multimodal foundation transformer models for multiscale genomics
Khan, S. A., Martinez‐de‐Morentin, X., Alsabbagh, A. R., Maillo, A., Lagani, V., Gómez-Cabrero, D., Lehmann, R., & Tegner, J. N. (2025) · Nature Methods · doi:10.1038/s41592-025-02918-6 · Cited by 29
“Although attention scores offer insights into protein language models by connecting residue contacts to attention patterns, relying solely on attention for interpretation is debated due to ongoing questions about its interpretability 51 .”
Mylonas, N., Mollas, I., & Tsoumakas, G. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00962-4 · Cited by 28
“Extracting information from the attention module of transformers has been a popular method for interpreting their decisions [15, 16], especially before its criticism [5, 17].”
Where is my attention? An explainable AI exploration in water detection from SAR imagery
Chen, L., Cai, X., Li, Z., Jin, X., & Ai, J. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103878 · Cited by 28
Learning to Scaffold: Optimizing Model Explanations for Teaching
Fernandes, P., Treviso, M., Pruthi, D., Martins, A., & Neubig, G. (2022) · neural information processing systems · doi:10.52202/068431-2617 · Cited by 28
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3035 · Cited by 27
RankingSHAP - Faithful Listwise Feature Attribution Explanations for Ranking Models
Heuss, M., de Rijke, M., & Anand, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729971 · Cited by 27
“An alternative notion of faithfulness is based on an information-theoretic notion of feature importance [45, 53].”
To what extent do human explanations of model behavior align with actual model behavior?
Prasad, G., Nie, Y., Bansal, M., Jia, R., Kiela, D., & Williams, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.1 · Cited by 27
Translational NLP: A New Paradigm and General Principles for Natural Language Processing Research
Newman-Griffis, D. R., Lehman, J. F., Rosé, C. P., & Hochheiser, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.325 · Cited by 27
On Sample Based Explanation Methods for NLP: Faithfulness, Efficiency and Semantic Evaluation
Zhang, W., Huang, Z., Zhu, Y., Ye, G., Cui, X., & Zhang, F. (2021) · doi:10.18653/v1/2021.acl-long.419 · Cited by 27
“, 2017), attention-based (Clark et al., 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), as well as sample-based (Koh and Liang, 2017; Yeh et al.”
BackgroundMethodRethinking data-driven networking with foundation models: challenges and opportunities
Le, F., Srivatsa, M., Ganti, R., & Sekar, V. (2022) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3563766.3564109 · Cited by 27
“1140094 [88] Sarah Wiegreffe and Yuval Pinter.”
On the Impact of Temporal Concept Drift on Model Explanations
Zhao, Z., Chrysostomou, G., Bontcheva, K., & Αλέτρας, Ν. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.298 · Cited by 27
Through the looking glass: evaluating post hoc explanations using transparent models
Velmurugan, M., Ouyang, C., Sindhgatta, R., & Moreira, C. (2023) · International Journal of Data Science and Analytics · doi:10.1007/s41060-023-00445-1 · Cited by 27
“While attention mechanisms are sometimes used as explanation and have been shown to be successful in some domains [53], the validity of attention as explanation is still debated [20, 31, 54].”
Lai, V., Cai, Z., & Tan, C. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1046 · Cited by 27
Normalized Attention Without Probability CagePreprint
Richter, O., & Wattenhofer, R. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09561 · Cited by 27
TENT: Tensorized Encoder Transformer for Temperature ForecastingPreprint
Bilgin, O., Mąka, P., Vergutz, T., & Mehrkanoon, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14742 · Cited by 27
“The attention weights have been previously used for feature selection purposes [33], as well as explainability [34].”
MethodA Theoretical Framework for AI-driven data quality monitoring in high-volume data environmentsPreprint
Bangad, N. J., Jayaram, V., Krishnappa, M. S., Banarse, A. R., Bidkar, D. M., Nagpal, A., & Parlapalli, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08576 · Cited by 27
“This layer employs techniques such as SHAP (SHapley Additive exPlanations) values and attention mechanisms to provide interpretable quality assessments [57,58].”
MethodSQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0
Lee, G., Hwang, S. W., & Cho, H. (2020) · Language Resources and Evaluation · Cited by 26
Adversarial Infidelity Learning for Model Interpretation
Liang, J., Bai, B., Cao, Y., Bai, K., & Wang, F. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403071 · Cited by 26
“• Combinatorial shortcuts problem: the entries of themask may not select good features, but rather act as additional features themselves for better approximation performances [19, 39], because it is a function of all the input features.”
BackgroundA First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
Rao, V. N., Zhen, X., Hovsepian, K., & Shen, M. (2021) · doi:10.18653/v1/2021.maiworkshop-1.4 · Cited by 26
The Taxonomy of Writing Systems: How to Measure How Logographic a System Is
Sproat, R., & Gutkin, A. (2021) · Computational Linguistics · doi:10.1162/coli_a_00409 · Cited by 26
InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States
Beigi, M., Shen, Y., Yang, R., Lin, Z., Wang, Q., Mohan, A., He, J., Jin, M., Lu, C., & Huang, L. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.751 · Cited by 25
Explainable artificial intelligence for molecular design in pharmaceutical research
Lamens, A., & Bajorath, J. (2026) · Chemical Science · doi:10.1039/d5sc08461j · Cited by 25
“Color intensity scales with the relative magnitude of weights.”
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
Ye, X., Nair, R., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.447 · Cited by 25
“However, it avoids the pitfalls of directly inspecting attention (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019) by running an integrated gradients-like procedure over all the attention links within transformers, yielding attribu-”
BackgroundReview of Natural Language Processing in Pharmacology
Trajanov, D., Trajkovski, V., Dimitrieva, M., Dobreva, J., Jovanovik, M., Klemen, M., Žagar, A., & Robnik‐Šikonja, M. (2023) · Pharmacological Reviews · doi:10.1124/pharmrev.122.000715 · Cited by 25
“However, multiple attention heads may be difficult to comprehend, and the alignment between attention explanations and the underlying model behavior (i.e. actual explanations) is questionable [90, 229].”
Attention weights accurately predict language representations in the brain
Lamarre, M., Chen, C., & Deniz, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.330 · Cited by 25
Less is More: Attention Supervision with Counterfactuals for Text Classification
Choi, S. L., Park, H., Yeo, J., & Hwang, S.-W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.543 · Cited by 25
“Meanwhile, (Wiegreffe and Pinter, 2019) develops diagnostics to decide when attention is good enough as explanation.”
BackgroundInfluentialTranslation Error Detection as Rationale Extraction
Fomicheva, M., Specia, L., & Αλέτρας, Ν. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.327 · Cited by 25
On Robustness and Bias Analysis of BERT-Based Relation Extraction
Li, L., Chen, X., Ye, H., Bi, Z., Deng, S., Zhang, N., & Chen, H. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-6471-7_4 · Cited by 25
“Note that, unlike previous counterfactual data augmentation approaches, such as [39,5], our method was a simple, yet effective, automatic algorithm that can be applied to other tasks (e.”
A disentangled linguistic graph model for explainable aspect-based sentiment analysis
Mei, X., Zhou, Y., Zhu, C., Wu, M., Li, M., & Pan, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110150 · Cited by 25
Exploring Social Posterior Collapse in Variational Autoencoder for Interaction ModelingPreprint
Tang, C., Zhan, W., & Tomizuka, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00298 · Cited by 25
“However, we need to be more careful when comparing the importance of two agents based on non-zero attention weights [36, 37].”
BackgroundA Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
Paul, D., Chowdhury, A., Xiong, X., Chang, F.-J., Carlyn, D., Stevens, S., Provost, K. L., Karpatne, A., Carstens, B. C., Rubenstein, D., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.04157 · Cited by 25
“There has been an ongoing debate on whether attention offers faithful interpretation (Wiegreffe & Pinter, 2019; Jain & Wallace, 2019; Serrano & Smith, 2019; Bibal et al., 2022).”
BackgroundChan, Y. H., Girish, D., Gupta, S., Xia, J., Kasi, C., He, Y., Wang, C., & Rajapakse, J. C. (2025) · NeuroImage · doi:10.1016/j.neuroimage.2025.121422 · Cited by 24
IMBERT: Making BERT Immune to Insertion-based Backdoor Attacks
He, X., Wang, J., Rubinstein, B. I. P., & Cohn, T. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.25 · Cited by 24
Language model and its interpretability in biomedicine: A scoping review
Lyu, D., Wang, X., Chen, Y., & Wang, F. (2024) · iScience · doi:10.1016/j.isci.2024.109334 · Cited by 26
Towards Balancing Preference and Performance through Adaptive Personalized Explainability
Silva, A., Tambwekar, P., Schrum, M., & Gombolay, M. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3635000 · Cited by 24
“…such as developing neural network models that can be readily interpreted [107], pursuing natural language generation for explanations [14, 29, 72, 109], presenting feature-importance maps for input samples [13, 35, 48, 83, 101, 102, 109], presenting relevant training data [11, 12, 54, 94], and…”
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.