Attention is not not Explanation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Sarah Wiegreffe, Yuval Pinter. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [101]
The cited source says what the claim attributes to it.
In the citing paper
“Attention-based explanations remain contested (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019).”
From The Attribution-Compression Frontier in Retrieval-Augmented Generation (2026) · cites this paper as Wiegreffe and Pinter, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“In order to make our many points in a succinct fashion as well as follow the conclusions drawn by Jain and Wallace, we focus on experimenting with the binary classification subset of their tasks, and on models with an LSTM architecture (Hochreiter and Schmidhuber, 1997), the only one the authors make firm conclusions…”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMethodInfluentialRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitability for providing explanations for model predictions is a topic of high interest (Xu et al., 2015; Rocktäschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019).”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 3×
“The tasks are as follows: to predict positive or negative sentiment from sentences (SST) and movie reviews (IMDB), to predict the topic of news articles as either baseball (neg.) or hockey (pos.) in 20 NEWSGROUPS and either world (neg.) or business (pos.) in AG NEWS, to predict whether a patient is diagnosed with diab…”
MethodLearning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“d our experiments to extractive tasks like question answering, as well as other attention-prone tasks, like seq2seq models. We experiment on the following datasets: Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB Large Movie Reviews Corpus (Maas et al.,2011), 20 NEWSGROUPS (hockey vs. baseball), the AG N…”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 5×
“This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
BackgroundMethodInfluentialTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 5×
“Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
BackgroundMethodInfluentialJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987 · cited 13×
“…greatly exacerbated when performed independently on each instance.2 Thus, it is no surprise that Jain and Wal- 2Indeed, the most open-ended task, question answering over CNN data, produces considerable difficulty to manipulate its scores by random permutation (Figure 6e in Jain and Wallace (2019)).”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 . · arXiv
- Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Cited by
Items 101–200 of 1,008
Shahid, M. R., & Debar, H. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00256 · Cited by 119
CDLM: Cross-Document Language Modeling
Caciularu, A., Cohan, A., Beltagy, I., Peters, M., Cattan, A., & Dagan, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.225 · Cited by 55
Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., & Labi, S. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104358 · Cited by 117
Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rocktäschel, T., & Krueger, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.12786 · Cited by 114
Unmasking the Mask – Evaluating Social Biases in Masked Language Models
Kaneko, M., & Bollegala, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21453 · Cited by 46
RTBAS: Defending LLM Agents Against Prompt Injection and Privacy LeakagePreprint
Zhong, P. Y., Chen, S., Wang, R., McCall, M., Titzer, B. L., Miller, H., & Gibbons, P. B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.08966 · Cited by 112
“Attention scores [43,49], which measure a transformer-based model’s “focus” on past tokens, is a widely-used technique in the machine learning community to explain a neural network’s internal processing [19, 45], prune irrelevant input texts [56], etc.”
Human-Level Interpretable Learning for Aspect-Based Sentiment Analysis
Yadav, R. K., Jiao, L., Granmo, O., & Goodwin, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17671 · Cited by 110
“However, these attention weights do not provide faithful explanations for classification (Serrano and Smith 2019; Wiegreffe and Pinter 2019; Brunner et al. 2020; Vashishth et al. 2019).”
BackgroundToward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 111
“Hence, post hoc analyses of attention weights and tests, such as the ones proposed in [94], are needed to make informed decisions about the extent to which model outputs can be interpreted via attention weights.”
BackgroundNeuron-Level Knowledge Attribution in Large Language Models
Yu, Z., & Ananiadou, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.191 · Cited by 110
“However, the validity of attributions is challenged by many studies (Serrano and Smith, 2019; Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Mohankumar et al., 2020; Etha-yarajh and Jurafsky, 2021; Bai et al., 2021).”
BackgroundDeep Entity Matching: Challenges and Opportunities
Li, Y., Li, J., Suhara, Y., Wang, J., Hirota, W., & Tan, W.-C. (2021) · Journal of Data and Information Quality · doi:10.1145/3431816 · Cited by 108
“Attention weight analysis [11, 26, 75] is the most common way to analyze the behavior of Transformer-based models such as BERT.”
BackgroundMethodMeasuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2010.12762 · Cited by 106
Self-citationA Survey of Neural Code Intelligence: Paradigms, Advances and BeyondPreprint
Sun, Q., Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.14734 · Cited by 105
Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2101698 · Cited by 104
Barragán-Montero, A., Bibal, A., Dastarac, M. H., Draguet, C., Valdés, G., Nguyen, D., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac678a · Cited by 84
Measuring the Mixing of Contextual Information in the Transformer
Ferrando, J., Gállego, G. I., & Costa‐jussà, M. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.595 · Cited by 104
Hase, P., Xie, H., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00786 · Cited by 104
“A large number of methods have been introduced for FI estimation, drawing upon local approximation models [47, 48, 31], attention weights [26, 64, 71], model gradients [52, 51, 59, 54], and model-based feature selection [6, 4, 65, 44, 11, 15].”
MethodOn the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries
Shi, T., Zhao, C., Boyd-Graber, J. L., Daumé, H., & Lee, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.167 · Cited by 103
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 103
“In contrast, Wiegreffe and Pinter [181] proposed the work of “Attention is not not explanation” specifically against the arguments in Reference [79], arguing that whether attention weights are faithful explanations is dependent on the definition of explanation and conducted four different experiments to prove when att…”
BackgroundInfluentialXDA: Accurate, Robust Disassembly with Transfer Learning
Pei, K., Guan, J., Williams-King, D., Yang, J., & Jana, S. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.23112 · Cited by 102
Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps
Barkan, O., Hauon, E., Caciularu, A., Katz, O., Malkiel, I., Armstrong, O., & Koenigstein, N. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482126 · Cited by 102
“Moreover, recent works [1, 43, 52] showed that the input’s sequence gradients have a high correlation”
BackgroundEvaluating the Utilities of Foundation Models in Single‐Cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2026) · Advanced Science · doi:10.1002/advs.202514490 · Cited by 61
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
Sood, E., Tannert, S., Frassinelli, D., Bulling, A., & Vu, N. T. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.2 · Cited by 99
Why Attentions May Not Be Interpretable?
Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., & Wang, F. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467307 · Cited by 99
“There have been lots of debates on whether attention is an explanation or not recently [12, 26, 34].”
BackgroundHuang, G., Li, Y., Jameel, S., Long, Y., & Papanastasiou, G. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.05.004 · Cited by 99
“Significant variability in the literature was observed across model, input-and output-based methodologies used (Table 1) and NLP and medical tasks addressed (Figures 2 and 3).”
MethodAn Interpretable Deep Learning Model for Automatic Sound Classification
Zinemanas, P., Rocamora, M., Miron, M., Font, F., & Serra, X. (2021) · Electronics · doi:10.3390/electronics10070850 · Cited by 98
“However, more research is needed to assess whether attention mechanisms can provide reliable explanations [33].”
BackgroundRepresentation learning for neural population activity with Neural Data Transformers
Ye, J., & Pandarinath, C. (2021) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.27358 · Cited by 98
“However, to date there has been no widely accepted theory for how to interpret such attention, and so attentionbased analysis is often debated [49].”
BackgroundNascita, A., Aceto, G., Ciuonzo, D., Montieri, A., Persico, V., & Pescapè, A. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3504955 · Cited by 98
“As an example of intrinsic explainability, attention mechanisms [41] can enhance neural network interpretability by dynamically assigning weights to distinct segments of input sequences, providing insights into which elements influ-ence the decision-making process.”
Counterfactual Explanations for Neural Recommenders
Tran, K. H., Ghazimatin, A., & Roy, R. S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463005 · Cited by 97
“Typical methods explaining state-of-theart neural recommenders face certain concerns: (i) they often rely on the attention mechanism to find important words [26], reviews [4], or regions in images [6], which is still controversial [15, 27]; (ii) use connecting paths between users and items [1, 29, 31] that may not”
MethodDeep emotion recognition in textual conversations: a survey
Pereira, P., Moniz, H., & Carvalho, J. P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11010-y · Cited by 98
“The Plutchik wheel of emotions [85] is also described.”
On Identifiability in Transformers
Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., & Wattenhofer, R. P. (2020) · International Conference on Learning Representations · Cited by 94
Understanding Attention for Text Classification
Sun, X., & Lu, W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.312 · Cited by 94
“Wiegreffe and Pinter (2019) suggested that the notion of explanation needs to be clearly defined, and the study of the explanation requires taking all components of a model into account.”
BackgroundQuantum circuit synthesis with diffusion models
Fürrutter, F., Muñoz-Gil, G., & Briegel, H. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00831-9 · Cited by 94
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P., & Sabharwal, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.99 · Cited by 48
MATE: Multi-view Attention for Table Transformer Efficiency
Eisenschlos, J., Gor, M., Müller, T., & Cohen, W. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.600 · Cited by 67
“It is important to note that in this context we are not measuring attention as an explanation method (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).”
BackgroundQED: A Framework and Dataset for Explanations in Question Answering
Lamm, M., Palomaki, J., Alberti, C., Andor, D., Choi, E., Soares, L. B., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00398 · Cited by 43
“QED exists in between relatively unstructured explanation forms on the one hand, such as attention distributions (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019; Mohankumar et al., 2020) or sequential outputs (Camburu et al., 2018, 2019; Narang et al., 2020; Kumar and Talukdar, 2020) and more…”
BackgroundTowards Transparent and Explainable Attention Models
Mohankumar, A. K., Nema, P., Narasimhan, S., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.387 · Cited by 91
A transformer-based model for default prediction in mid-cap corporate markets
Korangi, K., Mues, C., & Bravo, C. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2022.10.032 · Cited by 91
When Comparing to Ground Truth is Wrong: On Evaluating GNN Explanation Methods
Faber, L., Moghaddam, A. K., & Wattenhofer, R. P. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467283 · Cited by 91
A Comparative Study of Faithfulness Metrics for Model Interpretability Methods
Chan, C. S., Kong, H., & Guanqing, L. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.345 · Cited by 38
Explainable reinforcement learning (XRL): a systematic literature review and taxonomy
Bekkemoen, Y. (2023) · Machine Learning · doi:10.1007/s10994-023-06479-7 · Cited by 91
Guo, Z., Kamigaito, H., & Watanabe, T. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1178 · Cited by 90
Pârcălăbescu, L., & Frank, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.223 · Cited by 87
“But the equivalence of importance score and attention is debated and must be considered with care (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019) (see App.”
BackgroundPropagation Structure-Aware Graph Transformer for Robust and Interpretable Fake News Detection
Zhu, J., Gao, C., Yin, Z., Li, X., & Kurths, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672024 · Cited by 88
FairLens: Auditing black-box clinical decision support systems
Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102657 · Cited by 86
Saphra, N., & Wiegreffe, S. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.30 · Cited by 86
Self-citationNODE-GAM: Neural Generalized Additive Model for Interpretable Deep LearningPreprint
Chang, C., Caruana, R., & Goldenberg, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01613 · Cited by 87
“Although attention seems to be correlated with input importance [26], in the worst case they might not correlate well [25].”
BackgroundExplain and Predict, and then Predict Again
Zhang, Z., Rudra, K., & Anand, A. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441758 · Cited by 84
“However, recently the faithfulness of interpreting model prediction with soft attention weights has been called into question [11, 32].”
BackgroundContrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05276 · Cited by 40
“However, interpreting attention patterns without understanding the attended-to embeddings, or relying on attention alone for a faithful interpretation, can lead to faulty interpretations [Brunner et al., 2019, Jain and Wallace, 2019, Wiegreffe and Pinter, 2019].”
BackgroundDetecting ethnicity-targeted hate speech in Russian social media texts
Pronoza, E., Panicheva, P., Koltsova, O. Y., & Rosso, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102674 · Cited by 84
Audio Explainable Artificial Intelligence: A Review
Akman, A., & Schuller, B. W. (2023) · Intelligent Computing · doi:10.34133/icomputing.0074 · Cited by 83
Towards LLM-guided Causal Explainability for Black-box Text ClassifiersPreprint
Bhattacharjee, A., Moraffah, R., Garland, J., & Liu, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13340 · Cited by 81
“Out of the various explainability methods explored by the community, methods such as attention scores (Wiegreffe and Pinter 2019) are typically local and often do not generalize well (Jain and Wallace 2019; Balkir et al. 2022), and are also not applicable to black-box models where we only have…”
MethodA Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 80
“This view is nuanced by [161], who suggest that attention can serve as an explanation, but its validity depends on the context and testing methods.”
BackgroundInterpretation of NLP models through input marginalization
Kim, S., Yi, J., Kim, E., & Yoon, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.255 · Cited by 81
Interpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 80
“There has been a growing debate on the question of whether attention is interpretable [164, 143, 86, 203, 193, 30].”
BackgroundInfluentialXAI Handbook: Towards a Unified Framework for Explainable AI
Palacio, S., Lucieri, A., Munir, M., Ahmed, S., Hees, J., & Dengel, A. R. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00420 · Cited by 79
“The link between the intuitive notion of “attention” and its implementation in one of MDNet’s modules, has been recently contested [15, 38], undermining any causal relation drawn from it.”
BackgroundChalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., & Malakasiotis, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.22 · Cited by 78
Marasović, A., Bhagavatula, C., Park, J. S., Le Bras, R., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 51
Opitz, J., & Frank, A. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.48 · Cited by 78
“, 2020), we would have to, at least, analyze attention weights (Clark et al., 2019; Wiegreffe and Pinter, 2019) or gradients (Selvaraju et al.”
BackgroundWhich *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
Leveraging the attention mechanism to improve the identification of DNA N6-methyladenine sites
Zhang, Y., Liu, Y., Xu, J., Wang, X., Peng, X., Song, J., & Yu, D. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab351 · Cited by 77
Explanation Guided Contrastive Learning for Sequential Recommendation
Wang, L., Lim, E., Liu, Z., & Zhao, T. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557317 · Cited by 76
“However, attention as explanation is controversial [15, 39] since different attention distributions can produce the same model predictions.”
BackgroundMethodTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75
“Wiegreffe and Pinter [145] argue that attention could be explanations depending on the definition of explanations.”
BackgroundHuman Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?
Şen, C., Hartvigsen, T., Yin, B., Kong, X., & Rundensteiner, E. A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.419 · Cited by 74
“That is, they provide a look into the inner workings of a model, in that they produce an easily-understandable weighting of hidden states (Wiegreffe and Pinter, 2019).”
BackgroundTowards understanding structure–property relations in materials with interpretable deep learning
Vu, T.-S., Ha, M.-Q., Nguyen, D., Nguyen, V.-C., Abe, Y., Tran, T., Huan, T. D., Kino, H., Miyake, T., Tsuda, K., et al. (2023) · npj Computational Materials · doi:10.1038/s41524-023-01163-9 · Cited by 74
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers
Rajagopal, D., Balachandran, V., Hovy, E. H., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.64 · Cited by 40
“In natural language applications, feature attribution based on attention scores (Xu et al., 2015) has been the predominant method for developing inherently interpretable neural classifiers.”
MethodHuman-centered explainability for life sciences, healthcare, and medical informatics
Dey, S., Chakraborty, P., Kwon, B. C., Dhurandhar, A., Ghalwash, M. F., Suárez, F., Ng, K., Sow, D., Varshney, K. R., & Meyer, P. (2022) · Patterns · doi:10.1016/j.patter.2022.100493 · Cited by 72
“19,20 Even methods that may otherwise appear grounded in theory and widely adopted, such as SHAP, 21 have also been criticized.”
BackgroundEvaluating and Characterizing Human Rationales
Carton, S., Rathore, A., & Tan, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.747 · Cited by 71
KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations
Zanzotto, F. M., Santilli, A., Ranaldi, L., Onorati, D., Tommasino, P., & Fallucchi, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.18 · Cited by 71
“In fact, BERTviz is a good visualization mechanism for seeing how models assign weights to different input elements (Bahdanau et al., 2015; Belinkov and Glass, 2019), but it is extremely obscure in explaining causal relations in classification predictions (Wiegreffe and Pinter, 2019).”
BackgroundHierarchical label-wise attention transformer model for explainable ICD coding
Liu, L., Perez‐Concha, O., Nguyen, A. N., Bennett, V., & Jorm, L. R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104161 · Cited by 72
“knowledge towards the debate about the usefulness of attention mechanisms for explainability [52, 53].”
BackgroundAttention Is Not the Only Choice: Counterfactual Reasoning for Path-Based Explainable Recommendation
Li, Y., Sun, X., Chen, H., Zhang, S., Yang, Y., & Xu, G. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3373608 · Cited by 71
Multi-Instance Multi-Label Learning Networks for Aspect-Category Sentiment Analysis
Li, Y., Yin, C., Zhong, S.-H., & Xu, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.287 · Cited by 71
Explaining Question Answering Models through Text GenerationPreprint
Latcinnik, V., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05569 · Cited by 70
MARTA: Leveraging Human Rationales for Explainable Text Classification
Arous, I., Dolamic, L., Yang, J., Bhardwaj, A., Cuccu, G., & Cudré-Mauroux, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16734 · Cited by 41
“To explain individual predictions, a more popular approach is attention mechanisms, which identify parts of the input that are attended by the model for specific predictions (Xu et al. 2015; Bahdanau, Cho, and Bengio 2014).”
BackgroundMethodChrysostomou, G., & Αλέτρας, Ν. (2021) · doi:10.18653/v1/2021.acl-long.40 · Cited by 68
“In contrast to Jain and Wallace (2019), Wiegreffe and Pinter (2019) and Vashishth et al. (2019) demonstrate that attention weights can in certain cases provide robust explanations.”
BackgroundResultChallenges in Applying Explainability Methods to Improve the Fairness of NLP Models
Balkır, E., Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2022) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2022.trustnlp-1.8 · Cited by 67
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
Michael, J., Holtzman, A., Parrish, A., Mueller, A., Wang, A., Chen, A., Madaan, D., Nangia, N., Pang, R. Y., Phang, J., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.903 · Cited by 65
Ieremie, I., Ewing, R. M., & Niranjan, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac104 · Cited by 65
“Interpreting attention is an active and well-known area of research (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019), but the application to biological sequences is still lagging behind.”
Modarressi, A., Fayyaz, M., Yaghoobzadeh, Y., & Pilehvar, M. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.19 · Cited by 64
Causal Interpretation of Self-Attention in Pre-Trained Transformers
Rohekar, R. Y., Gurwicz, Y., & Nisimov, S. (2023) · neural information processing systems · doi:10.52202/075280-1366 · Cited by 64
Teach Me to Explain: A Review of Datasets for Explainable NLP.Preprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · Cited by 64
Self-citationGuerreiro, N. M., Voita, E., & Martins, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.75 · Cited by 63
Is Sparse Attention more Interpretable?
Meister, C., Lazov, S., Augenstein, I., & Cotterell, R. (2021) · doi:10.18653/v1/2021.acl-short.17 · Cited by 63
“…tasks, utilizing both LSTM and Transformer-based models, we observe how sparse attention affects the results of Jain and Wallace (2019) and Wiegreffe and Pinter (2019), additionally exploring whether it allows us to identify a core set of inputs that are important to models’ decisions.”
BackgroundMethodResultInfluentialQuantifying Context Mixing in Transformers
Mohebbi, H., Zuidema, W. H., Chrupała, G., & Alishahi, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.245 · Cited by 64
Interpretable Image Classification with Adaptive Prototype-based Vision Transformers
Ma, C., Donnelly, J. P., Liu, W., Vosoughi, S., Rudin, C. D., & Chen, C. (2024) · neural information processing systems · doi:10.52202/079017-1312 · Cited by 63
Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models.
Grimsley, C., Mayfield, E., & Bursten, J. R. S. (2020) · PhilPapers (PhilPapers Foundation) · Cited by 63
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
A Review on Language Models as Knowledge Bases
AlKhamissi, B., Li, M., Celikyilmaz, A., Diab, M., & Ghazvininejad, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06031 · Cited by 62
“However, many previous works have also contended whether attention can convey proper explanations (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
BackgroundAIST: An Interpretable Attention-Based Deep Learning Model for Crime Prediction
Rayhan, Y., & Hashem, T. (2023) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3582274 · Cited by 63
“A recent study [52] shows the conditions under which attentions can be regarded as faithful explanations, and nullifies the claim [25, 42] that criticizes attention as a form of faithful explanation due to its weak correlation with other feature importance metrics and the existence of alternate adversarial attention w…”
BackgroundInfluentialSparse Attention with Linear Units
Zhang, B., Titov, I., & Sennrich, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.523 · Cited by 62
“The explanatory power of standard attention weights is hotly debated (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019).”
BackgroundLi, M., Peng, P., Zhang, J., Wang, H., & Shen, W. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3313728 · Cited by 62
“However, attention mechanisms theoretically have the potential to improve interpretability and identify the true root causes while strengthening the representational power of models [10].”
BackgroundKotobi, A., Singh, K., Höche, D., Bari, S., Meißner, R. H., & Bande, A. (2023) · Journal of the American Chemical Society · doi:10.1021/jacs.3c07513 · Cited by 62
Li, D., Hu, B., Chen, Q., Wang, X., Qi, Q., Wang, L., & Liu, H. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106522 · Cited by 62
Self-Explaining Structures Improve NLP ModelsPreprint
Sun, Z., Fan, C. C., Han, Q., Sun, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01786 · Cited by 62
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Ruder, S., Vulić, I., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.184 · Cited by 61
“, 2018) and much work focuses on leveraging components such as attention for interpretability, which have not been designed with that goal in mind (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019).”
Background“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks
Liu, J., Capurro, D., Nguyen, A. N., & Verspoor, K. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104149 · Cited by 61
Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
Zhang, Y., Wang, X., Wu, L., & Wang, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34793 · Cited by 60
Sparse and Continuous Attention Mechanisms
Martins, A. F. T., Farinhas, A., Treviso, M. V., Niculae, V., Aguiar, P. M. Q., & Figueiredo, M. A. T. (2020) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2006.07214 · Cited by 60
“For many of these applications, current state-of-the-art models use discrete softmax attention, whose interpretation capabilities have been questioned in prior work [38–40].”
BackgroundA Survey on Mechanistic Interpretability for Multi-Modal Foundation ModelsPreprint
Lin, Z., Basu, S., Beigi, M., Manjunatha, V., Rossi, R. A., Wang, Z., Zhou, Y., Balasubramanian, S., Zarei, A., Rezaei, K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.17516 · Cited by 60
“Studies find that while attention should not be directly equated with explanation (Pruthi et al., 2019; Jain and Wal-lace, 2019; Wiegreffe and Pinter, 2019), it provides significant insights into the model’s operational behavior and helps in error diagnosis and hypothesis development (Park et al.,…”
Analyzing how BERT performs entity matching
Paganelli, M., Buono, F. D., Baraldi, A., & Guerra, F. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3529337.3529356 · Cited by 60
“Nevertheless, this is a controversial point since other papers, as [31, 33], demonstrated that low correlations happen only in limited conditions.”
Result
Metadata
- Authors
- Wiegreffe, S., & Pinter, Y.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 11-20
- OpenAlex
- W2968210605
- Cited by
- 966
- References
- 19
- Citations by year
- 2026: 147 · 2025: 108 · 2024: 105 · 2023: 133 · 2022: 137 · 2021: 216 · 2020: 108 · 2019: 11
- FWCI
- 45.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- attention, explanation, interpretability, neural network interpretability, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.