Rationalizing Neural Predictions
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Prediction without justification has limited applicability.As a remedy, we learn to extract pieces of input text as justifications -rationales -that are tailored to be short and coherent, yet sufficient for making the same prediction.Our approach combines two modular components, generator and encoder, which are trained to operate well together.The generator specifies a distribution over text fragments as candidate rationales and these are passed through the encoder for prediction.Rationales are never given during training.Instead, the model is regularized by desiderata for rationales.We evaluate the approach on multi-aspect sentiment analysis against manually annotated test cases.Our approach outperforms attention-based baseline by a significant margin.We also successfully illustrate the method on the question retrieval task. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A lot of work has also been done on argumentation and dialog games [6, 11, 51, 53] but the focus in these areas is generally the logical structure of the framework to express and to relate arguments or the protocol to exchange arguments.”
From Beyond explainability: justifiability and contestability of algorithmic decision systems (2022) · cites this paper as [53]
Experimental — accuracy not yet measuredIn the citing paper
“This model can achieve competitive accuracy while having built-in interpretability because the prediction is made exclusively based on rationales.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“Attention based models offer another lens to the inner workings of neural models (Bahdanau et al., 2015; Cheng et al., 2016; Martins and Astudillo, 2016; Chen et al., 2015; Xu and Saenko, 2015; Yang et al., 2015).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1607.01759 · Cited by 4,542
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · cited 1×
“For instance, we could use a convolutional network (Kim, 2014; Kalchbrenner et al., 2014), deep averaging network (Iyyer et al.”
MethodA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568 · cited 1×
“space of possible rationales in terms of their interaction with the encoder. We could also apply variance reduction techniques to increase stability of stochastic training (cf. (Weaver and Tao, 2001; Mnih et al., 2014; Ba et al., 2015; Xu et al., 2015)). 7 Acknowledgments We thank Prof. Julian McAuley for sharing the…”
MethodTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
Research works printed but not linked: 6
- [6]6. Cicero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid rep- resentations to retrieve semantically equivalent ques- tions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 694-699, Beijing, China, July. Association for Com- putational Linguistics.
- [7]7. Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015a. Retrofitting word vectors to semantic lexicons. In Pro- ceedings of NAACL.
- [9]9. Aurélie Herbelot and Eva Maria Vecchi. 2015. Build- ing a shared world: mapping distributional to model- theoretic semantic spaces. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Lin- guistics.
- [12]12. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
- [27]27. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Her- mann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Rea- soning about entailment with neural attention. In In- ternational Conference on Learning Representations.
- [29]29. Sebastian Thrun. 1995. Extracting rules from artifi- cial neural networks with distributed representations. In Advances in neural information processing systems (NIPS).
Cited by
Items 101–200 of 900
Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
Yu, M., Chang, S., Zhang, Y., & Jaakkola, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1420 · Cited by 118
“From the left part of Table 4, we observe that the original model of (Lei et al., 2016) has a hard time maintaining the accuracy compared to the classi-fier trained with full texts.”
BackgroundMethodResultInfluentialSelf-citationGet Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
Self-citationAn Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125
Synthesizing entity matching rules by examples
Singh, R., Meduri, V. V., Elmagarmid, A. K., Madden, S. R., Papotti, P., Quiané-Ruiz, J.-A., Solar-Lezama, A., & Tang, N. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3149193.3149199 · Cited by 125
“As remarked earlier, the main obstacles to deploy them is that humans do not understand and trust them [26], due to the lack of interpretability [27].”
BackgroundTeach Me to Explain: A Review of Datasets for Explainable Natural Language ProcessingPreprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12060 · Cited by 149
“To define highlights, we turn to Lei et al. (2016) who popularized extractive rationales , i.e., sub-sets of the input tokens that satisfy two properties: (i) compactness , the selected input tokens are short and coherent, and (ii) sufficiency , the selected tokens suffice for prediction as a…”
Background"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
“[37] Tao Lei, Regina Barzilay, and Tommi Jaakkola.”
BackgroundExplaining NLP Models via Minimal Contrastive Editing (MiCE)
Ross, A., Marasović, A., & Peters, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.336 · Cited by 86
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M., & Choi, Y. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.47 · Cited by 87
Incorporating Priors with Feature Attribution on Text Classification
Liu, F., & Avci, B. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1631 · Cited by 91
“Sentence On the other hand, the amount of research focusing on explainable natural language processing (NLP) models (Li et al., 2016; Murdoch et al., 2018; Lei et al., 2016) is modest as opposed to image explanation techniques.”
BackgroundMeasuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.804 · Cited by 119
“the input and is sufficient to make a prediction on its own, without the rest of the input (Lei et al., 2016).”
BackgroundMethodTraining Classifiers with Natural Language Explanations
Hancock, B., Varma, P., Wang, S., Bringmann, M., Liang, P., & Ré, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1175 · Cited by 109
“Finally, while we have used natural language explanations as input to train models, they can also be output to interpret models (Krening et al., 2017; Lei et al., 2016).”
MethodDeriving Machine Attention from Human Rationales
Bao, Y., Chang, S., Yu, M., & Barzilay, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1216 · Cited by 140
“We use a subset of the BeerAdvocate review dataset (McAuley et al., 2012) introduced by Lei et al. (2016).”
BackgroundMethodInfluentialSelf-citationYu, A. W., Lee, H., & Le, Q. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1172 · Cited by 92
“The key difference between our work and Lei et al. (2016)’s method is that our method optimizes for faster inference, and is more dynamic in its jumping.”
BackgroundMethodInferring Which Medical Treatments Work from Reports of Clinical Trials
Lehman, E., DeYoung, J., Barzilay, R., & Wallace, B. C. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1371 · Cited by 99
“An additional advantage of modeling this explicitly is that models will then be able to provide rationales for decisions (Lei et al., 2016; Zhang et al., 2016; Zaidan et al., 2007), i.e., snippets of text that support predictions.”
BackgroundSelf-citationLeveraging Demonstrations for Reinforcement Recommendation Reasoning over Knowledge Graphs
Zhao, K., Wang, X., Zhang, Y., Li, Z., Liu, Z., Xing, C., & Xie, X. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401171 · Cited by 129
“Studies suggest that concise explanations help reduce user cognitive loads and are considered to be more interpretable [18, 23].”
BackgroundSmith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction
Paranjape, B., Joshi, M., Thickstun, J., Hajishirzi, H., & Zettlemoyer, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.153 · Cited by 86
Review on Interpretable Machine Learning in Smart Grid
Xu, C., Liao, Z., Li, C., Zhou, X., & Xie, R. (2022) · Energies · doi:10.3390/en15124427 · Cited by 126
“[83] introduced a self-explanatory neural network in which an explanation generation module generates a subset of important features for each sample.”
BackgroundWhy We Should Have Seen That Coming
Wolf, M. J., Miller, K. W., & Grodzinsky, F. S. (2017) · The ORBIT Journal · doi:10.29297/orbit.v1i2.49 · Cited by 96
Measuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2010.12762 · Cited by 106
Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., & Labi, S. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104358 · Cited by 117
Jin, X., Wei, Z., Du, J., Xue, X., & Ren, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06194 · Cited by 120
“rks has been studied with vairous techniques, including probing learned features with auxiliary tasks (Tenney et al., 2019), or designing models with inherent interpretability (Bahdanau et al., 2015; Lei et al., 2016). A major line of work, local explanation algorithms, explains predictions by assigning importance sco…”
BackgroundLearning Latent Opinions for Aspect-level Sentiment Classification
Wang, B., & Lu, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12020 · Cited by 106
“In contrast with the soft attention mechanism, “hard” (stochastic) attention (Xu et al. 2015) is also leveraged by (Lei, Barzilay, and Jaakkola 2016) which uses a separate model to generate rationales for neural predictions.”
BackgroundBeyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Röth, V., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06178 · Cited by 88
“Lei, Barzilay, and Jaakkola (2016) train deep models that make predictions from text and simultaneously highlight contiguous subsets of words, called a “rationale,” to justify each prediction.”
BackgroundGenerating Hierarchical Explanations on Text Classification via Feature Interaction Detection
Chen, H., Zheng, G., & Ji, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.494 · Cited by 143
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
The “big red button” is too late: an alternative model for the ethical evaluation of AI systems
Arnold, T., & Scheutz, M. (2018) · Ethics and Information Technology · doi:10.1007/s10676-018-9447-7 · Cited by 94
“Ferreting out bias in training data is one area of increasing interest (Lei et al. 2016; Devlin 2016), as is being able to correct bias through supervised learning (Hardesty 2016; Hardt et al.”
BackgroundAspect-augmented Adversarial Networks for Domain Adaptation
Zhang, Y., Barzilay, R., & Jaakkola, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00077 · Cited by 88
“While traditional attention-based models typically induce attention in an unsupervised manner, they have to rely on a large amount of labeled data for the target task (Bahdanau et al., 2014; Rush et al., 2015; Chen et al., 2015; Cheng et al., 2016; Xu et al., 2015; Xu and Saenko, 2015; Yang et al., 2015; Martins and A…”
BackgroundSelf-citationExplainable Information Retrieval: A SurveyPreprint
Anand, A., Lyu, L., Idahl, M., Wang, Y., Wallat, J., & Zhang, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.02405 · Cited by 106
“[69] is the first work that proposes an end-to-end approach for the problem of rationale-based models in the context of vanilla text classification.”
BackgroundTowards Transparent and Explainable Attention Models
Mohankumar, A. K., Nema, P., Narasimhan, S., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.387 · Cited by 91
Liu, H., Zhong, C., Alnusair, A., & Islam, S. R. (2021) · Journal of Network and Systems Management · doi:10.1007/s10922-021-09606-8 · Cited by 103
“[36] propose an approach for sentiment analysis that uses a subset of text as an explanation of prediction instead of the entire text.”
MethodFew-Shot Self-Rationalization with Natural Language Prompts
Marasovic, A., Beltagy, I., Downey, D., & Peters, M. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.31 · Cited by 55
Document-Level Multi-Aspect Sentiment Classification as Machine Comprehension
Yin, Y., Song, Y., & Ming, Z. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1217 · Cited by 105
“Due to lack of space, we only present the results of TripAdvisor and the re-sults of BeerAdvocate have a similar behavior as TripAdvisor.”
BackgroundMethodInfluentialEvaluating neural network explanation methods using hybrid documents and morphosyntactic agreement
Poerner, N., Schütze, H., & Roth, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1032 · Cited by 77
“, 2015) or rationale models (Lei et al., 2016)), as these are very specific architectures that may not be not applicable to all tasks.”
BackgroundGreco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 104
Aligning Faithful Interpretations with their Social Attribution
Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00367 · Cited by 73
“Although the term “rationale” (Lei et al., 2016) is more commonly used for this format in the NLP community, it is controversial, as it has been used ambiguously for different things in NLP literature historically (e.”
Model Agnostic Contrastive Explanations for Structured DataPreprint
Dhurandhar, A., Pedapati, T., Balakrishnan, A., Chen, P., Shanmugam, K., & Puri, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00117 · Cited by 106
“There are other works that fall under (i) such as [19, 15] as well as those that fall under (ii) for vision [33, 34, 28] and NLP applications [21].”
BackgroundContrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83
“These could include discrete features in the input ( textual highlights ; Lei et al., 2016; see Fig.”
BackgroundInterpretable Neural Predictions with Differentiable Binary Variables
Bastings, J., Aziz, W. F., & Titov, I. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1284 · Cited by 291
“Lei et al. (2016) augment this model with a collection of latent variables which we denote by z = (cid:104) z 1 , . . . , z n (cid:105) .”
BackgroundMethodResultInfluentialLearning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction
Jie, Z., Li, J., & Lu, W. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.410 · Cited by 108
Nagahisarchoghaei, M., Nur, N., Cummins, L., Nur, N., Karimi, M. M., Nandanwar, S., Bhattacharyya, S., & Rahimi, S. (2023) · Electronics · doi:10.3390/electronics12051092 · Cited by 103
Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning
Gao, Y., Gu, S., Jiang, J., Hong, S.-S., Yu, D., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3644073 · Cited by 121
“[82] the authors proposed the continuity and sparsity regularization terms; In Erion et al.”
BackgroundMethodInfluentialEvaluating Explanation Without Ground Truth in Interpretable Machine Learning
Yang, F., Du, M., & Hu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06831 · Cited by 34
“Moreover, in natural language processing, similar human annotation, named rationale [22], has been extensively used for evaluation, which is a subset of features highlighted by annotators and regarded to be important for prediction.”
BackgroundSolving Aspect Category Sentiment Analysis as a Text Generation Task
Liu, J., Teng, Z., Cui, L., Liu, H., & Zhang, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.361 · Cited by 103
“A.2 Document-Level Datasets TripAdvisor (Wang et al., 2010) and BeerAdvocate (McAuley et al., 2012; Lei et al., 2016) contain seven aspects (value, room, location, cleanliness, check in/front desk, service, and business service) and four aspects (feel, look, smell, and taste) respectively.”
BackgroundPALM: Machine Learning Explanations For Iterative Debugging
Krishnan, S., & Wu, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077271 · Cited by 81
“[13, 14, 16].”
BackgroundInfluentialYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 114
Invariant RationalizationPreprint
Chang, S., Zhang, Y., Yu, M., & Jaakkola, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09772 · Cited by 98
“m is produced by an independent selection process (Lei et al., 2016).”
BackgroundMethodInfluentialSelf-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2022) · arXiv · doi:10.48550/arxiv.2211.08073 · Cited by 81
A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
Mohseni, S., Block, J. E., & Ragan, E. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05075 · Cited by 45
Rationalization for explainable NLP: a survey
Gurrapu, S., Kulkarni, A., Huang, L., Lourentzou, I., & Batarseh, F. A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1225093 · Cited by 58
“(A) An example of the Extractive Rationalization approch (Lei et al., 2016).”
MethodChalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., & Malakasiotis, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.22 · Cited by 78
Understanding Convolutional Neural Networks for Text ClassificationPreprint
Jacovi, A., Shalom, O. S., & Goldberg, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08037 · Cited by 116
“While recent works have begun to supply the means of interpreting predictions (Alvarez-Melis and Jaakkola, 2017; Lei et al., 2016; Guo et al., 2018), interpreting neural NLP models remains an under-explored area.”
BackgroundRisk assessment tools and data-driven approaches for predicting and preventing suicidal behavior
Velupillai, S., Hadlaczky, G., Baca-Garcia, E., Gorrell, G. M., Werbeloff, N., Nguyen, D., Patel, R., Leightley, D., Downs, J., Hotopf, M., et al. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00036 · Cited by 76
A Regularized Framework for Sparse and Structured Neural Attention
Niculae, V., & Blondel, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07704 · Cited by 71
“Furthemore, [26] sidesteps the need to backpropagate through the fused lasso, unlike our work, by using a stochastic training approach.”
MethodContextual Explanation Networks
Al-Shedivat, M., Dubey, A., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10301 · Cited by 64
“There are multiple other complementary approaches to interpretability ranging from a variety of visualization techniques [35, 36, 37, 38], to explanations by example [39, 40, 41, 42], to natural language rationales [43].”
BackgroundAgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and SecurityPreprint
Liu, D., Ren, Q., Qian, C., Shao, S., Xie, Y., Li, Y., Yang, Z., Luo, H., Wang, P., Liu, Q., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.18491 · Cited by 78
On the Word Alignment from Neural Machine Translation
Li, X., Li, G., Liu, L., Meng, M. Q. ‐., & Shi, S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1124 · Cited by 107
“Therefore, it is closely related to many works on visualizing and interpreting neural networks (Lei et al., 2016; Bach et al., 2015; Zintgraf et al., 2017).”
BackgroundHow do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
De Cao, N., Schlichtkrull, M. S., Aziz, W., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.262 · Cited by 75
Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach
Bang, S., Xie, P., Lee, H., Wu, W., & Xing, E. P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17358 · Cited by 95
Akula, A. R., Wang, K., Liu, C., Saba-Sadiya, S., Lu, H., Todorovic, S., Chai, J., & Zhu, S.-C. (2022) · iScience · doi:10.1016/j.isci.2021.103581 · Cited by 64
“For example, the reasoning behind the predictions made by linear regression models, decision trees, and And-Or Graphs (Li et al., 2013; Zhang et al., 2017) is easier to understand without using any external XAI models and hence are considered as intrinsically explainable.”
BackgroundLocal Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 103
“If the probability of classifying the perturbed text into the same class as that of the original text is above a predefined threshold, then the selected predicates will be considered as the ultimate key features to interpret the prediction results.”
BackgroundMethodInfluentialBeyond explainability: justifiability and contestability of algorithmic decision systems
Henin, C., & Le Métayer, D. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01251-8 · Cited by 71
“For example, [14] refers to explanations “which are presumed to be justifying” and [50] considers that justifications are ways to make understandable the inner operations of a complex system (in a white-box setting).”
BackgroundLeveraging Latent Features for Local Explanations
Luss, R., Chen, P., Dhurandhar, A., Sattigeri, P., Zhang, Y., Shanmugam, K., & Tu, C. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467265 · Cited by 98
“Belonging to this second type are multiple local explanation methods that generate explanations for images [10,11,32,36,38] and some others for NLP applications [23].”
BackgroundA Multi-Component Framework for the Analysis and Design of Explainable Artificial Intelligence
Kim, M.-Y., Atakishiyev, S., Babiker, H. K. B., Farruque, N., Goebel, R., Zaïane, O. R., Motallebi, M.-H., Rabelo, J., Syed, T., Yao, H., et al. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040045 · Cited by 57
Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
Fan, M., Wei, W., Xie, X., Liu, Y., Guan, X., & Liu, T. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3021924 · Cited by 65
“and semantic segmentation [37] in computer vision-related tasks, and the rational annotations [38] used in natural language processing, it is a time-consuming and error-prone job to manually construct ground truth annotations in Android malware analysis.”
BackgroundLearning Structured Natural Language Representations for Semantic Parsing
Cheng, J., Reddy, S., Saraswat, V., & Lapata, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1005 · Cited by 70
“A sideproduct of our modeling framework is that the induced intermediate representations can contribute to rationalizing neural predictions (Lei et al., 2016).”
BackgroundMethodInfluentialVisual Interaction with Deep Learning Models through Collaborative Semantic Inference
Gehrmann, S., Strobelt, H., Kruger, R., Pfister, H., & Rush, A. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934595 · Cited by 58
“The most commonly applied decision-understanding techniques present overlays over images [22,66] or texts [20,54,55].”
BackgroundHuman-grounded Evaluations of Explanation Methods for Text Classification
Lertvittayakumjorn, P., & Toni, F. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1523 · Cited by 60
“Besides, there are other explanation methods designed for specific deep learning architectures such as attention mechanism (Ghaeini et al., 2018) and extractive rationale generation (Lei et al., 2016).”
MethodRepresentation Engineering: A Top-Down Approach to AI Transparency
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01405 · Cited by 65
A Computational Model of Commonsense Moral Decision Making
Kim, R., Kleiman-Weiner, M., Abeliuk, A., Awad, E., Dsouza, S., Tenenbaum, J. B., & Rahwan, I. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278770 · Cited by 84
“In addition, Deep Learning models have been criticized as “blackbox” algorithms that defy attempts at interpretation (Lei, Barzilay, and Jaakkola, 2016).”
MethodContrastive Explanations with Local Foil TreesPreprint
van der Waa, J., Robeer, M., van Diggelen, J., Brinkhuis, M. J. S., & Neerincx, M. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07470 · Cited by 83
“A number of examples of common methods are: ordering the feature’s contribution to an output (Datta et al., 2016; Lei et al., 2016; Ribeiro et al., 2016), attention maps and saliency of the features (Selvaraju et al., 2016; Montavon et al., 2017; Sun-dararajan et al., 2017; Zhang et al., 2017),…”
MethodExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning
Saha, S., Yadav, P., Bauer, L., & Bansal, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.609 · Cited by 99
“…in NLP: Explanation datasets in NLP (Wiegreffe and Marasovi´c, 2021) take three major forms: (1) extractive rationales (Zaidan et al., 2007; Lei et al., 2016; Yu et al., 2019; DeYoung et al., 2020), (2) free-form or natural language explanations (Camburu et al., 2018; Rajani et al., 2019;…”
BackgroundPrompting Contrastive Explanations for Commonsense Reasoning Tasks
Paranjape, B., Michael, J., Ghazvininejad, M., Hajishirzi, H., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.366 · Cited by 93
How Interpretable are Reasoning Explanations from Prompting Large Language Models?
Wei Jie, Y., Satapathy, R., Goh, R., & Cambria, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.138 · Cited by 27
“Extractive rationalizing model (Lei et al., 2016), designed to be faithful, generally comprises two separate components: explainer and predictor.”
BackgroundFormative Essay Feedback Using Predictive Scoring Models
Woods, B., Adamson, D., Miel, S., & Mayfield, E. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098160 · Cited by 54
“Our results in this area are parallel to recent work in the deep learning domain on creating textual rationales for network predictions [16].”
BackgroundResultUnderstanding Unequal Gender Classification Accuracy from Face Images
Muthukumar, V., Pedapati, T., Ratha, N., Sattigeri, P., Wu, C.-W., Kingsbury, B. E. D., Kumar, A., Thomas, S., Mojsilović, A., & Varshney, K. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00099 · Cited by 42
“One class of interpretability methods explains a neural network’s decision on each image [2, 31, 33, 20].”
BackgroundCan I Trust the Explainer? Verifying Post-hoc Explanatory MethodsPreprint
Camburu, O.-M., Giunchiglia, E., Foerster, J., Lukasiewicz, T., & Blunsom, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02065 · Cited by 82
“The RCNN [15] consists of two modules: a generator followed by an encoder, both instantiated with recurrent convolutional neural networks [14].”
MethodInfluentialCoherent Hierarchical Multi-Label Classification NetworksPreprint
Giunchiglia, E., & Lukasiewicz, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10151 · Cited by 62
“, [19]), and study how MCM and MCLoss can be modified to improve the interpretability of C-HMCNN(h).”
BackgroundMultiVerS: Improving scientific claim verification with weak supervision and full-document context
Wadden, D., Lo, K., Wang, L., Cohan, A., Beltagy, I., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.6 · Cited by 60
“Another alternative to supervised rationale selection is to treat rationales as latent variables (Lei et al., 2016; Paranjape et al., 2020).”
BackgroundExplain and Predict, and then Predict Again
Zhang, Z., Rudra, K., & Anand, A. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441758 · Cited by 84
“We consider the following competitors that also use a pipelined approach to showcase the effectiveness of our approach • Leiet al.[19]: An end-to-end explain-then-predictapproach where rationale generator and decoder are not supervised on rationales; • DeYoung et al. [7]: An improvement of the…”
BackgroundMethodInfluentialInterpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 80
“A rationale is defined to be a short yet sufficient piece of text serving as justification for the model prediction [113].”
BackgroundMethodInfluentialExplainable Sentiment Analysis with Applications in Medicine
Zucco, C., Liang, H., Fatta, G. D., & Cannataro, M. (2018) · 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621359 · Cited by 59
“In [67], authors propose a modular neural framework to automatically generate interpretable and extractive rationale.”
BackgroundLearning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
Chen, H., & Ji, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.347 · Cited by 74
Froomkin, A. M., Kerr, I., & Pineau, J. (2025) · Robot Law: Volume II · doi:10.4337/9781800887305.00012 · Cited by 59
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58
“…gradient-based methods such as deepLIFT (Shriku-mar, Greenside, and Kundaje 2017) and integrated gradients (Sundararajan, Taly, and Yan 2017); and 3) attention methods which explicitly model feature importances rather than calculating them retroactively (e.g. ( Lei, Barzilay, and Jaakkola 2016)).”
MethodHuman Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?
Şen, C., Hartvigsen, T., Yin, B., Kong, X., & Rundensteiner, E. A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.419 · Cited by 74
“An alternative approach, referred to as “rationale mechanism”, can be seen as a type of hard attention (Lei et al., 2016; Bao et al., 2018).”
BackgroundQuantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
Mohseni, S., Block, J. E., & Ragan, E. D. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450689 · Cited by 74
“, evidence supporting the target label [38], provides an explanation quality score [20].”
BackgroundTeach Me to Explain: A Review of Datasets for Explainable NLP.Preprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · Cited by 64
AIST: An Interpretable Attention-Based Deep Learning Model for Crime Prediction
Rayhan, Y., & Hashem, T. (2023) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3582274 · Cited by 63
“Besides human evaluations [17, 26, 36], explanations that align directly with the input have been considered as plausible explanations [29].”
BackgroundShared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior
Boggust, A., Hoover, B., Satyanarayan, A., & Strobelt, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501965 · Cited by 45
“Figure 3: Using Shared Interest we identify eight recurring patterns in model behavior across two domains and saliency methods: an ImageNet [10] classifcation model with LIME [34] and a BeerAdvocate sentiment regression model [7, 25, 27] with Integrated Gradients [7, 41].”
MethodA Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 80
“Research[173] enhances interpretability by extracting key text segments, termed "rationales", serving as justifications for model predictions.”
BackgroundRegularizing Black-box Models for Improved InterpretabilityPreprint
Plumb, G., Al-Shedivat, M., Cabrera, Á. A., Perer, A., Xing, E. P., & Talwalkar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06787 · Cited by 74
“…e.g. , Wang and Rudin, 2015; Caru-ana et al., 2015) to model-agnostic, post-hoc algorithms for interpreting complex, black-box predictors such as ensem-1 bles and deep neural networks ( e.g. , Ribeiro et al., 2016; Lei et al., 2016; Lundberg and Lee, 2017; Selvaraju et al., 2017; Kim et al., 2018).”
BackgroundEvaluating and Characterizing Human Rationales
Carton, S., Rathore, A., & Tan, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.747 · Cited by 71
Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition
Ouchi, H., Suzuki, J., Kobayashi, S., Yokoi, S., Kuribayashi, T., Konno, R., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.575 · Cited by 57
A Formal Framework to Characterize Interpretability of ProceduresPreprint
Dhurandhar, A., Iyengar, V. S., Luss, R., & Shanmugam, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02952 · Cited by 57
“Other works also investigate methods of the type (b) discussed above for vision [45] and NLP applications [32].”
MethodDomain Knowledge Aided Explainable Artificial Intelligence for Intrusion Detection and ResponsePreprint
Islam, S. R., Eberle, W., Ghafoor, S., Siraj, A., & Rogers, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09853 · Cited by 64
“(Lei, Barzilay, and Jaakkola 2016) propose an approach for sentiment analysis where a subset of text from the whole text is selected as the rationale for the prediction.”
BackgroundGradient Estimation with Stochastic Softmax TricksPreprint
Paulus, M. B., Choi, D., Tarlow, D., Krause, A., & Maddison, C. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08063 · Cited by 100
“8 Our model used word embeddings from [48] and convolutional neural networks with one (simple) and three (complex) layers to produce a distribution over k -hot binary latent masks.”
BackgroundMethodInfluentialProoFVer: Natural Logic Theorem Proving for Fact Verification
Krishna, A., Riedel, S., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00503 · Cited by 51
“By design, we ensure that the generated proof is solely used for determining the label of claim, making it a faithful by construction model (Lei et al., 2016; Jain et al., 2020).”
MethodExplainable Information Retrieval
Anand, A., Sen, P., Saha, S., Verma, M., & Mitra, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3594249 · Cited by 51
“based models [15, 16, 43] that claim to be interpretable by design.”
Background
Metadata
- Authors
- Lei, T., Barzilay, R., & Jaakkola, T.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 107-117
- OpenAlex
- W2440159969
- Cited by
- 777
- References
- 27
- Citations by year
- 2026: 27 · 2025: 33 · 2024: 39 · 2023: 88 · 2022: 92 · 2021: 138 · 2020: 126 · 2019: 101 · 2018: 78 · 2017: 39 · 2016: 4
- FWCI
- 39.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- explanation generation, question retrieval, model regularization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.