Rationalizing Neural Predictions
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Prediction without justification has limited applicability.As a remedy, we learn to extract pieces of input text as justifications -rationales -that are tailored to be short and coherent, yet sufficient for making the same prediction.Our approach combines two modular components, generator and encoder, which are trained to operate well together.The generator specifies a distribution over text fragments as candidate rationales and these are passed through the encoder for prediction.Rationales are never given during training.Instead, the model is regularized by desiderata for rationales.We evaluate the approach on multi-aspect sentiment analysis against manually annotated test cases.Our approach outperforms attention-based baseline by a significant margin.We also successfully illustrate the method on the question retrieval task. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A lot of work has also been done on argumentation and dialog games [6, 11, 51, 53] but the focus in these areas is generally the logical structure of the framework to express and to relate arguments or the protocol to exchange arguments.”
From Beyond explainability: justifiability and contestability of algorithmic decision systems (2022) · cites this paper as [53]
Experimental — accuracy not yet measuredIn the citing paper
“This model can achieve competitive accuracy while having built-in interpretability because the prediction is made exclusively based on rationales.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“Attention based models offer another lens to the inner workings of neural models (Bahdanau et al., 2015; Cheng et al., 2016; Martins and Astudillo, 2016; Chen et al., 2015; Xu and Saenko, 2015; Yang et al., 2015).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1607.01759 · Cited by 4,542
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · cited 1×
“For instance, we could use a convolutional network (Kim, 2014; Kalchbrenner et al., 2014), deep averaging network (Iyyer et al.”
MethodA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568 · cited 1×
“space of possible rationales in terms of their interaction with the encoder. We could also apply variance reduction techniques to increase stability of stochastic training (cf. (Weaver and Tao, 2001; Mnih et al., 2014; Ba et al., 2015; Xu et al., 2015)). 7 Acknowledgments We thank Prof. Julian McAuley for sharing the…”
MethodTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
Research works printed but not linked: 6
- [6]6. Cicero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid rep- resentations to retrieve semantically equivalent ques- tions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 694-699, Beijing, China, July. Association for Com- putational Linguistics.
- [7]7. Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015a. Retrofitting word vectors to semantic lexicons. In Pro- ceedings of NAACL.
- [9]9. Aurélie Herbelot and Eva Maria Vecchi. 2015. Build- ing a shared world: mapping distributional to model- theoretic semantic spaces. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Lin- guistics.
- [12]12. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
- [27]27. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Her- mann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Rea- soning about entailment with neural attention. In In- ternational Conference on Learning Representations.
- [29]29. Sebastian Thrun. 1995. Extracting rules from artifi- cial neural networks with distributed representations. In Advances in neural information processing systems (NIPS).
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“In [43], an interpretability technique for neural networks operating in the natural language processing (NLP) domain was proposed.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310
Deep learning for sentiment analysis: A survey
Zhang, L., Wang, S., & Liu, B. (2018) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1253 · Cited by 1,928
“Lei et al. 77 proposed to use a neural network approach to extract pieces of input text as rationales (reasons) for review ratings.”
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167
“This is normally done by introducing a novel form of output, such as feature heatmaps (71), rationales (87), feature hierarchies (82) or identifying important elements in the training set (69).”
BackgroundAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Machine learning in medicine: a practical introduction
Sidey-Gibbons, J. A. M., & Sidey-Gibbons, C. J. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0681-4 · Cited by 1,271
“It should also be acknowledged that whilst the ’Black Box’ concept does generally apply to models which utilize non-linear transformations, such as the neural networks, work is being carried out to facilitate feature identification in complex algorithms [12].”
BackgroundJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987
“Such instantiations explicitly select (modest) subsets of inputs to be considered when making a prediction, which are then by construction responsible for model output (Lei et al., 2016; Peters et al., 2018).”
BackgroundA Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966
“Generally, many accept the notion of extractive methods such as Lei et al. (2016), in which explanations come directly from the input itself (as in attention), as plausible.”
BackgroundMethodDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
“Recent XAI techniques of rationalization or verbalizing explanations by mimicking human generated explanations may be relevant to supporting these social goals (e.g., [26, 61, 88]).”
BackgroundJain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 467
“Such instantiations explicitly select (modest) subsets of inputs to be considered when making a prediction, which are then by construction responsible for model output (Lei et al., 2016; Peters et al., 2018).”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“…covered existing NLP models for generating explanations along with predictions [43, 53]: an intelligible model ( Logistic Regression classifier), a encoder-decoder rationale generator (Lei et al. [40]), and a recent contextual embedding-based model ( RoBERTa with linear classification layer).”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
“…covered existing NLP models for generating explanations along with predictions [43, 53]: an intelligible model ( Logistic Regression classifier), a encoder-decoder rationale generator (Lei et al. [40]), and a recent contextual embedding-based model ( RoBERTa with linear classification layer).”
BackgroundMachine Learning in Computer-Aided Synthesis Planning
Coley, C. W., Green, W. H., & Jensen, K. F. (2018) · Accounts of Chemical Research · doi:10.1021/acs.accounts.8b00087 · Cited by 704
This Looks Like That: Deep Learning for Interpretable Image Recognition
Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., & Rudin, C. (2018) · arXiv · doi:10.48550/arxiv.1806.10574 · Cited by 702
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690
“…inputs necessary to uphold a decision is not new; Li et al. (2016) use reinforcement learning to attempt to construct such a minimal set of words, Lei et al. (2016) train an encoder to constrain the input prior to clas-sification, and much of the work that has been done on extractive…”
BackgroundA graph-convolutional neural network model for the prediction of chemical reactivity
Coley, C. W., Jin, W., Rogers, L., Jamison, T. F., Jaakkola, T. S., Green, W. H., Barzilay, R., & Jensen, K. F. (2019) · Chemical Science · doi:10.1039/c8sc04228d · Cited by 679
“Instead, the architecture of deep learning models should inherently enable some degree of interpretability as has been done in natural language processing [21].”
BackgroundSelf-citationHateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 513
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
“Finally, with respect to texts, in Reference [65] the authors develop an approach f that incorporates rationales as part of the learning process of b.”
BackgroundUnderstanding Neural Networks through Representation Erasure
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412
“Task, Dataset and Training Inspired by recent visualization work from Lei et al. (2016), we focus on the task of document-level aspect rating prediction (Tang et al., 2015a; Tang et al., 2015b).”
BackgroundMethodInfluentialSentiment Analysis: Mining Opinions, Sentiments, and Emotions
Liu, B. (2015) · Cited by 656
On Interpretability of Artificial Neural Networks: A Survey
Fan, F.-L., Xiong, J., Li, M., & Wang, G. (2021) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2021.3066428 · Cited by 561
“R. C. Fong and A. Vedaldi [52] explored to learn the smallest region to delete, which is to find the optimal 𝒎 ∗ : where 𝒎 is the soft mask, 𝑓 𝑐 (𝑥 0 ; 𝒎) represents the loss of the network for an image 𝑥 0 with the soft mask, and 𝑛 is the number of pixels.”
BackgroundExplainable Prediction of Medical Codes from Clinical Text
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1100 · Cited by 597
“Lei et al. (2016) model “rationales” through a latent variable, which tags each word as relevant to the document label.”
BackgroundDeep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 335
“Doshi-Velez and Kim [25] identify two classes of evaluation for interpretability: (1) the model is useful for a practical (and perhaps simplified) application used as a proxy to test interpretability [50], [62], [49] or (2) the model is learned using a specific hypothesis class already established to be interpretable…”
MethodTowards Robust Interpretability with Self-Explaining Neural Networks
Alvarez-Melis, D., & Jaakkola, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 397
“More related to our approach is the work of Lei et al. [11] and Al-Shedivat et al. [19].”
MethodSelf-citationERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Multi-grained Attention Network for Aspect-Level Sentiment Classification
Fan, F., Feng, Y., & Zhao, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1380 · Cited by 520
“Attention mechanisms (Wang et al., 2016; Lei et al., 2016; Li et al., 2017) are studied to enhance the influence of aspects on the final representation for prediction.”
BackgroundExplain Yourself! Leveraging Language Models for Commonsense Reasoning
Rajani, N. F., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1487 · Cited by 385
“Lei et al. (2016) proposed an approach for rationale generation for sentiment analysis by highlighting complete phrases in the input text that by itself is sufficient to predict the desired output.”
BackgroundLi, O., Liu, H., Chen, C., & Rudin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11771 · Cited by 541
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
“For example, Lei et al. (2016) defined a generator that learns a distribution over text fragments as candidate rationales for justifying predictions, evaluated on sentiment analysis.”
BackgroundRight for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515
“This approach to explanations can be domain- and model-specific (e.g. “annotator rationales” used to explain text classifications in [Li et al., 2016; Lei et al., 2016; Zhang et al., 2016]).”
MethodInterpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M., Preece, A., Julier, S., Rao, R. M., et al. (2017) · 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 422
“[34] developed a local explanation approach that reveals the most relevant sentences in sentiment prediction from text documents.”
BackgroundAccountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457
Using deep learning to model the hierarchical structure and function of a cell
Ma, J., Yu, M. K., Fong, S., Ono, K., Sage, E., Demchak, B., Sharan, R., & Ideker, T. (2018) · Nature Methods · doi:10.1038/nmeth.4627 · Cited by 435
“Therefore, in attention-based neural networks 42,43 , a separate module preselects key ‘inter-pretable’ features for input to a black-box model.”
BackgroundDeep learning in drug discovery: an integrative review and future challenges
Askr, H., Elgeldawi, E., Aboul Ella, H., Elshaier, Y. A. M. M., Gomaa, M. M., & Hassanien, A. E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10306-1 · Cited by 390
Sentiment Analysis: Mining Opinions, Sentiments, and Emotions
Liu, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108639286 · Cited by 362
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382
“To facilitate research on this task, we construct SCIFACT, an expert-annotated dataset of 1,409 scientific claims accompanied by abstracts that support or refute each claim, and annotated with rationales (Lei et al., 2016) justifying each SUPPORTS / REFUTES decision.”
MethodXGNN: Towards Model-Level Explanations of Graph Neural Networks
Yuan, H., Tang, J., Hu, X., & Ji, S. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403085 · Cited by 272
“According to [21, 42], the loss function for the action at at step t can be mathematically written as”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
“We build on recent developments in interpretable machine learning that provide useful frameworks for generating explanations of machine predictions [34, 35, 45, 50, 64, 65].”
BackgroundMethodExplanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
Dhurandhar, A., Chen, P., Luss, R., Tu, C., Ting, P., Shanmugam, K., & Das, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07623 · Cited by 144
“Other works also investigate methods of the type (b) for vision [34, 35, 29] and NLP applications [22].”
MethodR3: Reinforced Ranker-Reader for Open-Domain Question Answering
Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., & Jiang, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12053 · Cited by 351
“For example, Lei, Barzilay, and Jaakkola 2016 propose to first extract informative text fragments then feed them to text classification and question retrieval models.”
BackgroundBERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
“Methods that learn to explain by extracting rationales from the input and only using those rationales to make a prediction are faithful by design [8, 13, 41, 60].”
BackgroundCutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325
“Lei et al. (2016), jointly modeled both a classification decision, and the selection of the most relevant subsection of a document for making the classification decision.”
MethodPathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., & Boyd-Graber, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1407 · Cited by 245
“m is reduction . More importantly, Anchors is designed to nd explanations for model predictions. Otherinterpretationmethodshaveusedsimilar search process based on input perturbation (Li et al., 2016; Lei et al., 2016) (Ribeiro et al., 2016; Li et al., 2016; Ebrahimi et al., 2017). Our work is fundamentally different i…”
MethodPost-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207
“The idea is that by enforcing the model to reason about its behavior, the model can generalize better (Lei et al., 2016; Camburu et al., 2018; Liu et al., 2019a; Rajani et al., 2019; Kumar and Talukdar, 2020; Latcinnik and Berant, 2020).”
BackgroundExplainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
“Rationales [258] is a method for justifying the predictions made by DNNs in natural language processing tasks, such as sentiment analysis, by extracting pieces of the input text as justifications, or rationales.”
MethodA Simple, Fast Diverse Decoding Algorithm for Neural Generation
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08562 · Cited by 195
“, 2015), neural model visualization (Lei et al., 2016), and coreference (Clark and Manning, 2016).”
BackgroundOne Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268
“Work in the natural language processing and computer vision domains that generates rationales/explanations derived from input text [11, 28, 39] would be considered as local self explanations.”
BackgroundSeq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., & Rush, A. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865044 · Cited by 239
Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E. T., Wu, J. T., Welt, J., Foote, J., Moseley, E. T., Grant, D. W., Tyler, P. D., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0192360 · Cited by 208
“In this approach, users are shown a list of phrases or concepts that are most salient for a particular prediction; this list either manifests as an actual list or as highlights of the original text [67, 68].”
BackgroundExplainability in human–agent systems
Rosenfeld, A., & Richardson, A. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09408-y · Cited by 224
“Within these papers, an explanation is inspected to insure that it provides the necessary information in the context of a specific application [84, 102].”
BackgroundInfluentialUnderstanding Convolutional Neural Networks for Text Classification
Jacovi, A., Sar Shalom, O., & Goldberg, Y. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5408 · Cited by 226
“While recent works have begun to supply the means of interpreting predictions (Alvarez-Melis and Jaakkola, 2017; Lei et al., 2016; Guo et al., 2018), interpreting neural NLP models remains an under-explored area.”
BackgroundHolzinger, A., Dehmer, M., Emmert-Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurisica, I., & Díaz-Rodríguez, N. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 213
“…of providing humanly understandable statements, e.g., logical relations, showing positive words drawing to a conclusion, verbal chunks or sentences [199] that indicate causality, and that the model produces explanations which are non-contradictory, non-redundant, fluent and cover all important…”
BackgroundSentiment Analysis by Capsules
Wang, Y., Sun, A., Han, J., Liu, Y., & Zhu, X. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186015 · Cited by 210
“The approach in [19] combines two modular components, generator and encoder, to extract pieces of input text as justifications.”
MethodInvestigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219
“been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating posthoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodMueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
“…Jain et al., 2016; Krizhevsky, et al., 2015; Kumar, Wong, and Taylor, 2017; Landecker, Thomure, Bettencourt, Mitchell, Kenyon, and Brumby, 2013; Lei, Barzilay, and Jaakkola, 2016; Li, Chen, Hovy, and Jurafsky, 2015; Li, Yosinski, Clune, Lipson, and Hopcroft, 2015; Liu ,et al., 2017; Mahendran…”
BackgroundA Diagnostic Study of Explainability Techniques for Text Classification
Atanasova, P., Simonsen, J. G., Lioma, C., & Augenstein, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.263 · Cited by 177
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Wortman Vaughan, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.8036d03b · Cited by 210
A causal framework for explaining the predictions of black-box sequence-to-sequence models
Alvarez-Melis, D., & Jaakkola, T. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1042 · Cited by 178
“Interpretability has been approached primarily from two main angles: model interpretability , i.e., making the architecture itself interpretable, and prediction interpretability , i.e., explaining particular predictions of the model (cf. (Lei et al., 2016)).”
BackgroundMethodSelf-citationPeng, Y., Xie, G., Shao, P., Ren, W., Li, M., Hu, Y., Yang, L., Shi, H., & Luo, X. (2022) · Applied Catalysis B: Environmental · doi:10.1016/j.apcatb.2022.121345 · Cited by 201
Bastings, J., & Filippova, K. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.14 · Cited by 150
Towards Explainable NLP: A Generative Explanation Framework for Text Classification
Liu, H., Yin, Q., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1560 · Cited by 160
“Lei et al. (2016) rationalize neural prediction by extracting the phrases from the input texts as the explanations.”
MethodWolf, M. J., Miller, K., & Grodzinsky, F. S. (2017) · ACM SIGCAS Computers and Society · doi:10.1145/3144592.3144598 · Cited by 172
Towards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 120
Neural Machine Translation and Sequence-to-sequence Models: A TutorialPreprint
Neubig, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01619 · Cited by 217
“There are also methods for hard attention that make a hard binary decision about whether to focus on a particular context, with motivations ranging from learning explainable models [64], to processing text incrementally [122, 45].”
MethodClassification of Explainable Artificial Intelligence Methods through Their Output Formats
Vilone, G., & Longo, L. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030032 · Cited by 152
“[128] 2016 L C T Relevance and Discriminative Loss Hendricks et al.”
BackgroundLearning to Faithfully Rationalize by Construction
Jain, S., Wiegreffe, S., Pinter, Y., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.409 · Cited by 136
Coarse-to-Fine Question Answering for Long Documents
Choi, E., Hewlett, D., Uszkoreit, J., Polosukhin, I., Lacoste, A., & Berant, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1020 · Cited by 177
“Treating sentence selection as a latent variable has been explored in classifi-cation (Yessenalina et al., 2010; Lei et al., 2016), however, to our knowledge, has not been applied for question answering.”
BackgroundConsiderations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176
“…et al., 2016, Liu and Tsang, 2016, Safavian and Landgrebe, 1991, Wang et al., 2017]), methods of probing black box models (e.g. [Ribeiro et al., 2016, Lei et al., 2016, Adler et al., 2016, Selvaraju et al., 2016, Smilkov et al., 2017, Shrikumar et al., 2016, Kindermans et al., 2017, Ross et al.,…”
BackgroundMethodGhimire, P., Kim, K., & Acharya, M. (2024) · Buildings · doi:10.3390/buildings14010220 · Cited by 175
“These models can perform diverse actions including responding to questions, conducting searches, making predictions, and carrying out requested behaviors [10].”
Everitt, T., Lea, G., & Hutter, M. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/768 · Cited by 105
Asking ‘Why’ in AI: Explainability of intelligent systems – perspectives and challenges
Preece, A. (2018) · Intelligent Systems in Accounting, Finance and Management · doi:10.1002/isaf.1422 · Cited by 153
“WhiletheapproachisgenerallyapplicabletoMLsystems,its performance has yet to be demonstrated on deep neural networks.”
BackgroundOn the Privacy Risks of Model Explanations
Shokri, R., Strobel, M., & Zick, Y. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462533 · Cited by 111
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Han, X., Wallace, B. C., & Tsvetkov, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.492 · Cited by 104
Visible Machine Learning for Biomedicine
Yu, M. K., Ma, J., Fisher, J., Kreisberg, J. F., Raphael, B. J., & Ideker, T. (2018) · Cell · doi:10.1016/j.cell.2018.05.056 · Cited by 162
“Generally, model interpretation tries to explain a model’s internal logic after a model has been trained (Ribeiro et al., 2016) or to force the model to have fewer parameters, which makes it easier to interpret (Lei et al., 2016).”
BackgroundYe, H., Jiang, X., Luo, Z., & Chao, W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1168 · Cited by 136
“The justification for charge decision is as important as deciding the charge itself (Hendricks et al., 2016; Lei et al., 2016).”
BackgroundReinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
Shen, T., Zhou, T., Long, G., Jiang, J., Wang, S., & Zhang, C. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/604 · Cited by 143
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
“While the algorithms are sophisticated, the measures of interpretability are often not—it is common for researchers to simply appeal to the face-validity of the results that they find (i.e., “this result makes sense to the human reader”) [Caruana et al., 2015, Lei et al., 2016, Ribeiro et al., 2016].”
BackgroundMethodInfluentialOn the Explainability of Natural Language Processing Deep Models
Zini, J. E., & Awad, M. (2023) · ACM Computing Surveys · doi:10.1145/3529755 · Cited by 120
“According to [55], a “rationalež can be deined as “sub-sets of the words from the input text that satisfy two key properties.”
BackgroundMethodInfluentialLet's Talk About Race: Identity, Chatbots, and AI
Schlesinger, A., O'Hara, K. P., & Taylor, A. S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173889 · Cited by 138
“There is growing research on interpretability of neural networks, but interpretability of a neural net does not necessarily mean there is room for adjustability [5,57].”
Recent advances in document summarization
Yao, J.-G., Wan, X., & Xiao, J. (2017) · Knowledge and Information Systems · doi:10.1007/s10115-017-1042-4 · Cited by 177
“Sentences generated for such scenarios should be concise and coherent, while being sufficient for making the same predictions when only using these sentences alone without referring to the full passage of review [95].”
BackgroundIntegrating Stance Detection and Fact Checking in a Unified Corpus
Baly, R., Mohtarami, M., Glass, J., Màrquez, L., Moschitti, A., & Nakov, P. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2004 · Cited by 126
“This is best supported by showing extracts (such as sentences or phrases) from the retrieved documents that illustrate the detected stance (Lei et al., 2016).”
BackgroundHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
“We use a rationale-style neural architecture [52] as the classifier underpinning our tool, producing both explanations and predictions.”
MethodA Just and Comprehensive Strategy for Using NLP to Address Online Abuse
Jurgens, D., Hemphill, L., & Chandrasekharan, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1357 · Cited by 121
“Therefore, abuse detection methods are needed to justify why a particular act was a violation to build legitimacy; a natu-ral starting point for NLP in building legitimacy is recent work from explainable ML (Ribeiro et al., 2016; Lei et al., 2016; Carton et al., 2018).”
BackgroundAnswering Questions about Charts and Generating Visual Explanations
Kim, D. H., Hoque, E., & Agrawala, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376467 · Cited by 120
“Lei et al. [33] present a method for sentiment prediction in review text, where it automatically extracts phrases from the review that leads to a particular prediction score.”
MethodMulti-Objective Molecule Generation using Interpretable SubstructuresPreprint
Jin, W., Barzilay, R., & Jaakkola, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03244 · Cited by 330
“Prior work on interpretability primarily focuses on finding rationales (i.e., explanations) of model predictions in image and text classification (Lei et al., 2016; Ribeiro et al., 2016; Sundararajan et al., 2017) and molecule property prediction (McCloskey et al., 2019; Ying et al., 2019; Lee et…”
BackgroundSelf-citationDo Feature Attribution Methods Correctly Attribute Features?
Zhou, Y., Booth, S., Ribeiro, M. T., & Shah, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i9.21196 · Cited by 103
A survey on XAI and natural language explanations
Cambria, E., Malandri, L., Mercorio, F., Mezzanzanica, M., & Nobani, N. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2022.103111 · Cited by 131
“Word annotation is the fastest and simplest method of the text category, in which a message is conveyed by highlighting or changing the colour of a specific part of the text Liu, Zhang and Gulla (2020); Ribeiro et al. (2016); Lei, Barzilay and Jaakkola (2016); Kim et al. (2015a).”
MethodInterpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models
Louis, A., DIJCK, G. V., & Spanakis, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30232 · Cited by 247
TED: Teaching AI to Explain its Decisions
Hind, M., Wei, D., Campbell, M., Codella, N. C. F., Dhurandhar, A., Mojsilović, A., Natesan Ramamurthy, K., & Varshney, K. R. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314273 · Cited by 92
“Work in the natural language processing and computer vision domains that generate rationales/explanations derived from input text (e.g., (Lei, Barzilay, and Jaakkola 2016; Ainur, Choi, and Cardie 2010; Hendricks et al. 2016)).”
Background
Metadata
- Authors
- Lei, T., Barzilay, R., & Jaakkola, T.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 107-117
- OpenAlex
- W2440159969
- Cited by
- 777
- References
- 27
- Citations by year
- 2026: 27 · 2025: 33 · 2024: 39 · 2023: 88 · 2022: 92 · 2021: 138 · 2020: 126 · 2019: 101 · 2018: 78 · 2017: 39 · 2016: 4
- FWCI
- 39.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- explanation generation, question retrieval, model regularization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.