The DOI matches its DataCite record; 23 of the 27 references it cites were found in a registry or an index.
e-SNLI: Natural Language Inference with Natural Language Explanations
Open accessAbstract
In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time. In this work, we extend the Stanford Natural Language Inference dataset with an additional layer of human-annotated natural language explanations of the entailment relations. We further implement models that incorporate these explanations into their training process and output them at test time. We show how our corpus of explanations, which we call e-SNLI, can be used for various goals, such as obtaining full sentence justifications of a model's decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets. Our dataset thus opens up a range of research directions for using natural language explanations, both for improving models and for asserting their trust.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Rationale supervision work, such as e-SNLI (Camburu et al., 2018) and ERASER (DeYoung et al., 2020), provides gold explanations for post hoc evaluation or training but does not embed quality criteria into the generation process itself.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Camburu et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“researchers increasingly draw on an LLMs' own text generation capabilities to produce natural language explanations (NLEs) alongside predictions [22,23]”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although LLMs can be prompted to self-explain [2], these explanations are unreliable [8].”
From ProSLM: A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering (2024) · cites this paper as [2]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The PredictAndExplain method, introduced in [2], illustrates that models can be trained to generate explanations of their outputs without sacrificing performance, but this approach requires access to a reliable explanation-augmented dataset for training.”
From ProSLM: A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering (2024) · cites this paper as [2]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Therefore, our encoders are 2048-bidirectional-LSTMs [15] with max-pooling, resulting in a sentence representation dimension of 4096.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“In computer vision, pretrained ImageNet-based encoders [10] provide standard image feature extractors for other downstream tasks.”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“Attention-based models, such as [2, 25], offer some degree of interpretability and have been shown to also improve performance on downstream tasks.”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“To address this deficiency, we have collected a large corpus of human-annotated explanations for the Stanford Natural Language Inference (SNLI) dataset [3].”
BackgroundMethodA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 1×
“We test how our explanations affect the direct transfer in both label prediction and explanation generation by looking at SICK-E [20] and MultiNLI [26].”
BackgroundSupervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · cited 8×
“Throughout our experiments, our models follow the architecture presented in Conneau et al. [7], as we build directly on top of their code 2 .”
BackgroundMethodInfluentialA Decomposable Attention Model for Natural Language Inference
Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1244 · Cited by 1,351 · cited 3×
“Glockner et al. show that models achieving high accuracies on SNLI, such as [21, 22, 6], show dramatically reduced performance on this simpler dataset, while the model of Chen et al. [5] is more robust due to incorporating exter-nal knowledge.”
BackgroundResultKiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · cited 2×
“Traditional approaches make use of very large unsupervised datasets [17, 14], taking weeks to train.”
BackgroundAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 3×
“Recently, Gururangan et al. [13] cast doubt on whether models trained on SNLI are learning to understand language, or are largely fixating on spurious correlations, also called artifacts.”
BackgroundInfluential
Research works printed but not linked: 6
- [4][4] Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. (2015). Listen, attend and spell. CoRR , abs/1508.01211. - · arXiv
- [5][5] Chen, Q., Zhu, X., Ling, Z., Inkpen, D., and Wei, S. (2017). Natural language inference with external knowledge. CoRR , abs/1711.04289. - · arXiv
- [6][6] Chen, Q., Zhu, X., Ling, Z., Wei, S., and Jiang, H. (2016). Enhancing and combining sequential and tree LSTM for natural language inference. CoRR , abs/1609.06038. - · arXiv
- [10][10] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A largescale hierarchical image database. In Proc. of CVPR . -
- [19][19] Liu, P., Qiu, X., and Huang, X. (2016). Modelling interaction of sentence pair with coupledlstms. CoRR , abs/1605.05573. - · arXiv
- [24][24] Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "why should I trust you?": Explaining the predictions of any classifier. CoRR , abs/1602.04938. - · arXiv
Cited by
Sorted by citations
Items 101–200 of 369
Requirements Satisfiability with In-Context Learning
Santos, S., Breaux, T. D., Norton, T. B., Haghighi, S., & Ghanavati, S. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00025 · Cited by 27
“The extended dataset e-SNLI includes explanations to support the predicted label [7].”
BackgroundTian, Y., Han, Y., Chen, X., Wang, W. W., & Chawla, N. V. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703577 · Cited by 63
Learning to Explain: Generating Stable Explanations Fast
Situ, X., Ingrid Zukerman, A., Paris, C. L., Maruf, S., & Haffari, G. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.415 · Cited by 27
“Unlike these approaches, we do not change the classifier or require an expensive process to collect human rationales, as done in (Camburu et al., 2018).”
MethodEntailment Tree Explanations via Iterative Retrieval-Generation Reasoner
Ribeiro, D. N., Wang, S., Ma, X., Dong, R., Wei, X., Zhu, H., Chen, X., Xu, P., Huang, Z., Arnold, A., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.35 · Cited by 23
Harnessing Prior Knowledge for Explainable Machine Learning: An Overview
Beckh, K., Müller, S., Jakobs, M., Toborek, V., Tan, H., Fischer, R., Welke, P., Houben, S., & von Rueden, L. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00038 · Cited by 23
“For a natural language inference task, Camburu et al. [85] incorporate explanations of textual entailment, i.e., whether a premise sentence entails a hypothesis sentence, into the training process using a negative log-likelihood for classification and an explanation loss.”
MethodCorrectness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21
On the Diversity and Limits of Human Explanations
Tan, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.158 · Cited by 33
“Camburu et al. (2018) and Rajani et al. (2019) collect both evidence and proximal mechanism.”
BackgroundMethodInfluentialE-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning
Chen, J., Xu, R., Fu, Z., Shi, W., Li, Z., Zhang, X., Sun, C., Li, L., Xiao, Y., & Zhou, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.311 · Cited by 30
“vić, 2021): 1) highlights explanations (Camburu et al., 2018; Yang et al., 2018; Thorne et al., 2018; Kwiatkowski et al., 2019), which are subsets of the input elements to explain a prediction, e.”
BackgroundWhy do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 26
“…more interpretable via neural module networks (Jiang and Bansal, 2019; Gupta et al., 2020), graph–based networks (Tu et al., 2019; Qiu et al., 2019), pipeline models (Lehman et al., 2019), or by generating textual explanations (Camburu et al., 2018; Rajani et al., 2019; Liu et al., 2019a).”
BackgroundUnification-based Reconstruction of Multi-hop Explanations for Science Questions
Valentino, M., Thayaparan, M., & Freitas, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.15 · Cited by 26
“…development of models for explanation generation is being explored in different NLP tasks, ranging from open domain question answering (Yang et al., 2018; Thayaparan et al., 2019), to textual entailment (Camburu et al., 2018) and natural language premise selection (Ferreira and Freitas, 2020b,a).”
BackgroundLEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation
Lee, D., Khanna, R., Lin, B. Y., et al. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.42 · Cited by 20
“Another effort can be found in (Camburu et al., 2018) work to extend the Stanford Natural Language Inference dataset with natural language explanations––this extension was done for the important textual entailment recognition task.”
BackgroundLearning from the Best: Rationalizing Predictions by Adversarial Information Calibration
Sha, L., Camburu, O.-M., & Lukasiewicz, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17623 · Cited by 21
“2019; Carton, Mei, and Resnick 2018) and natural language explanations (Hendricks et al. 2016; Camburu et al. 2018; Park et al. 2018; Kim et al. 2018).”
Self-citationUNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Chan, A., Sanjabi, M., Mathias, L., Tan, L., Nie, S., Peng, X., Ren, X., & Firooz, H. (2022) · doi:10.18653/v1/2022.bigscience-1.5 · Cited by 26
“Meanwhile, more existing works focus on improving the plausibility of free-text rationales (Narang et al., 2020; Lakhotia et al., 2020; Camburu et al., 2018), often with task-specific pipelines (Rajani et al., 2019; Kumar & Talukdar, 2020).”
BackgroundInfluentialMETGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation
Hong, R., Zhang, H., Yu, X., & Zhang, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.145 · Cited by 21
KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference
Chen, Q., Ji, F., Zeng, X., Li, F., Zhang, J., Chen, H., & Zhang, Y. (2021) · doi:10.18653/v1/2021.acl-long.196 · Cited by 21
“(Camburu et al., 2018) extend the SNLI dataset to e-SNLI 8 with natural language explanations of the ground truth labels.”
BackgroundInfluentialLIREx: Augmenting Language Inference with Relevant Explanations
Zhao, X., & Vydiswaran, V. G. V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17708 · Cited by 32
Hase, P., Zhang, S., Xie, H., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.390 · Cited by 62
“We introduce a new rationalizing model, ST-R A , while also testing a reasoning multi-task model, MT-R E , and two other methods from past works (Camburu et al., 2018; Rajani et al., 2019).”
BackgroundMethodInfluentialmultiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
Saha, S., Yadav, P., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.287 · Cited by 19
“…2016; Yu et al., 2019; DeYoung et al., 2020) where a subset of the input text explain a prediction, (2) free-form or natural language explanations (Camburu et al., 2018; Ra-jani et al., 2019; Zhang et al., 2020; Kumar and Talukdar, 2020) that are not constrained to the input, and (3)…”
BackgroundGonzález, A. V., Bansal, G., Fan, A., Mehdad, Y., Jia, R., & Iyer, S. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.95 · Cited by 20
“Many prior works evaluate the quality of NL explanations by comparing similarity with human-written explanations for tasks such as NLI, common-sense reasoning and fact verification (DeYoung et al., 2020; Paranjape et al., 2020; Swanson et al., 2020; Camburu et al., 2018; Rajani et al., 2019).”
BackgroundMethodMDACE: MIMIC Documents Annotated with Code Evidence
Cheng, H. N., Jafari, R., Russell, A., Klopfer, R., Lu, E., Striner, B., & Gormley, M. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.416 · Cited by 18
NORMSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly
Fung, Y. R., Chakrabarty, T., Guo, H., Rambow, O., Muresan, S., & Ji, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.941 · Cited by 49
To what extent do human explanations of model behavior align with actual model behavior?
Prasad, G., Nie, Y., Bansal, M., Jia, R., Kiela, D., & Williams, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.1 · Cited by 22
“Although we think these methods are reasonable starting points, further work is required both to validate these methods as well as explore other conversion methods such as using explanation generation (e.g., Hase et al. 2020; Camburu et al. 2018).”
BackgroundMethodCLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_5 · Cited by 20
Employing Co-Learning to Evaluate the Explainability of Multimodal Sentiment Analysis
Jain, D. K., Rahate, A., Joshi, G., Walambe, R. A., & Kotecha, K. V. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3176403 · Cited by 17
“Incorporating external knowledge by knowledge infusion techniques [10] is also used for natural language explanations.”
MethodTextual entailment as an evaluation metric for abstractive text summarization
Bhuyan, S. S., Mahanta, S. K., Pakray, P., & Favre, B. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100028 · Cited by 18
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming
Zhang, H., Huang, J., Li, Z., Naik, M., & Xing, E. P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.191 · Cited by 31
Alignment Rationale for Natural Language Inference
Jiang, Z., Zhang, Y., Zhao, Y., Zhao, J., & Liu, K. (2021) · doi:10.18653/v1/2021.acl-long.417 · Cited by 17
“We use four datasets SNLI (Bowman et al., 2015), E SNLI (Camburu et al., 2018), BNLI (Glockner et al., 2018) and HANS as our testbeds.”
BackgroundMethodVisual Attention Prompted Prediction and Learning
Zhang, Y., Pan, B., Gu, S., Bai, G., Qiu, M., Yang, X., & Zhao, L. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/610 · Cited by 18
“These signals are aligned with prediction loss to further enhance model performance [Shen et al. , 2021; Camburu et al. , 2018; Hajialigol et al. , 2023; Bai and Zhao, 2022].”
FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation
Lakhotia, K., Paranjape, B., Ghoshal, A., Yih, S., Mehdad, Y., & Iyer, S. V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.301 · Cited by 26
“We do not evaluate on the ERASER datasets of e-SNLI and CoS-E since they only use single sentence input passages.”
MethodReconstructing Implicit Knowledge with Language Models
Becker, M., Liang, S., & Frank, A. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.2 · Cited by 17
“, 2020), which offers statements expressing generic knowledge; the other on e-SNLI (Camburu et al., 2018), which comprises explanations of inferential commonsense knowledge.”
BackgroundMethodA survey on improving NLP models with human explanations
Hartmann, M., & Sonntag, D. (2022) · doi:10.18653/v1/2022.lnls-1.5 · Cited by 24
Less is More: Attention Supervision with Counterfactuals for Text Classification
Choi, S. L., Park, H., Yeo, J., & Hwang, S.-W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.543 · Cited by 16
“Specifically, we start by loosening the definition of human annotation (Camburu et al., 2018; Zhong et al., 2019) into the task-level annotation: it reduces annotation cost to the size of vocabulary, or often to zero, when public resources such as sentiment lexicon replace such annotation.”
MethodRationale-Inspired Natural Language Explanations with Commonsense.Preprint
Majumder, B. P., Camburu, O.-M., Lukasiewicz, T., & McAuley, J. (2021) · arXiv (Cornell University) · Cited by 16
Self-citationREV: Information-Theoretic Evaluation of Free-Text Rationales
Chen, H., Brahman, F., Ren, X., Ji, Y., Choi, Y., & Swayamdipta, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.112 · Cited by 29
Explaining Data Patterns in Natural Language with Language Models
Singh, C., Morris, J. X., Aneja, J., Rush, A. M., & Gao, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.3 · Cited by 17
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
Don-Yehiya, S., Venezian, E., Raffel, C., Slonim, N., & Choshen, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.46 · Cited by 44
A Fine-grained Interpretability Evaluation Benchmark for Neural NLP
Wang, L., Shen, Y., Peng, S., Zhang, S., Xiao, X., Liu, H., Tang, H., Chen, Y., Wu, H., & Wang, H. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.6 · Cited by 21
Li, Z., Sharma, P., Lù, X. H., Cheung, J. C. K., & Reddy, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.75 · Cited by 25
“…ef-516 forts in collecting and harnessing explanation data for language understanding and reasoning tasks, such as natural language inference (Camburu et al., 2018; Kumar and Talukdar, 2020), commonsense question answering (Rajani et al., 2019), document classification (Srivastava et al.,…”
BackgroundRecognizing textual entailment: A review of resources, approaches, applications, and challenges
Putra, I. M. S., Siahaan, D., & Saikhu, A. (2023) · ICT Express · doi:10.1016/j.icte.2023.08.012 · Cited by 19
Generating Fact Checking Explanations
Atanasova, P. (2020) · Springer eBooks · doi:10.1007/978-3-031-51518-7_4 · Cited by 14
The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient SubsetsPreprint
Camburu, O.-M., Giunchiglia, E., Foerster, J., Lukasiewicz, T., & Blunsom, P. (2020) · arXiv.org · doi:10.48550/arxiv.2009.11023 · Cited by 24
“A large number of explanatory methods have been developed with the goal of shedding light on black-box neural models (Ribeiro et al., 2016; Lundberg and Lee, 2017; Simonyan et al., 2014; Sundararajan et al., 2017; Carter et al., 2019; Camburu et al., 2018; Park et al., 2018; Kim et al., 2018).”
BackgroundSelf-citationRissanen Data Analysis: Examining Dataset Characteristics via Description LengthPreprint
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03872 · Cited by 13
“Dataset and Model We use e-SNLI (Camburu et al., 2018), which annotated each example in SNLI (Bowman et al.”
BackgroundMethodInfluentialThinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 32
“…[Kassner and Schütze, 2020, Richardson and Sabharwal, 2019, Richardson et al., 2020], and can be fine-tuned to correctly draw complex deductive inferences [Clark et al., 2020, Betz et al., 2020] and to generate informal reasons [Rudinger et al., 2020, Camburu et al., 2018, Brahman et al., 2020].”
BackgroundConnecting Attributions and QA Model Behavior on Realistic Counterfactuals
Ye, X., Nair, R., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.447 · Cited by 15
“At-tributions are a flexible explanation format and can be applied to many domains, including sentiment analysis (Guan et al., 2019; De Cao et al., 2020), visual recognition (Simonyan et al., 2013), and natural language inference (Camburu et al., 2018; Thorne et al., 2019).”
BackgroundAn Empirical Study on Explanations in Out-of-Domain Settings
Chrysostomou, G., & Αλέτρας, Ν. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.477 · Cited by 17
Investigating the Benefits of Free-Form Rationales
Sun, J., Swayamdipta, S., May, J., & Ma, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.432 · Cited by 23
C-STS: Conditional Semantic Textual Similarity
Deshpande, A., Jimenez, C., Chen, H., Murahari, V., Graf, V., Rajpurohit, T., Kalyan, A., Chen, D., & Narasimhan, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.345 · Cited by 17
Predicting explainable dementia types with LLM-aided feature engineering
Kashyap, A. M., Rao, D., Boland, M. R., Shen, L., & Callison-Burch, C. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf156 · Cited by 16
“Other approaches generate Natural Language Explanations in addition to the outcome of interest (Camburu et al. 2018, Rajani et al. 2019, Kumar and Talukdar 2020).”
Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension
Inoue, N., Trivedi, H. K., Sinha, S., Balasubramanian, N., & Inui, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.490 · Cited by 14
“Abstractive explainer A similar pipeline model has been proposed for textual entailment (Camburu et al., 2018) and commonsense QA (Rajani et al., 2019), where the model first generates an explanation, and then the downstream classifier consumes it to predict a task label.”
BackgroundMethodInfluentialINTERACTION: A Generative XAI Framework for Natural Language Inference Explanations
Yu, J., Cristea, A. I., Harit, A., Sun, Z., Aduragba, O. T., Shi, L., & Moubayed, N. A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892336 · Cited by 15
“Our work uses the declarative induction technique , which tackles the challenging task of providing human-readable representations as part of the results, such as in [1], [22].”
BackgroundMethodInfluentialGenerating knowledge aware explanation for natural language inference
Yang, Z., Xu, Y., Hu, J., & Dong, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103245 · Cited by 15
F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering
Schuff, H., Adel, H., & Vu, N. T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.575 · Cited by 15
“In natural language processing (NLP), a variety of tasks have been addressed regarding explainability of neural networks, such as textual entailment (Camburu et al., 2018), sentiment classification (Clos et al., 2017), machine translation (Stahlberg et al., 2018) and question answering (Yang et…”
BackgroundExplaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
Chen, H., Feng, S., Ganhotra, J., Wan, H., Gunasekara, C., Joshi, S., & Ji, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.306 · Cited by 15
“Datasets e-SNLI (Camburu et al., 2018) is natural language inference task, where the model predicts the semantic relationship between two input sentences as entailment, contradiction, or neutral.”
BackgroundMethodInfluential“Why is this misleading?”: Detecting News Headline Hallucinations with Explanations
Shen, J., Liu, J., Finnie, D., Rahmati, N., Bendersky, M., & Najork, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583375 · Cited by 17
“Furthermore, diferent from the case in the news domain HHD with limited labeled data, there are many large-scale publicly available NLI datasets [5, 7, 42, 58, 63].”
BackgroundMethodFixing Model Bugs with Natural Language Patches
Murty, S., Manning, C. D., Lundberg, S., & Ribeiro, M. T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.797 · Cited by 25
Sivamayilvelan, K., Rajasekar, E., Subramaniyaswamy, V., Balachandran, S., & Suresh, V. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111622 · Cited by 12
Yang, Q., Li, Y., Hu, B., Ma, L., Ding, Y., & Zhang, M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548284 · Cited by 13
“e-SNLI-VE is the current biggest VE-NLE dataset that combines SNLI-VE [41] and e-SNLI [5].”
BackgroundWhat’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 25
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts
Zhou, B. Z., Richardson, K., Yu, X., & Roth, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.142 · Cited by 19
InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations
Feldhus, N., Wang, Q., Anikina, T., Chopra, S., Oguz, C., & Möller, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.359 · Cited by 15
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 36
“In this work, we use the trained ETPA model 4 from Camburu et al. [4], which gave the highest percentage of correct explanations ( 64 .”
BackgroundMethodInfluentialSelf-citationLIREx: Augmenting Language Inference with Relevant ExplanationPreprint
Zhao, X., & Vydiswaran, V. G. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09157 · Cited by 12
Self-training with Few-shot Rationalization
Bhat, M. M., Sordoni, A., & Mukherjee, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.836 · Cited by 20
“These include Movies for sentiment analysis (Pang and Lee, 2004), e-SNLI (Camburu et al., 2018) for natural language inference, FEVER (Thorne et al., 2018) for fact extraction and verification, BoolQ (Clark et al., 2019) for reading comprehension, and Evidence Inference (Lehman et al., 2019) over…”
MethodKnowledge is a Region in Weight Space for Fine-tuned Language Models
Gueta, A., Venezian, E., Raffel, C., Slonim, N., Katz, Y., & Choshen, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.95 · Cited by 40
SIDU-TXT: An XAI algorithm for NLP with a holistic assessment approach
Jahromi, M. N. S., Muddamsetty, S. M., Jarlner, A. S. S., Høgenhaug, A. M., Gammeltoft‐Hansen, T., & Moeslund, T. B. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100078 · Cited by 11
Zero-shot LLM-guided Counterfactual Generation: A Case Study on NLP Model Evaluation
Bhattacharjee, A., Moraffah, R., Garland, J., & Liu, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825537 · Cited by 25
What Gets Echoed? Understanding the “Pointers” in Explanations of Persuasive Arguments
Atkinson, D., Srinivasan, K., & Tan, C. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1289 · Cited by 17
“but also informs the process of collecting explanations that machine learning systems learn from (Hancock et al., 2018; Rajani et al., 2019; Camburu et al., 2018).”
BackgroundWhat-if I ask you to explain: Explaining the effects of perturbations in procedural text
Rajagopal, D., Tandon, N., Clark, P., Dalvi, B. B., & Hovy, E. H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.300 · Cited by 12
“Recently, several datasets with natural language explanations have been introduced, e.g., in natural language inference (Camburu et al., 2018), visual question answering (Park et al., 2018), and multi-ears less protected → (MORE/+) sound enters the ear → (MORE/+) sound hits ear drum → (MORE/+) more…”
BackgroundInfluentialTang, Z. S., Hahn-Powell, G., & Surdeanu, M. (2020) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2020.acl-srw.23 · Cited by 11
“Similarly, (Blunsom et al., 2018) learned NL explanations for the natural language inference (NLI) task, and (Ye et al., 2018) applied this idea to crime case prediction.”
BackgroundOn Sample Based Explanation Methods for NLP: Faithfulness, Efficiency and Semantic Evaluation
Zhang, W., Huang, Z., Zhu, Y., Ye, G., Cui, X., & Zhang, F. (2021) · doi:10.18653/v1/2021.acl-long.419 · Cited by 14
“The Discussion of Explanation Faithfulness in NLP The issue of Faithfulness of Explanations was primarily discussed under the explanation generation context (Camburu et al., 2018) where there is no guarantee that a generated explanation would be faithful to a model’s inner-workings (Jacovi and Goldberg, 2020).”
BackgroundESPRIT: Explaining Solutions to Physical Reasoning Tasks
Rajani, N. F., Zhang, R., Tan, Y. C., Zheng, S., Weiss, J., Vyas, A., Gupta, A., Xiong, C., Socher, R., & Radev, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.706 · Cited by 15
“Several recent papers propose to use natural language for explanation and commonsense reasoning (Lei et al., 2016; Camburu et al., 2018; Forbes and Choi, 2017; Chai et al., 2018; Forbes et al., 2019; Rajani et al., 2019; DeYoung et al., 2020).”
BackgroundExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Li, D., Yu, J., Hu, B., Xu, Z., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.129 · Cited by 13
Where to start? Analyzing the potential value of intermediate models
Choshen, L., Venezian, E., Don-Yehiya, S., Slonim, N., & Katz, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.90 · Cited by 23
Explainable Video Entailment with Grounded Visual Evidence
Chen, J., & Golisano, Y. K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00203 · Cited by 10
“Natural language inference [9, 8, 26, 3] is the task of understanding if a hypothesis sentence is entailed or contradicted by a premise sentence, which is a fundamental task in natural language understanding.”
BackgroundWhat to Learn, and How: Toward Effective Learning from Rationales
Carton, S., Kanoria, S., & Tan, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.86 · Cited by 20
“• E-SNLI (Camburu et al., 2018”
Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction
Li, D., Hu, B., Chen, Q., Xu, T., Tao, J., & Zhang, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21342 · Cited by 13
“One practical approach is to extract prediction’s rationales from input (Lipton 2016; Camburu et al. 2018; Thorne et al. 2019).”
BackgroundCREST: A Joint Framework for Rationalization and Counterfactual Text Generation
Treviso, M., Ross, A., Guerreiro, N. M., & Martins, A. F. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.842 · Cited by 18
RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees
Liu, T., Guo, Q., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.483 · Cited by 14
Extractive and Abstractive Explanations for Fact-Checking and Evaluation of News
Kazemi, A., Li, Z., Pérez‐Rosas, V., & Mihalcea, R. F. (2021) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/2021.nlp4if-1.7 · Cited by 11
“Camburu et al. (2018) propose e-SNLI, a natural language (NL) inference dataset augmented with human-annotated NL explanations.”
BackgroundSILT: Efficient transformer training for inter-lingual inference
Huertas‐Tato, J., Martín, A. G., & Camacho, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116923 · Cited by 15
“[19] extend the SNLI dataset by including human explanation and show that it can be used to train new models able to provide justifications.”
BackgroundExPUNations: Augmenting Puns with Keywords and Explanations
Sun, J., Narayan-Chen, A., Oraby, S., Cervone, A., Chung, T., Huang, J., Liu, Y., & Peng, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.304 · Cited by 10
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations
Clinciu, M., Eshghi, A., & Hastie, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.202 · Cited by 13
Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools
Feldhus, N., Schwarzenberg, R., & Möller, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.11 · Cited by 9
“Finally, tangentially related to our work are datasets that supply explanations on top of texts and labels, usually collected from human anno-tators. e-SNLI (Camburu et al., 2018) probably is the most famous example in this line of work.”
BackgroundTowards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
Lai, C., Song, S., Meng, S., Li, J., Yan, S., & Hu, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.28065 · Cited by 9
On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs
Calderon, N., & Reichart, R. (2025) · Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) · doi:10.18653/v1/2025.naacl-long.29 · Cited by 9
Rationalizing predictions by adversarial information calibration
Sha, L., Camburu, O.-M., & Lukasiewicz, T. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103828 · Cited by 11
“Rationalization Selector-predictor [75, 8, 142, 17], natural language explanations [46, 13] Yes Yes”
BackgroundSelf-citationAutoCAD: Automatically Generate Counterfactuals for Mitigating Shortcut Learning
Wen, J., Zhu, Y., Zhang, J., Zhou, J., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.170 · Cited by 10
Japanese Realistic Textual Entailment Corpus
Hayashibe, Y. (2020) · Language Resources and Evaluation · Cited by 9
Delbari, Z., Moosavi, N. S., & Pilehvar, M. T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29743 · Cited by 10
“Motivated by the work of Mathew et al. (2021) and Camburu et al. (2018), where they employed explanations pertinent to the task to enhance performance, our goal is to employ rationales for strengthening our model’s understanding of the task.”
MethodHuman-annotated rationales and explainable text classification: a survey
Herrewijnen, E., Nguyen, D., Bex, F., & van Deemter, K. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1260952 · Cited by 11
Schuff, H., Yang, H.-Y., Adel, H., & Vu, N. T. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.3 · Cited by 9
“Compared to PRED - EXPL , the VANILLA model achieves a notable increase in label accuracy as well as BLEURT scores.”
BackgroundMethodResultInfluentialAre Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
Saha, S., Hase, P., Rajani, N. F., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.137 · Cited by 10
Robust Integration of Contextual Information for Cross-Target Stance Detection
Beck, T., Waldis, A., & Gurevych, I. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.43 · Cited by 10
“To showcase, we use the e-SNLI (Camburu et al., 2018) corpus for natural language inference and the Snopes (Hanselowski et al., 2019) corpus for claim verification.”
MethodMulti-Defendant Legal Judgment Prediction via Hierarchical Reasoning
Lyu, Y., Hao, J., Wang, Z., Zhao, K., Gao, S., Ren, P., Chen, Z., Wang, F., & Ren, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.145 · Cited by 14
A Comparative Review of Deep Learning Techniques on the Classification of Irony and Sarcasm in Text
Boutsikaris, L., & Polykalas, S. E. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3515935 · Cited by 11
“…FLUTE demonstrates superior performance compared to those trained on dataset designed to enhance the Stanford natural language inference (e-SNLI) dataset by including human-annotated explanations [103], indicating its efficacy in generating high-quality explanations for figurative language instances.”
Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference
Feng, Y., Yang, X., Zhu, X., & Greenspan, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00458 · Cited by 11
“To evaluate the model inter-pretability, we derive the predicted rationales from the natural logic programs and compare it with human annotations in e-SNLI (Camburu et al., 2018).”
MethodCuring the SICK and Other NLI Maladies
Kalouli, A.-L., Hu, H., Webb, A. F., Moss, L. S., & de Paiva, V. (2022) · Computational Linguistics · doi:10.1162/coli_a_00465 · Cited by 8
“Last but not least, we think our method is in line with NLI datasets asking annotators to provide explanations for their annotations (Camburu et al. 2018; Nie et al. 2020), a step toward providing more information behind the annotators’ decision and building models that capture the nuance of human…”
FLamE: Few-shot Learning from Natural Language Explanations
Zhou, Y., Zhang, Y., & Tan, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.372 · Cited by 12
Zaman, K., & Belinkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.101 · Cited by 8
Metadata
- Authors
- Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P.
- Affiliations
- University of Oxford (GB)
- University College London (GB)
- Google (United States) (US)
- ORCID
- Lukasiewicz, Thomas
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 9539-9549
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1812.01193
- OpenAlex
- W2891012317
- Semantic Scholar
- c242438dac5aa4d9b13766c14240bb8426690d58
- Cited by
- 367
- References
- 27
- Citations by year
- 2025: 19 · 2024: 33 · 2023: 94 · 2022: 73 · 2021: 84 · 2020: 55 · 2019: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language inference, natural language explanations, entailment relations, explainable AI, out-of-domain transfer, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
