Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set.Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“61 (Landis and Koch, 1977).”
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“All models are based on pre-trained GloVe embeddings (Pennington et al., 2014), which are either fine-tuned during training (RESIDUAL-STACKED-ENCODER and ESIM) or stay fixed (DECOMPOSABLE ATTENTION).”
MethodInfluentialA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 6×
“The SNLI dataset (Stanford Natural Language Inference, Bowman et al., 2015) consists of 570k sentence-pairs manually labeled as entailment, contradiction, and neutral.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 3×
“The MultiNLI dataset (Multi-Genre Natural Language Inference, Williams et al., 2018) was collected similarly to SNLI, though covering a wider range of genres, and supporting a cross-genre evaluation.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426 · cited 1×
“Premises are image captions from Young et al. (2014), while hypotheses were generated by crowd-sourced workers who were shown a premise and asked to generate entailing, contradicting, and neutral sentences.”
BackgroundAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“Inspired by the work of Jia and Liang (2017) on reading comprehension, we create a new NLI test set with examples that capture various kinds of lexical knowledge (Table 1).”
MethodA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 2×
“Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
BackgroundMethodEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”
BackgroundMethodAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“Our results are in line with Gururangan et al. (2018) and Po-liak et al. (2018), who showed that the label can be identified by looking only at the hypothesis and exploiting annotation artifacts such as word choice and sentence length.”
ResultReasoning about Entailment with Neural Attention
Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582 · cited 1×
“…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
Research works printed but not linked: 2
- Christiane Fellbaum. 1998. WordNet . Wiley Online Library.
- Yixin Nie and Mohit Bansal. 2017. Shortcutstacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . · arXiv
Cited by
Items 201–300 of 350
Multilevel Image-Enhanced Sentence Representation Net for Natural Language Inference
Zhang, K., Lv, G., Wu, L., Chen, E., Liu, Q., Wu, H., Xie, X., & Wu, F. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2932410 · Cited by 10
“Besides, recent NLI models concerned more about the structures and global semantic of whole sentences, but less about external lexical knowledge, which led them to fail to capture many simple inferences that require lexical and world knowledge [48].”
BackgroundMethodEuropean Language Grid: A Language Technology Platform for Multilingual Europe
Rehm, G. (2022) · Cognitive technologies · doi:10.1007/978-3-031-17258-8 · Cited by 32
Infusing Knowledge into the Textual Entailment Task Using Graph Convolutional Networks
Kapanipathi, P., Thost, V., Patel, S. S., Whitehead, S., Abdelaziz, I., Balakrishnan, A., Chang, M., Fadnis, K., Gunasekara, C., Makni, B., et al. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6318 · Cited by 21
Bhatt, S., Jain, R., Dandapat, S., & Sitaram, S. (2021) · Cited by 23
Chen, H., Chen, B., & Zhou, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.354 · Cited by 17
“Similar situations, where models improve their performance by recognizing the spurious patterns, are identified by parts of the AI community, including annotation artifacts in natural language inference (NLI) (Poliak et al., 2018; Gururangan et al., 2018; Glockner et al., 2018) and shallow template matches in named ent…”
BackgroundInfluentialMitra, A., Shrivastava, I., & Baral, C. R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6371 · Cited by 12
Semantics Altering Modifications for Evaluating Comprehension in Machine Reading
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17622 · Cited by 16
Probing Pre-Trained Language Models for Disease Knowledge
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.266 · Cited by 13
Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions
Do, N., & Pavlick, E. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.181 · Cited by 15
“This technique has been used widely in probing/analysis work, e.g., (Glockner et al., 2018; Naik et al., 2018; Jia and Liang, 2017; Nie et al., 2019; McCoy et al., 2019).”
MethodUnsupervised Natural Language Inference Using PHL Triplet Generation
Varshney, N., Banerjee, P., Gokhale, T., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.159 · Cited by 10
Reich, A., Chen, J., Agrawal, A., Zhang, Y., & Yang, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.154 · Cited by 10
Manino, E., Rozanova, J., Carvalho, D. S., Freitas, A., & Cordeiro, L. C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.185 · Cited by 14
Curing the SICK and Other NLI Maladies
Kalouli, A.-L., Hu, H., Webb, A. F., Moss, L. S., & de Paiva, V. C. V. (2022) · Computational Linguistics · doi:10.1162/coli_a_00465 · Cited by 18
“…not allow the models to learn to efficiently solve inferences requiring true (semantic) understanding of language (Dasgupta et al. 2018; Nie, Wang, and Bansal 2018; Naik et al. 2018; Glockner, Shwartz, and Goldberg 2018; McCoy, Pavlick, and Linzen 2019; Richardson et al. 2020; Yanaka et al. 2020).”
Mullenbach, J., Gordon, J. A. R., Peng, N., & May, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1625 · Cited by 8
“Despite the strong performance of recent systems on SNLI (e.g., Chen et al., 2017; Parikh et al., 2016), Glockner et al. (2018) show that by making trivial changes to the test set, these methods suffered.”
BackgroundGenerating benchmarks for factuality evaluation of language models
Muhlgay, D., Ram, O., Magar, I., Levine, Y., Ratner, N., Belinkov, Y., Abend, O., Leyton-Brown, K., Shashua, A., & Shoham, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.06908 · Cited by 8
“…Datasets Contrastive evaluation, in which the model’s ability to discern between similar positive and negative examples is tested, is a long-standing approach used in a wide range of tasks (Sennrich, 2017; Burlot and Yvon, 2017; Glockner et al., 2018; Kaushik et al., 2020; Glock-ner et al., 2018).”
Adversarial natural language processing: overview, challenges, and policy implications
Shaw, L., Ansari, M. W., & Ekin, T. (2025) · Data & Policy · doi:10.1017/dap.2025.10030 · Cited by 8
Meaning in Context: Ontologically and linguistically motivated representations of objects and events
Basile, V., Caselli, T., & Radicioni, D. P. (2019) · Applied Ontology · doi:10.3233/ao-190221 · Cited by 7
“remain open – see for instance the work by Glockner et al. (2018), highlighting how modern neural”
BackgroundHow Fast can BERT Learn Simple Natural Language Inference?
Lin, Y., & Su, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.51 · Cited by 12
“Corpora such as HOTPOTQA (Yang et al., 2018), Breaking-NLI (Glockner et al., 2018), CommonSense QA (Talmor et al., 2019), DROP (Dua et al., 2019) and ROPES (Lin et al., 2019) have also been created recently to evaluate more diverse and difficult NLI cases.”
A Multilingual Evaluation of NER Robustness to Adversarial Inputs
Srinivasan, A., & Vajjala, S. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.4 · Cited by 11
Yanaka, H., & Mineshima, K. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.26 · Cited by 24
“…on various NLI tasks and discovered that generalization capacity is limited for understanding diverse linguistic phenomena (Naik et al., 2018; Glockner et al., 2018; McCoy et al., 2019; Rozen et al., 2019; Goodwin et al., 2020; Yanaka et al., 2020; Hu et al., 2021; Yanaka et al., 2021) and…”
BackgroundCan Transformers Reason in Fragments of Natural Language?
Schlegel, V., Pavlov, K. V., & Pratt-Hartmann, I. E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.768 · Cited by 18
Kovatchev, V., Chatterjee, T., Govindarajan, V. S., Chen, J., Choi, E., Chronis, G., Das, A., Erk, K., Lease, M., Li, J. J., et al. (2022) · doi:10.18653/v1/2022.dadc-1.5 · Cited by 11
Path-Based Contextualization of Knowledge Graphs for Textual EntailmentPreprint
Fadnis, K., Talamadupula, K., Kapanipathi, P., Ishfaq, H., Roukos, S., & Fokoue, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02085 · Cited by 10
“Datasets: We use multiple NLI datasets to evaluate our approach: SNLI [Bowman et al., 2015], SciTail [Khot et al., 2018], and BreakingNLI [Glockner et al., 2018].”
MethodLearning as Abduction: Trainable Natural Logic Theorem Prover for Natural Language InferencePreprint
Abzianidze, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15909 · Cited by 7
“NLI datasets, one might miss the systems’ inability of generalization (Glockner et al., 2018) or the exploitation of annotation artefacts (Poliak et al.”
BackgroundDuTrust: A Sentiment Analysis Dataset for Trustworthiness EvaluationPreprint
Wang, L., Liu, H., Peng, S., Tang, H., Xiao, X., Chen, Y., Wu, H. R., & Wang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13140 · Cited by 7
“…model brittleness (Ettinger et al. [2017], Nie et al. [2020]); and pattern-based methods exploit spurious patterns summarized from the dataset to generate adversarial examples and use them to evaluate models (Glockner et al. [2018], McCoy et al. [2019], Ribeiro et al. [2020], Gui et al. [2021]).”
MethodProceedings of the 5th Workshop on Argument Mining
(2018) · doi:10.18653/v1/w18-52 · Cited by 7
Zhang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03288 · Cited by 11
NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic
Zheng, Z., & Zhu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.554 · Cited by 9
Matina: A Large-Scale 73B Token Persian Text Corpus
Hosseinbeigi, S. B., Taherinezhad, F., Faili, H., Baghbani, H., Nadi, F., & Amiri, M. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.462 · Cited by 12
Teaching Syntax by Adversarial Distraction
Kim, J., Malon, C. D., & Kadav, A. (2018) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/w18-5512 · Cited by 8
“Our adversarial examples attack NLI systems from a new direction: not in their failure to capture the relations between words as in Glockner et al. (2018), but their failure to consider the syntax and word order in premises and hypotheses to decide upon the entailment relation.”
BackgroundTowards Verified Robustness under Text Deletion Interventions
Welbl, J., Huang, P.-S., Stanforth, R., Gowal, S., Dvijotham, K., Szummer, M., & Kohli, P. (2020) · International Conference on Learning Representations · Cited by 11
Bilinear Fusion of Commonsense Knowledge with Attention-Based NLI Models
Gajbhiye, A., Winterbottom, T. M., Moubayed, N. A., & Bradley, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61609-0_50 · Cited by 6
“atedTo christmas is useful to ascertain the inference relationship. Due to the lack of such common knowledge, state-of-the-art NLI models perform substantially worse for such premise-hypothesis pairs [9]. Incorporating external commonsense knowledge in deep neural NLI models is challenging. Existing models require con…”
BackgroundEntailment Relation Aware Paraphrase Generation
Sancheti, A., Srinivasan, B. V., & Rudinger, R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21376 · Cited by 10
Partial-input baselines show that NLI models can ignore context, but they don’t.
Srikanth, N. P., & Rudinger, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.350 · Cited by 16
Adversarial Analysis of Natural Language Inference Systems
Chien, T., & Kalita, J. K. (2020) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2020.00008 · Cited by 9
“In particular, the idea of adversarial examples or attacks was brought over from computer vision, and various methods of slightly perturbing inputs have been developed that cause models to fail catastrophically [1, 2, 3].”
BackgroundInfluentialDiscriminatively-Tuned Generative Classifiers for Robust Natural Language Inference
Ding, X., Liu, T., Chang, B., Sui, Z., & Gimpel, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.657 · Cited by 8
Mining Knowledge for Natural Language Inference from Wikipedia Categories
Chen, M., Chu, Z., Stratos, K., & Gimpel, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.313 · Cited by 6
The Extraordinary Failure of Complement Coercion Crowdsourcing
Elazar, Y., Basmov, V., Ravfogel, S., Goldberg, Y., & Tsarfaty, R. (2020) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2020.insights-1.17 · Cited by 6
Self-citationPosthoc Verification and the Fallibility of the Ground Truth
Ding, Y., Botzer, N., & Weninger, T. (2022) · doi:10.18653/v1/2022.dadc-1.3 · Cited by 8
Towards Improvement of Grounded Cross-lingual Natural Language Inference with VisioTextual Attention
De, A., Desarkar, M. S., & Ekbal, A. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100023 · Cited by 9
Posing Fair Generalization Tasks for Natural Language InferencePreprint
Geiger, A., Cases, I., Karttunen, L., & Potts, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00811 · Cited by 7
“, 2015a) have significant holes in their knowledge of lexical and compositional semantics (Glockner et al., 2018; Naik et al., 2018; Nie et al., 2018; Yanaka et al., 2019; Dasgupta et al., 2018).”
NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance
Talman, A., Apidianaki, M., Chatzikyriakidis, S., & Tiedemann, J. (2021) · Helda - University of Helsinki Institutional Repository · doi:10.48550/arxiv.2104.04751 · Cited by 12
“To the contrary, small tweaks or perturbations in the data, such as replacing words with mutually exclusive cohyponyms and antonyms (Glockner et al., 2018) or changing the order of the two sentences (Wang et al., 2019b), has been shown to hurt the performance of NLI models.”
BackgroundVAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLIPreprint
Kazoom, R., Cohen, O., Puzis, R., Shabtai, A., & Hadar, O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.00965 · Cited by 6
“Improving the robustness and performance of NLI models remains a significant challenge in natural language understanding (Glockner, Shwartz, and Goldberg 2018; Car-mona, Mitchell, and Riedel 2018).”
Improving the robustness of NLI models with minimax training
Korakakis, M., & Vlachos, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.801 · Cited by 15
“Consequently, NLI models that exploit shortcuts perform well on in-distribution samples, but are brittle when tested on out-of-distribution adversarial test sets that explicitly target such phenomena (Naik et al., 2018; Glockner et al., 2018).”
BackgroundHow Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?
Srikanth, N. P., Carpuat, M. J., & Rudinger, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00692 · Cited by 10
“In contrast with these studies, our goal is not to generate a test suite of difficult examples that “break” models (Glockner et al., 2018), but rather to carefully measure the role of paraphrastic variability in model performance.”
ResultAdversarial Natural Language Processing: Overview, Challenges and Future Directions
Shaw, L., Ansari, M. W., & Ekin, T. (2025) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2025.108 · Cited by 7
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning
Gupta, V., Zhang, S., Vempala, A., He, Y., Choji, T., & Srikumar, V. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.231 · Cited by 19
“Additionally, NLI models (e.g. Ribeiro et al., 2016, 2018a,b; Zhao et al., 2018; Iyyer et al., 2018; Glockner et al., 2018; Naik et al., 2018; McCoy et al., 2019; Nie et al., 2019; Liu et al., 2019a) must be subjected to numerous test sets with adversarial settings.”
BackgroundTemporal Natural Language Inference: Evidence-Based Evaluation of Temporal Text Validity
Hosokawa, T., Jatowt, A., & Sugiyama, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_28 · Cited by 11
DE-CO: A Two-Step Spelling Correction Model for Combating Adversarial Typos
Liu, Z., Wang, F., Lin, Z., Wang, L., & Yin, Z. (2020) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa-bdcloud-socialcom-sustaincom51426.2020.00095 · Cited by 5
“Reference [18] demonstrates an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym.”
BackgroundEffective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5
Bigoulaeva, I., Sachdeva, R. S., Madabushi, H. T., Villavicencio, A., & Gurevych, I. (2022) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2022.flp-1.8 · Cited by 9
Adversarial Examples for Evaluating Math Word Problem SolversPreprint
Kumar, V., Maheshwary, R., & Pudi, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05925 · Cited by 7
“Prior works in (Glockner et al., 2018; McCoy et al., 2019) uses adversarial examples to show deficien-cies of NLI models.”
MethodAutoCAD: Automatically Generating Counterfactuals for Mitigating Shortcut LearningPreprint
Wen, J., Zhu, Y., Zhang, J., Zhou, J., & Huang, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.16202 · Cited by 5
“• Stress (Naik et al., 2018) • Break (Glockner et al., 2018): The Breaking NLI dataset is an adversarial test set for evaluating the ability of lexical inferences.”
BackgroundSemantic Stealth: Adversarial Text Attacks on NLP Using Several MethodsPreprint
Dey, R., Debnath, A., Dutta, S. K., Ghosh, K., Mitra, A., Chowdhury, A. R., & Sen, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.05159 · Cited by 5
“Liang et al. (2017) [12] and Glockner et al. (2018) [13] played pivotal roles in advancing natural language processing (NLP) through their contributions to various tasks, including sentiment analysis, named entity recognition, and text summarization.”
BackgroundCurriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Chen, Z., & Gao, Q. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.234 · Cited by 4
LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI
Tarunesh, I., Aditya, S., & Choudhury, M. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09691-y · Cited by 5
“In temporal, models are sensitive to lexical substitutions (Glockner et al., 2018) such as replacing “before” (“after”) with “earlier than” (“later than”).”
Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task
Tikhonova, M., Mikhailov, V., Pisarevskaya, D., Malykh, V., & Shavrina, T. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000225 · Cited by 8
“Many works show that NLI models learn shallow heuristics and spurious correlations in the training data (Naik et al., 2018; Glockner et al., 2018; Sanchez et al., 2018), stimulating a targeted evaluation of LMs on out-of-distribution sets covering inference phenomena of interest (Yanaka et al.,…”
BackgroundInfluentialEnvironment Diversification with Multi-Head Neural Network for Invariant Learning
Huang, B., Liao, K.-T., Kao, C.-S., & Lin, S.-D. (2022) · neural information processing systems · doi:10.52202/068431-0067 · Cited by 6
Geiger, A., Richardson, K., & Potts, C. (2020) · arXiv (Cornell University) · Cited by 4
Android Security using NLP Techniques: A ReviewPreprint
Şen, S., & Can, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03072 · Cited by 4
“Adversarial examples on especially neural NLP models have been investigated in the last few years [Ebrahimi et al., 2018; Glockner et al., 2018; Pruthi et al., 2019].”
BackgroundLayne, J., Ratul, Q. E. A., Serra, E., & Jajodia, S. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3663481 · Cited by 5
“Further, though attempts are made to preserve part-of-speech and semantic similarity, altering single words can result in a loss of context and generate nonsensical sentences with respect to grammar.”
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets
Talman, A., Apidianaki, M., Chatzikyriakidis, S., & Tiedemann, J. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.20 · Cited by 4
Evaluation of Coreference Resolution Systems Under Adversarial Attacks
Chai, H., Zhao, W., Eger, S., & Strube, M. (2020) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2020.codi-1.16 · Cited by 3
“As opposed to evaluating on standard benchmarks, recent work (Glockner et al., 2018; Pruthi et al., 2019; Eger et al., 2019; Eger and Benz, 2020) investigates the generalization ability of NLP systems under adversarial attacks.”
BackgroundTabPert : An Effective Platform for Tabular Perturbation
Jain, N., Gupta, V., Rai, A., & Kumar, G. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.39 · Cited by 5
“, 2019), and (c) due to lexical overlap, neutrals (N) are closer to entailments (E) (Glockner et al., 2018).”
BackgroundExplainable lexical entailment with semantic graphs
Kovacs, A., Gémes, K., Kornai, A., & Recski, G. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000092 · Cited by 3
“Glockner, Shwartz, and Goldberg (2018) constructed a new NLI test set from SNLI by replacing a single word in sentences from the training set, and used this new benchmark to expose top NLI systems’ inability to perform true lexical inference.”
FANCY: A Diagnostic Data-Set for NLI Models
Rocchietti, G., Achena, F., Marziano, G., Salaris, S., & Lenci, A. (2022) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.10804 · Cited by 3
Logic-Based Inference With Phrase Abduction Using Vision-and-Language Models
Tomihari, A., & Yanaka, H. (2023) · IEEE Access · doi:10.1109/access.2023.3274489 · Cited by 3
“However, they suffer from low inter-pretability and explainability, and they have some limitations in what they can do, such as their generalization ability [10].”
BackgroundSimple but Challenging: Natural Language Inference Models Fail on Simple Sentences
Luo, C., Liu, W., Lin, J., Zou, J., Xiang, M., & Ding, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.252 · Cited by 8
“Regarding the event coreference bias, many studies have mentioned the event coreference problem in NLI tasks (Bowman et al., 2015; Williams et al., 2018; Glockner et al., 2018; Storks et al., 2019).”
BackgroundMonotonicity Reasoning in the Age of Neural Foundation Models
Chen, Z., & Gao, Q. (2023) · Journal of Logic Language and Information · doi:10.1007/s10849-023-09411-3 · Cited by 3
Do Language Models Know Who Did What to Whom?
Denning, J. M., Guo, X., Snefjella, B., & Blank, I. (2026) · Open Mind · doi:10.1162/opmi.a.365 · Cited by 4
“…when tested on corpora “from the wild” can break down when tested on carefully crafted stimuli, like those used in controlled psycholinguistic experiments conducted with humans (e.g., Chaves and Richter, 2021; Glockner et al., 2018; McCoy et al., 2019; Rosenman et al., 2020; Sinha et al., 2021).”
Kovatchev, V., Gold, D., Martí, M. A., Salamó, M., & Zesch, T. (2020) · Universitätsbibliographie, Universität Duisburg-Essen · Cited by 7
Improving Generalization by Incorporating Coverage in Natural Language InferencePreprint
Moosavi, N. S., Utama, P. A., Rücklé, A., & Gurevych, I. (2019) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.1909.08940 · Cited by 5
“– Glockner et al. (2018) is an adversarially constructed dataset, in which premises are taken from SNLI and the hypotheses differ from the premise only by one word.”
Geometry matters: Exploring language examples at the decision boundaryPreprint
Datta, D., Kumar, S., Barnes, L. E., & Fletcher, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07212 · Cited by 3
Probing Pre-Trained Language Models for Disease KnowledgePreprint
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07285 · Cited by 4
“One established approach is to create adversarial stress tests (Naik et al., 2018; Glockner et al., 2018; Aspillaga et al., 2020), in which synthetically generated examples are created to specifically test for phenomena that are known to confuse NLI models.”
BackgroundTeach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference
Rozen, O., Amar, S., Shwartz, V., & Dagan, I. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.8 · Cited by 2
“For example, Glockner et al. (2018) showed that substituting a single term in the premise with a similar but mutually-exclusive term (e.g. guitar and piano ) confused NLI models that predicted entailment.”
BackgroundSelf-citationContinuous Entailment Patterns for Lexical Inference in Context
Schmitt, M., & Schütze, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.556 · Cited by 2
“com/mnschmit/conan trained with lexical knowledge have been reported to struggle with this task (Glockner et al., 2018; Schmitt and Schütze, 2019), making LIiC an important evaluation criterion for general language understanding.”
BackgroundAnnotation Difficulties in Natural Language Inference
Kalouli, A.-L., Real, L., Buis, A., Palmer, M. S., & de Paiva, V. (2021) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2021.17804 · Cited by 2
“One strand focuses on detecting bias or artifacts in the training sets [Gururangan et al. 2018, Poliak et al. 2018] and creating challenging datasets that expose the generalization dif-ficulties of the models [Glockner et al. 2018, Nie et al. 2018, McCoy et al. 2019, inter alia].”
BackgroundResultA resampling-based method to evaluate NLI models
Salvatore, F. D. S., Finger, M., Hirata, R., & Patriota, A. G. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000268 · Cited by 5
“For example, Glockner et al. (2018) observe that a statistical model is capable of learning synonym and antonym inference when sufficient examples were added in training.”
InfluentialPetouo, F., Arafat, Y., Mushabbir, M. A., Hasan, K., & Mahmud, H. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10441070 · Cited by 2
Understanding generative AI to harness its potentials and mini- mize risks: A perspective
Caselli, T. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.111951 · Cited by 2
Composition of Sentence Embeddings: Lessons from Statistical Relational Learning
Sileo, D., de Cruys, T. V., Pradel, C., & Müller, P. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1004 · Cited by 6
Basile, A., & Caselli, T. (2019) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 2
Knowledge Adaptive Neural Network for Natural Language Inference
Zhang, Q., Yang, Y. T., Chen, C., He, L., & Yu, Z. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851884 · Cited by 3
“[9], that require rich semantic relation information such as synonyms and antonyms.”
BackgroundA guide to the dataset explosion in QA, NLI, and commonsense reasoning
Rogers, A., & Rumshisky, A. (2020) · doi:10.18653/v1/2020.coling-tutorials.5 · Cited by 4
“Furthermore, the models trained on such datasets are unaware of lexical knowledge that would have enabled them to solve simple WordNet-based permutations of the original data (Glockner et al., 2018).”
A Logic-Based Framework for Natural Language Inference in Dutch
Abzianidze, L., & Kogkalidis, K. (2021) · Computational Linguistics in the Netherlands Journal · doi:10.48550/arxiv.2110.03323 · Cited by 5
“Despite their undisputable performance, such models suffer from a variety of downsides, opaqueness and unpredictability being the most striking (Sanchez et al. 2018, Glockner et al. 2018, McCoy et al. 2019).”
BackgroundTextual Adversarial Attacking with Limited Queries
Zhang, Y., Yang, J., Li, X., Liu, H., & Shao, K. (2021) · Electronics · doi:10.3390/electronics10212671 · Cited by 2
“This may include selecting synonyms as substitutes [13,28,29], combinations [30,31], or sememe-based wordreplacements [11].”
BackgroundGenerating Content-Preserving and Semantics-Flipping Adversarial Text
Pei, W., & Yue, C. (2022) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3488932.3517397 · Cited by 2
A Report on the FigLang 2022 Shared Task on Understanding Figurative Language
Saakyan, A., Chakrabarty, T., Ghosh, D., & Muresan, S. (2022) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2022.flp-1.26 · Cited by 14
“These can allow large language models (LLMs) to achieve near human-level performance on in-domain test sets, yet turn brittle when evaluated against out-of-domain or adversarial examples (Glockner et al., 2018; Ribeiro et al., 2016, 2020).”
BackgroundRealistic Data Augmentation Framework for Enhancing Tabular Reasoning
Kumar, D., Gupta, V., Sharma, S., & Zhang, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.324 · Cited by 4
Understanding Roles and Entities: Datasets and Models for Natural Language InferencePreprint
Mitra, A., Shrivastava, I., & Baral, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09720 · Cited by 2
“Glockner et al. (2018) creates a NLI test set which shows the inability of the current state of the art systems to accurately perform inference requiring lexical and world knowledge.”
BackgroundContinuous Entailment Patterns for Lexical Inference in ContextPreprint
Schmitt, M., & Schütze, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03695 · Cited by 2
“com/mnschmit/conan trained with lexical knowledge have been reported to struggle with this task (Glockner et al., 2018; Schmitt and Schütze, 2019), making LIiC an important evaluation criterion for general language understanding.”
BackgroundMRCLens: an MRC Dataset Bias Detection ToolkitPreprint
Zhong, Y., Wang, H., & Xing, E. P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.08943 · Cited by 2
“Perturbing the original dataset and reevaluating models using the perturbed data is a method used frequently in various fields of NLP (Belinkov & Bisk, 2017; Carlini & Wagner, 2018; Glockner et al., 2018).”
MethodA Survey of Parameters Associated with the Quality of Benchmarks in NLPPreprint
Mishra, S. R., Arunkumar, A., Bryan, C., & Baral, C. R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07566 · Cited by 2
“…Distribution SNLI Summarization Compiled from (Gururangan et al., 2018) Feature Hypernyms and Hyponyms High Response SNLI Summarization Modified from (Glockner et al., 2018; Richardson and Sabharwal, 2019; Levy et al., 2015) Feature Modifiers and Superlatives Uniform Distribution SNLI…”
BackgroundMethodInfluentialDo Large Language Models know who did what to whom?Preprint
Denning, J. M., Guo, X., Snefjella, B., & Blank, I. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/z56wj · Cited by 2
Using Clustering to Improve the Performance of few-shot Learning
Zhang, Y., Wu, C., Shi, R., & Zhang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447442 · Cited by 1
“• SNLI [16]: SNLI task is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contra-diction, and neutral.”
BackgroundDikshit, S., Dixit, R., Tiwari, R., & Jain, P. (2025) · SN Computer Science · doi:10.1007/s42979-025-04146-3 · Cited by 1
Therapist Self-Disclosure as a Natural Language Processing Task
Shapira, N., & Alfi‐Yogev, T. (2024) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/2024.clpsych-1.5 · Cited by 8
“…marking potential shallow heuristics, such as syntactic features, that a machine learning model might exploit to predict the correct label for the incorrect reasons (see shallow heuristics: Hendrycks et al., 2021; Wu et al., 2021; Kaushik et al., 2019; Geirhos et al., 2020; Glockner et al., 2018).”
BackgroundTextual Paraphrase Dataset for Deep Language Modelling
Kanerva, J., Ginter, F., Chang, L., Skantsi, V., Kilpeläinen, J., Kupari, H.-M., Piirto, A., Saarni, J., Sevón, M., & Tarkka, O. (2022) · Cognitive technologies · doi:10.1007/978-3-031-17258-8_27 · Cited by 1
Systematic Monotonicity and Consistency for Adversarial Natural Language Inference
Nutakki, B., Badola, A., & Padmanabhan, V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22695-3_25 · Cited by 1
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274 · Cited by 7
Metadata
- Authors
- Glockner, M., Shwartz, V., & Goldberg, Y.
- Affiliations
- Technische Universität Darmstadt (DE)
- Bar-Ilan University (IL)
- Corresponding author
- Goldberg, Yoav
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 650-655
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2798665661
- Cited by
- 319
- References
- 20
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 13 · 2023: 26 · 2022: 48 · 2021: 65 · 2020: 69 · 2019: 72 · 2018: 14
- FWCI
- 22.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (DA 1600/1-1)
- Israel Science Foundation
- Intel Collaboration Research Institute for Computational Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- natural language inference, lexical inference, world knowledge, SNLI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.