Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set.Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“61 (Landis and Koch, 1977).”
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“All models are based on pre-trained GloVe embeddings (Pennington et al., 2014), which are either fine-tuned during training (RESIDUAL-STACKED-ENCODER and ESIM) or stay fixed (DECOMPOSABLE ATTENTION).”
MethodInfluentialA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 6×
“The SNLI dataset (Stanford Natural Language Inference, Bowman et al., 2015) consists of 570k sentence-pairs manually labeled as entailment, contradiction, and neutral.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 3×
“The MultiNLI dataset (Multi-Genre Natural Language Inference, Williams et al., 2018) was collected similarly to SNLI, though covering a wider range of genres, and supporting a cross-genre evaluation.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426 · cited 1×
“Premises are image captions from Young et al. (2014), while hypotheses were generated by crowd-sourced workers who were shown a premise and asked to generate entailing, contradicting, and neutral sentences.”
BackgroundAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“Inspired by the work of Jia and Liang (2017) on reading comprehension, we create a new NLI test set with examples that capture various kinds of lexical knowledge (Table 1).”
MethodA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 2×
“Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
BackgroundMethodEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”
BackgroundMethodAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“Our results are in line with Gururangan et al. (2018) and Po-liak et al. (2018), who showed that the label can be identified by looking only at the hypothesis and exploiting annotation artifacts such as word choice and sentence length.”
ResultReasoning about Entailment with Neural Attention
Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582 · cited 1×
“…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
Research works printed but not linked: 2
- Christiane Fellbaum. 1998. WordNet . Wiley Online Library.
- Yixin Nie and Mohit Bansal. 2017. Shortcutstacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . · arXiv
Cited by
Items 101–200 of 350
Posing Fair Generalization Tasks for Natural Language Inference
Geiger, A., Cases, I., Karttunen, L., & Potts, C. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1456 · Cited by 35
“, 2015a) have significant holes in their knowledge of lexical and compositional semantics (Glockner et al., 2018; Naik et al., 2018; Nie et al., 2018; Yanaka et al., 2019; Dasgupta et al., 2018).”
DRr-Net: Dynamic Re-Read Network for Sentence Semantic Matching
Zhang, K., Lv, G., Wang, L., Wu, L., Chen, E., Wu, F., & Xie, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017442 · Cited by 42
“…of sentence understanding, we also select the challenging hard subset from (Gururan-gan et al. 2018), in which the premise-oblivious model cannot classify accurately, and lexical subset from (Glockner, Shwartz, and Goldberg 2018), which requires lexical and world knowledge, as our SNLI test sets.”
BackgroundWhat Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 48
“This shows the high capacity for transformer models to absorb new tasks with minimal costs, as also observed in Richardson et al. (2020) for NLI.”
BackgroundInfluentialANLIzing the Adversarial Natural Language Inference DatasetPreprint
Williams, A., Thrush, T., & Kiela, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12729 · Cited by 50
“…existing NLI datasets, primarily focused on annotation artifacts (Gururangan et al., 2018; Geiger et al., 2018; Poliak et al., 2018b; Tsuchiya, 2018; Glockner et al., 2018a; Geva et al., 2019) and diagnostic datasets (McCoy et al., 2019; Naik et al., 2018; Nie et al., 2019; Yanaka et al., 2019;…”
BackgroundMethodOpen-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38
“Good progress is being made on these tasks, although questions still remain about how much true generalization is actually occurring (Glockner, Shwartz, and Goldberg, 2018; Gururangan et al., 2018).”
BackgroundDo Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
Yanaka, H., Mineshima, K., Bekki, D., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.543 · Cited by 63
DQI: Measuring Data Quality in NLPPreprint
Mishra, S. R., Arunkumar, A., Sachdeva, B., Bryan, C., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00816 · Cited by 50
“(Glockner et al., 2018) generate entailment samples by replacing words with their synonyms, hyponyms and hypernyms.”
BackgroundInfluentialLinear Connectivity Reveals Generalization StrategiesPreprint
Juneja, J., Bansal, R., Cho, K., Sedoc, J., & Saphra, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12411 · Cited by 47
“Diagnostic sets in NLP can be based on natural adversarial data (Gulordava et al., 2018; Linzen et al., 2016), constructed by perturbations and rephrases (Sanchez et al., 2018; Glockner et al., 2018), or generated artificially from templates (Rudinger et al., 2018; Zhao et al., 2018).”
BackgroundShortcut Learning Explanations for Deep Natural Language Processing: A Survey on Dataset Biases
Dogra, V., Verma, S., Kavita, K., Woźniak, M., Shafi, J., & Ijaz, M. F. (2024) · IEEE Access · doi:10.1109/access.2024.3360306 · Cited by 46
“The modern NLI systems have a limited capacity for generalization and frequently miss out on simple inferences that require lexical and contextual knowledge [44].”
BackgroundLanguage (Re)modelling: Towards Embodied Language Understanding
Tamari, R., Shani, C., Hope, T., Petruck, M. R. L., Abend, O., & Shahaf, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.559 · Cited by 45
Figurative Language in Recognizing Textual Entailment
Chakrabarty, T., Ghosh, D., Poliak, A., & Muresan, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.297 · Cited by 54
Evaluating Gender Bias in Natural Language InferencePreprint
Sharma, S., Dey, M., & Sinha, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05541 · Cited by 46
Explaining Deep Neural NetworksPreprint
Camburu, O.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01496 · Cited by 49
“Such assumptions may be poor since recent works show that neural networks can learn to heavily rely on surprising spurious correlations even when they provide correct answers (Gururangan et al., 2018; McCoy et al., 2019; Glockner et al., 2018a).”
BackgroundMethodInfluentialDiversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets
Rozen, O., Shwartz, V., Aharoni, R., & Dagan, I. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1019 · Cited by 41
“that neural models often fail on particular inference types, like recognizing semantic relations and negation (Poliak et al., 2018; Naik et al., 2018; Glockner et al., 2018).”
BackgroundSelf-citationA Logic-Driven Framework for Consistency of Neural ModelsPreprint
Li, T., Gupta, V., Mehta, M., & Srikumar, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00126 · Cited by 46
Misleading Failures of Partial-input Baselines
Feng, S., Wallace, E., & Boyd-Graber, J. L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1554 · Cited by 55
“, 2019) and challenge sets (Glockner et al., 2018; Naik et al., 2018).”
ConjNLI: Natural Language Inference Over Conjunctive Sentences
Saha, S., Nie, Y., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.661 · Cited by 50
T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step
Chen, Z., Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.515 · Cited by 32
“Many benchmarks have been established to evaluate base abilities on question-answering tasks (Rajpurkar et al., 2016; Clark et al., 2018; Glockner et al., 2018), natural language understanding tasks (Wang et al., 2018, 2019; Hendrycks et al., 2020), and common-sense reasoning tasks (Lu et al.,…”
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.659 · Cited by 32
Text Adversarial Attacks and Defenses: Issues, Taxonomy, and Perspectives
Han, X., Zhang, Y., Wang, W., & Wang, B. (2022) · Security and Communication Networks · doi:10.1155/2022/6458488 · Cited by 41
“[110] argued that mods lack lexical and world knowledge, and they used a synonym, hypernym, co-hyponyms, and antonyms to generate a new NLI test set without introducing new vocabulary.”
BackgroundShifting the Baseline: Single Modality Performance on Visual Navigation & QA
Thomason, J., Gordon, D., & Bisk, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00613 · Cited by 20
“This work directly extends observations made within both the Computer Vision (Goyal et al., 2018) and Natural Language (Glockner et al., 2018; Poliak et al., 2018; Gururangan et al., 2018; Kaushik and Lipton, 2018) communities that sophisticated models often perform well by fitting to spurious correlations in the data,…”
BackgroundInfluentialNoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Ravichander, A., Dalmia, S., Ryskina, M., Metze, F., Hovy, E. H., & Black, A. W. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.259 · Cited by 49
IndoNLI: A Natural Language Inference Dataset for Indonesian
Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.821 · Cited by 65
“…from various aspects such as antonymy, numerical reasoning, word overlap, negation, length mismatch, and spelling error (Naik et al., 2018), lexical overlap, subsequence and constituent (McCoy et al., 2019), lexical inferences (Glockner et al., 2018) and syntactic structure (Po-liak et al., 2018a).”
BackgroundTextual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning
Sainz, O., González-Dios, I., de Lacalle, O. L., Min, B., & Agirre, E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.187 · Cited by 57
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Dixit, T., Paranjape, B., Hajishirzi, H., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.216 · Cited by 58
Mining Discourse Markers for Unsupervised Sentence Representation LearningPreprint
Sileo, D., de Cruys, T. V., Pradel, C., & Müller, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.11850 · Cited by 32
“Inspired by recent work (Dasgupta et al., 2018; Poliak et al., 2018; Levy et al., 2018; Glockner et al., 2018) on the unexpected properties of recent manually labelled datasets (e.”
BackgroundMethodAchieving Verified Robustness to Symbol Substitutions via Interval Bound PropagationPreprint
Huang, P.-S., Stanforth, R., Welbl, J., Dyer, C., Yogatama, D., Gowal, S., Dvijotham, K., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01492 · Cited by 36
“It is worth noting that symbol substitution is general enough to encompass other threat models such as lexical entailment perturbations (Glockner et al., 2018), and could potentially be extended to the addition of pre/postfixes (Jia and Liang, 2017; Wallace et al., 2019).”
BackgroundBenchmarking Robustness of Machine Reading Comprehension Models
Si, C., Yang, Z., Cui, Y., Ma, W., Liu, T., & Wang, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.56 · Cited by 30
Models in the Wild: On Corruption Robustness of Neural NLP Systems
Rychalska, B., Basaj, D., Gosiewska, A., & Biecek, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36718-3_20 · Cited by 50
Robustness to Modification with Shared Words in Paraphrase Identification
Shi, Z., & Huang, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.16 · Cited by 28
Metaethical perspectives on ‘benchmarking’ AI ethics
LaCroix, T., & Luccioni, A. S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00703-x · Cited by 37
“, 2019), logical reasoning (Glockner et al., 2018), or even simple negation (Hossain et al.”
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 34
Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging
Lee, N., Wu, C.-S., & Fung, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1143 · Cited by 25
“Recognizing textual entailment that aims to determine whether a hypothesis h can justifiably be inferred from a premise (Dang et al., 2007; Bowman et al., 2015; Parikh et al., 2016; Chen et al., 2017b; Glockner et al., 2018).”
BackgroundMethodTextFirewall: Omni-Defending Against Adversarial Texts in Sentiment Classification
Wang, W., Wang, R., Ke, J., & Wang, L. (2021) · IEEE Access · doi:10.1109/access.2021.3058278 · Cited by 28
“The sentence-level adversarial attack usually appeared in other NLP tasks like natural language inference [29], neural machine translation [30], reading comprehension [31], and question answering [32].”
BackgroundAnalyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 41
Han, X., & Tsvetkov, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.374 · Cited by 38
Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified SentencesPreprint
Geiger, A., Cases, I., Karttunen, L., & Potts, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13033 · Cited by 29
“However, the corpora themselves have been scrutinized and found to contain patterns that lower our confidence that the models are learning robust semantic representations (Rudinger et al., 2017; Poliak et al., 2018; Glockner et al., 2018; Gururangan et al., 2018; Tsuchiya, 2018).”
BackgroundHy-NLI: a Hybrid system for Natural Language Inference
Kalouli, A.-L., Crouch, R., & de Paiva, V. C. V. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.459 · Cited by 34
“…Poliak et al., 2018) and “breaking” the models with adversarial datasets that expose the generalization and compositionality difficulties of the models (Glockner et al., 2018; Nie et al., 2018; Zhu et al., 2018; Dasgupta et al., 2018; Naik et al., 2018; McCoy et al., 2019; Richardson et al., 2020).”
BackgroundExposing the Limits of Video-Text Models through Contrast Sets
Park, J. S., Shen, S., Farhadi, A., Darrell, T., Choi, Y., & Rohrbach, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.261 · Cited by 38
“Template-based (McCoy et al., 2019; Glockner et al., 2018) and manually crafted (Gardner et al.”
MethodNeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation
Howard, P., Singer, G., Lal, V., Choi, Y., & Swayamdipta, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.371 · Cited by 33
Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference TaskPreprint
Tarunesh, I., Aditya, S., & Choudhury, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07229 · Cited by 30
“Several work (McCoy et al., 2019; Kaushik et al., 2020; Glockner et al., 2018) developed targeted evaluation sets to adversarially challenge these large PTLMs and demonstrated shortcomings.”
BackgroundDeep Learning for Natural Language Inference
Bowman, S. R., & Zhu, X. (2019) · doi:10.18653/v1/n19-5002 · Cited by 23
“We will present how NLI models are sensitive or robust to different newly proposed tests (Glockner et al., 2018; Wang et al., 2018; Naik et al., 2018; Poliak et al., 2018).”
BackgroundAvoiding Inference Heuristics in Few-shot Prompt-based Finetuning
Utama, P. A., Moosavi, N. S., Sanh, V., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.713 · Cited by 35
“Heuristics in models are often investigated using constructed “challenge datasets” consisting of counter-examples to the spurious cues, which mostly result in incorrect predictions (Jia and Liang, 2017; Glockner et al., 2018; Naik et al., 2018; McCoy et al., 2019).”
BackgroundOn the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 35
Frame- and Entity-Based Knowledge for Common-Sense Argumentative Reasoning
Botschen, T., Sorokin, D., & Gurevych, I. (2018) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/w18-5211 · Cited by 23
“Glockner et al. (2018) point out the deficiency of state-of-the-art approaches for understanding entailment on the large-scale SNLI corpus (Stanford Natural Language Inference) (Bowman et al., 2015).”
BackgroundGokhale, T., Mishra, S. R., Luo, M., Sachdeva, B., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.213 · Cited by 34
Does Data Augmentation Improve Generalization in NLP?Preprint
Jha, R., Lovering, C., & Pavlick, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.15012 · Cited by 25
“A wave of recent work has adopted a strategy of constructing evaluation sets composed of “adver-sarial examples” (a.k.a. “challenge examples” or “probing sets”) in order to analyze and expose weaknesses in the decision procedures learned by neural NLP models (Jia and Liang, 2017; Glockner et al., 2018; Dasgupta et al.…”
BackgroundUnderstanding Model Robustness to User-generated Noisy Texts
Náplava, J., Popel, M., Straka, M., & Straková, J. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.38 · Cited by 31
A logical-based corpus for cross-lingual evaluation
Salvatore, F. D. S., Finger, M., & Hirata, R. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6103 · Cited by 25
“In an experiment, we verify the deficiency reported in Gururangan et al. (2018); Glockner et al. (2018).”
BackgroundMethodExplaining Simple Natural Language Inference
Kalouli, A.-L., Buis, A., Real, L., Palmer, M. S., & de Paiva, V. C. V. (2019) · Linguistic Annotation Workshop · doi:10.18653/v1/w19-4016 · Cited by 24
“However, it was recently shown that this high performance can drop significantly by slightly modifying the training process (Poliak et al., 2017; Glockner et al., 2018).”
BackgroundLi, T., Zhu, X., Liu, Q., Chen, Q., Chen, Z., & Wei, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12104 · Cited by 21
“The Glockner and Swapping Test (Glockner et al., 2018) take the premises from SNLI and uses lexical replacement to generate hypotheses for different NLI categories (i.e., entailment, contradiction, and neutral) by re-placing a single word in the premise and asking human subjects to confirm the…”
BackgroundMethodInfluentialSciCo: Hierarchical Cross-Document Coreference for Scientific ConceptsPreprint
Cattan, A., Johnson, S., Weld, D. S., Dagan, I., Beltagy, I., Downey, D., & Hope, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08809 · Cited by 24
“Hierarchy Model We consider a referential hierarchical relation between concepts x and y as an entailment relation between y and x [Glockner et al., 2018].”
BackgroundMethodImage-Enhanced Multi-level Sentence Representation Net for Natural Language Inference
Zhang, K., Lv, G., Wu, L., Chen, E., Liu, Q., Wu, H., & Wu, F. (2018) · Proceedings · doi:10.1109/icdm.2018.00090 · Cited by 23
“[39] proposed a simple but challenging lexical test set.”
BackgroundInfluentialSemantic Representations for NLP Using VerbNet and the Generative Lexicon
Brown, S. W., Bonn, J., Kazeminejad, G., Zaenen, A., Pustejovsky, J. D., & Palmer, M. S. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.821697 · Cited by 31
“Once the data sets are corrected/expanded to include more representative language patterns, performance by these systems plummets (Glockner et al., 2018; Gururangan et al., 2018; McCoy et al., 2019).”
BackgroundAn Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference
Liu, T., Xin, Z., Ding, X., Chang, B., & Sui, Z. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.48 · Cited by 29
Knowledge-Enhanced Natural Language Inference Based on Knowledge Graphs
Wang, Z., Li, L., & Zeng, D. D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.571 · Cited by 19
“• BNLI is a dataset constructed based on SNLI (Glockner et al., 2018).”
BackgroundMethodInfluentialMitigating the Position Bias of Transformer Models in Passage Re-ranking
Hofstätter, S., Lipani, A., Althammer, S., Zlabinger, M., & Hanbury, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_16 · Cited by 28
Counterfactual can be strong in medical question and answering
Yang, Z., Liu, Y., Ouyang, C., Ren, L., & Wen, W. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103408 · Cited by 23
Evaluating Deep Learning Techniques for Natural Language Inference
Eleftheriadis, P., Perikos, I., & Hatzilygeroudis, I. (2023) · Applied Sciences · doi:10.3390/app13042577 · Cited by 18
CREST: A Joint Framework for Rationalization and Counterfactual Text Generation
Treviso, M., Ross, A., Guerreiro, N. M., & Martins, A. F. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.842 · Cited by 40
Sevastjanova, R., Kalouli, A.-L., Beck, C., Hauptmann, H., & El‐Assady, M. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14541 · Cited by 26
Natural Language Inference with Hierarchical BiLSTM Max Pooling Architecture.Preprint
Talman, A., Yli-Jyrä, A., & Tiedemann, J. (2018) · arXiv (Cornell University) · Cited by 18
Schmitt, M., & Schütze, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1086 · Cited by 22
“Glockner et al. (2018) investigated LI as a way of evaluating NLI systems and found that even simple cases are challenging to current systems.”
BackgroundMethodInfluentialAlignment Rationale for Natural Language Inference
Jiang, Z., Zhang, Y., Zhao, Y., Zhao, J., & Liu, K. (2021) · doi:10.18653/v1/2021.acl-long.417 · Cited by 23
“We use four datasets SNLI (Bowman et al., 2015), E SNLI (Camburu et al., 2018), BNLI (Glockner et al., 2018) and HANS as our testbeds.”
MethodVisual Comparison of Language Model Adaptation
Sevastjanova, R., Cakmak, E., Ravfogel, S., Cotterell, R., & El‐Assady, M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209458 · Cited by 27
“Probing classifiers [12,29,37] and adversarial testing [20,41,58] are the most common methods used in computational linguistics to understand such properties.”
MethodReaSCAN: Compositional Reasoning in Language GroundingPreprint
Wu, Z., Kreiss, E., Ong, D. C., & Potts, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08994 · Cited by 21
“However, the major benchmarks in the field right now mostly do not support rich grounding, and it is often unclear whether they support learning compositional structures, as evidenced by their common failures at simple adversarial tests involving compositionality [7, 8, 9].”
BackgroundProceedings of the 13th Linguistic Annotation Workshop
Friedrich, A., Zeyrek, D., & Hoek, J. (2019) · doi:10.18653/v1/w19-40 · Cited by 17
Gupta, V., Bhat, R. A., Ghosal, A., Shrivastava, M., Singh, M., & Srikumar, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00482 · Cited by 22
“Investigation is needed around how the model arrives at a particular prediction: is it looking at the correct evidence? Furthermore, NLI models should be subjected to multiple test sets with adversarial settings (e.g. Ribeiro et al., 2016, 2018a,b; Zhao et al., 2018; Iyyer et al., 2018; Glockner et al., 2018; Naik et…”
BackgroundOn the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Mussmann, S., Jia, R., & Liang, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.305 · Cited by 20
ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation
Stojanovski, D., Krojer, B., Peskov, D., & Fraser, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.417 · Cited by 18
“Our work is related to adversarial datasets for testing robustness used in Natural Language Processing tasks such as studying gender bias (Zhao et al., 2018; Rudinger et al., 2018; Stanovsky et al., 2019), natural language inference (Glockner et al., 2018) and classification (Wang et al., 2019).”
MethodExplaining Contextualization in Language Models using Visual Analytics
Sevastjanova, R., Kalouli, A.-L., Beck, C., Schäfer, H., & El‐Assady, M. (2021) · doi:10.18653/v1/2021.acl-long.39 · Cited by 27
“Adversarial Testing – Adversarial testing has shown that LMs struggle in making generalizations on basic lexical relations (Glockner et al., 2018), identifying ungrammaticality (Marvin and Linzen, 2018), efficiently capturing challenging linguistic phenomena, such as negation (Dasgupta et al., 2018;…”
BackgroundAdversarial Examples for Evaluating Math Word Problem Solvers
Kumar, V., Maheshwary, R., & Pudi, V. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.230 · Cited by 29
Compositional Evaluation on Japanese Textual Entailment and Similarity
Yanaka, H., & Mineshima, K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00518 · Cited by 41
“Since previous work has demonstrated the usefulness of analyzing the generalization ability of models in challenging NLI in English (Naik et al., 2018; Glockner et al., 2018; McCoy et al., 2019; Rozen et al., 2019; Goodwin et al., 2020; Yanaka et al., 2021), we should continue this line of research in other languages.”
BackgroundSemantically Distributed Robust Optimization for Vision-and-Language Inference
Gokhale, T., Chaudhary, A., Banerjee, P., Baral, C. R., & Yang, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.118 · Cited by 16
Building human-like communicative intelligence: A grounded perspective
Dubova, M. (2021) · Cognitive Systems Research · doi:10.1016/j.cogsys.2021.12.002 · Cited by 21
Larson, S., Zheng, A., Mahendran, A., Tekriwal, R., Cheung, A., Guldan, E., Leach, K., & Kummerfeld, J. K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.650 · Cited by 28
“, 2019), as can one-word changes to sentences that require world knowledge (Glockner et al., 2018), and changing syntax can confuse pretrained models (Iyyer et al.”
BackgroundImproving Robustness by Augmenting Training Sentences with Predicate-Argument StructuresPreprint
Moosavi, N. S., de Boer, M., Utama, P. A., & Gurevych, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12510 · Cited by 16
“This adversarial setting is also common in NLI, e.g., (Naik et al., 2018; Glockner et al., 2018).”
BackgroundTemporal validity reassessment: commonsense reasoning about information obsoleteness
Hosokawa, T., Jatowt, A., & Sugiyama, K. (2024) · Discover Computing · doi:10.1007/s10791-024-09433-w · Cited by 17
“After that, many NLI datasets [49, 50], including Multi-genre Natural Language Inference (MNLI) [51] and Scitail [52], have been constructed.”
A Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 15
Exploring Transitivity in Neural NLI Models through Veridicality
Yanaka, H., Mineshima, K., & Inui, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.78 · Cited by 22
Feature-Level Debiased Natural Language Understanding
Lyu, Y., Li, P., Yang, Y., de Rijke, M., Ren, P., Zhao, Y., Yin, D., & Ren, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26567 · Cited by 29
Knowledge Enhanced Attention for Robust Natural Language InferencePreprint
Li, A. H., & Sethy, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00102 · Cited by 17
“In fact, the 94.1% accuracy equals to the estimated human performance reported in Glockner et al. (2018).”
BackgroundMethodInfluentialZhu, Y., Si, J., Zhao, Y., Zhu, H., Zhou, D., & He, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.826 · Cited by 26
KETM:A Knowledge-Enhanced Text Matching method
Jiang, K., Zhao, Y., Jin, G., Zhang, Z., & Cui, R. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191337 · Cited by 12
“To verify the knowledge of the adversarial dataset, we train on the Snli training set and test on the adversarial dataset BreakNLI dataset[33].”
MethodDeep Neural Model Inspection and Comparison via Functional Neuron Pathways
Fiacco, J., Choudhary, S., & Rosé, C. P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1575 · Cited by 14
“The specific goal of our evaluation will be to demonstrate the broad applicability of the approach, and position it as building on and extending the existing body of work exploring interpretability of previously defined neural models (Glockner et al., 2018; Mudrakarta et al., 2018).”
BackgroundResultAnalyzing Machine‐Learned Representations: A Natural Language Case Study
Dasgupta, I., Guo, D., Gershman, S. J., & Goodman, N. D. (2020) · Cognitive Science · doi:10.1111/cogs.12925 · Cited by 21
Li, T., Rashid, A., Jafari, A., Sharma, P., Ghodsi, A., & Rezagholizadeh, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.65 · Cited by 17
Enhancing Unsupervised Pretraining with External Knowledge for Natural Language Inference
Yang, X., Zhu, X., Zhao, H., Zhang, Q., & Feng, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18305-9_38 · Cited by 11
Hu, H., Zhou, H., Tian, Z., Zhang, Y., Patterson, Y., Li, Y., Nie, Y., & Richardson, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.331 · Cited by 15
Robustness and Sensitivity of BERT Models Predicting Alzheimer’s Disease from Text
Novikova, J. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.37 · Cited by 15
Evaluating Natural Language Inference Models: A Metamorphic Testing Approach
Jiang, M., Bao, H., Tu, K., Zhang, X., & Ding, Z. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00033 · Cited by 13
Chinese adversarial examples generation approach with multi-strategy based on semantic
Ou, H., Yu, L., Tian, S., & Chen, X. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01652-1 · Cited by 11
“Then Jia, Belinkov, Glockner et al. [6–8] successively proved the weakness of Natural Language Processing (NLP) models that can be easily deceived by adversarial examples by adding character-level, word-level, and sentence-level perturbations.”
BackgroundMaharana, A., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.333 · Cited by 14
SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics
Yanaka, H., Mineshima, K., & Inui, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.10 · Cited by 17
AutoCAD: Automatically Generate Counterfactuals for Mitigating Shortcut Learning
Wen, J., Zhu, Y., Zhang, J., Zhou, J., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.170 · Cited by 26
Emelin, D., Titov, I., & Sennrich, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.616 · Cited by 36
Back to Square One: Bias Detection, Training and Commonsense Disentanglement in the Winograd SchemaPreprint
Elazar, Y., Zhang, H., Goldberg, Y., & Roth, D. (2021) · arXiv (Cornell University) · Cited by 10
Self-citationLanguage Models for Lexical Inference in Context
Schmitt, M., & Schütze, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.108 · Cited by 9
Metadata
- Authors
- Glockner, M., Shwartz, V., & Goldberg, Y.
- Affiliations
- Technische Universität Darmstadt (DE)
- Bar-Ilan University (IL)
- Corresponding author
- Goldberg, Yoav
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 650-655
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2798665661
- Cited by
- 319
- References
- 20
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 13 · 2023: 26 · 2022: 48 · 2021: 65 · 2020: 69 · 2019: 72 · 2018: 14
- FWCI
- 22.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (DA 1600/1-1)
- Israel Science Foundation
- Intel Collaboration Research Institute for Computational Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- natural language inference, lexical inference, world knowledge, SNLI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.