Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set.Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“61 (Landis and Koch, 1977).”
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“All models are based on pre-trained GloVe embeddings (Pennington et al., 2014), which are either fine-tuned during training (RESIDUAL-STACKED-ENCODER and ESIM) or stay fixed (DECOMPOSABLE ATTENTION).”
MethodInfluentialA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 6×
“The SNLI dataset (Stanford Natural Language Inference, Bowman et al., 2015) consists of 570k sentence-pairs manually labeled as entailment, contradiction, and neutral.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 3×
“The MultiNLI dataset (Multi-Genre Natural Language Inference, Williams et al., 2018) was collected similarly to SNLI, though covering a wider range of genres, and supporting a cross-genre evaluation.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426 · cited 1×
“Premises are image captions from Young et al. (2014), while hypotheses were generated by crowd-sourced workers who were shown a premise and asked to generate entailing, contradicting, and neutral sentences.”
BackgroundAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“Inspired by the work of Jia and Liang (2017) on reading comprehension, we create a new NLI test set with examples that capture various kinds of lexical knowledge (Table 1).”
MethodA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 2×
“Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
BackgroundMethodEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”
BackgroundMethodAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“Our results are in line with Gururangan et al. (2018) and Po-liak et al. (2018), who showed that the label can be identified by looking only at the hypothesis and exploiting annotation artifacts such as word choice and sentence length.”
ResultReasoning about Entailment with Neural Attention
Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582 · cited 1×
“…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
Research works printed but not linked: 2
- Christiane Fellbaum. 1998. WordNet . Wiley Online Library.
- Yixin Nie and Mohit Bansal. 2017. Shortcutstacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . · arXiv
Cited by
Items 301–350 of 350
SETI: Systematicity Evaluation of Textual Inference
Fu, X., & Frank, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.252 · Cited by 12
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 3
A Simple Method to Improve the Performance of Small Pre-trained Language Models on Few-shot Tasks
Zhang, Y., Wu, C., Shi, R., & Zhang, Y. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580281 · Cited by 1
“These tasks include SST-2 [20], MR [21], CR, Subj [22], TREC [23], MNLI [24], SNLI [25], QNLI [26], MRPC [27], QQP [28].”
MethodEBFP: Example-Based Further Pre-training
Zhang, Y., Wu, C., Zhao, W., & Zhang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5669-8_35 · Cited by 1
Mixture of Prompt Experts for Natural Language Inference
Zheng, Z., & Zhu, X. (2024) · Conference proceedings - Canadian Conference on Electrical and Computer Engineering · doi:10.1109/ccece59415.2024.10667143 · Cited by 1
“However, more recent research [14], [15], [19]–[25] highlighted the limitations of models trained or fine-tuned on the general large-scaled NLI training sets when confronted with diverse types of linguistic phenomena like negation, quantification and monotonicity, which demonstrates the lack of the…”
BackgroundKim, S., Jeong, S., & Kim, H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112608 · Cited by 1
External Knowledge-Driven Argument Mining: Leveraging Attention-Enhanced Multi-Network Models
Gemechu, D. T., & Reed, C. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.216 · Cited by 1
Analyzing natural language inference from a rigorous point of view
Salvatore, F. D. S. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2020.tde-05012021-151600 · Cited by 1
Unsupervised Pre-training with Structured Knowledge for Improving Natural Language InferencePreprint
Yang, X., Zhu, X., Shi, Z., & Li, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03941 · Cited by 1
“Also, new test datasets are proposed, e.g., Glockner test set (Glockner et al., 2018).”
BackgroundMethodInfluentialParaphrasing, textual entailment, and semantic similarity above word levelPreprint
Kovatchev, V. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05387 · Cited by 2
Evaluating Text Classification Robustness to Part-of-Speech Adversarial ExamplesPreprint
Samadi, A., & Sullivan, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.08374 · Cited by 1
Testing the Reasoning Power for NLI Models with Annotated Multi-perspective Entailment Dataset
Yu, D., Liu, L., Chen, Y., & Li, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_2
Defences and threats in safe deep learning
Chan, A. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/152976
Towards Evaluating Veracity of Textual Statements on the Web
Zhang, Q. (2021) · UCL Discovery (University College London)
Machine reading between the lines (RBL) of medical complaints
Galitsky, B. A. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-824521-7.00014-4
Lu, R., HangCheng, Z., Meng, R., Zhao, J., JiaoYu, Q., Feng, W., & ChenZi, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13140
“[23] Glockner M, Shwartz V, Goldberg Y.”
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?
Gabriel, S., Palangi, H., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.413 · Cited by 1
Detecting Contradiction and Entailment in Multilingual Text
Verma, A., Srivastav, J., Gera, P., & Mohapatra, A. K. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10307714
Kalouli, A.-L., Abzianidze, L., & Chatzikyriakidis, S. (2023) · Journal of Logic Language and Information · doi:10.1007/s10849-023-09408-y
FiRo: Finite-context Indexing of Restricted Output Space for NLP Models Facing Noisy Input
Nguyen, M., & Chen, N. F. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.54
Semantic Stealth: Adversarial Text Attacks on NLP using Several MethodsPreprint
Sen, J., Dey, R., Debnath, A., Dutta, S. K., Ghosh, K., Mitra, A., & Roychowdhury, A. (2024) · doi:10.36227/techrxiv.171392778.89337138/v1
Refining Natural Language Inferences Using Cross-Document Structure Theory
Janz, A., Kurowski, D., Baran, J., Moska, J., Bernaś, T., & Oleksy, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70816-9_21
Sentence-Space Metrics (SSM) for the Evaluation of Sentence Comprehension
Lin, J., Chen, H., & Ding, N. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_29
Understanding and Reasoning with Negation
Hossain, M. M. (2022) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2048613
Self-Explaining Counterfactual Data Augmentation for NLP
An, D., Wang, F., Lu, J., & Zhang, S. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035355
“In-Domain Out-of-Domain SNLI [22] Human CAD [4] Breaking NLI [23] ACC”
Explainable Numerical Claim Verification
Aarnes, P. R. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761666
“For transformer models [33], adversarial techniques include para-phrasing, back-translation, and synonym substitutions to preserve semantics [7, 9, 14, 21, 37].”
Noghabaei, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.06234
“Properly handling negation is vital for NLI, yet models frequently perform poorly on negation-involved examples (Glockner et al., 2018).”
Understanding and Guarding against Natural Language Adversarial Examples
Mozes, M. (2024) · UCL Discovery (University College London) · Cited by 2
Transfer learning and robustness for natural language processing
Jin, D. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1
Kanerva, J. (2024) · UTUPub (University of Turku)
Neuro-Symbolic Methods for Natural Language Inference and Question Answering
Feng, Y. (2022) · QSpace (Queen's University Library) · Cited by 1
Persona-Aware Alignment Framework for Personalized Dialogue Generation
Li, G., Liu, X., Wu, Z., & Dai, X. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.57 · Cited by 2
“For the English experiments, we further trained this RoBERTa model on SNLI (Glockner et al., 2018) and MultiNLI (Kim et al., 2019) datasets.”
Towards optimal adversarial texts: character, word, and sentence
Wang, P., Li, D., & Li, Q. (2025) · Cybersecurity · doi:10.1186/s42400-025-00500-3
BRIDGE-Embed: A Novel LLM-Based Document and Sentence Retrieval Method
Ning, T., Wu, Y., & Liu, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3830-0_28
Profiling German Text Simplification with Model-Fingerprints
Klöser, L., Beele, M., & Kraft, B. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15638-9_9
InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks
Banerjee, S., Sarkar, M., Saha, P., Mathew, B., & Mukherjee, A. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/36pvjo35qvn7 · Cited by 2
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability
Chowdhury, J. R., & Caragea, C. (2023) · neural information processing systems · doi:10.52202/075280-3038 · Cited by 5
Chowdhury, J. R., & Caragea, C. (2023) · neural information processing systems · doi:10.52202/075280-1268 · Cited by 3
Pitkiin kysymyksiin vastaamisen toteutettavuustutkimus
Antikainen, S. (2022) · Aaltodoc (Aalto University)
Semantic and Discursive Representation for Natural Language Understanding
Sileo, D. (2019) · theses.fr (ABES) · doi:10.70675/06f08c91z80ddz448az8a2dz424ef97b76d2
On the Two-Stage BERT-Self Attack Against the Sentiment Classification
Luo, J.-K., Peng, X.-A., Dong, J., Xu, S.-S., Gao, X., & Fu, Q. (2025) · China Automation Congress (CAC) · doi:10.1109/cac67268.2025.11487268
“As to the semantic preservation, the decision logic of the language reasoning system is deconstructed in [30] through an artificial intervention mechanism, while an adversarial example construction framework is developed in [31] for machine reading comprehension systems.”
Mersinias, M., & Valvis, P. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/45snhhbhahiy
“The usage of contrast sets (Gardner et al., 2020), checklist sets (Ribeiro et al., 2020) or other adversarial sets (Jia and Liang, 2017), (Wallace et al., 2019), (Bartolo et al., 2020), (Glockner et al., 2018), (McCoy et al., 2019) makes performance plummet and thus highlights this issue.”
BackgroundHEH ebaluazioa PhrasIS datu-basean: inferentziaren mugak aztertzen
Larraya, J. F. A. (2024) · Communities in ADDI (University of the Basque Country)
Causal Adversarial Training for Improving Language Understanding and Robustness
Zhu, B., Xie, Y., Feng, Y., Wang, F. L., Xie, H., & Rao, Y. (2026) · Computational Linguistics · doi:10.1162/coli.a.644
Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data CurationPreprint
Kazoom, R., Cohen, O., Puzis, R., Shabtai, A., & Hadar, O. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18681
“Despite rapid progress, even state-of-the-art models remain brittle, often relying on spurious lexical cues or failing under simple syntactic and semantic variations (Glockner et al., 2018; Carmona et al., 2018).”
自然言語処理における文字レベルと単語レベルの表現を組み合わせるメカニズムの研究
Thenot, B. (2020) · Institutional Repositories DataBase (IRDB) · doi:10.15083/0002004904
Metadata
- Authors
- Glockner, M., Shwartz, V., & Goldberg, Y.
- Affiliations
- Technische Universität Darmstadt (DE)
- Bar-Ilan University (IL)
- Corresponding author
- Goldberg, Yoav
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 650-655
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2798665661
- Cited by
- 319
- References
- 20
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 13 · 2023: 26 · 2022: 48 · 2021: 65 · 2020: 69 · 2019: 72 · 2018: 14
- FWCI
- 22.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (DA 1600/1-1)
- Israel Science Foundation
- Intel Collaboration Research Institute for Computational Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- natural language inference, lexical inference, world knowledge, SNLI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.