CoChat

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Open access
Glockner, Max, Shwartz, Vered, Goldberg, Yoav · 2018 · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set.Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p18-2103

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • The Measurement of Observer Agreement for Categorical Data

    Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×

    “61 (Landis and Koch, 1977).”
  • Glove: Global Vectors for Word Representation

    Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×

    “All models are based on pre-trained GloVe embeddings (Pennington et al., 2014), which are either fine-tuned during training (RESIDUAL-STACKED-ENCODER and ESIM) or stay fixed (DECOMPOSABLE ATTENTION).”
    MethodInfluential
  • A large annotated corpus for learning natural language inference

    Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 6×

    “The SNLI dataset (Stanford Natural Language Inference, Bowman et al., 2015) consists of 570k sentence-pairs manually labeled as entailment, contradiction, and neutral.”
    BackgroundMethodInfluential
  • A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

    Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 3×

    “The MultiNLI dataset (Multi-Genre Natural Language Inference, Williams et al., 2018) was collected similarly to SNLI, though covering a wider range of genres, and supporting a cross-genre evaluation.”
    Background
  • From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426 · cited 1×

    “Premises are image captions from Young et al. (2014), while hypotheses were generated by crowd-sourced workers who were shown a premise and asked to generate entailing, contradicting, and neutral sentences.”
    Background
  • Adversarial Examples for Evaluating Reading Comprehension Systems

    Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×

    “Inspired by the work of Jia and Liang (2017) on reading comprehension, we create a new NLI test set with examples that capture various kinds of lexical knowledge (Table 1).”
    Method
  • A Decomposable Attention Model for Natural Language Inference

    Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 2×

    “Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
    BackgroundMethod
  • Enhanced LSTM for Natural Language Inference

    Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×

    “, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”
    BackgroundMethod
  • Annotation Artifacts in Natural Language Inference Data

    Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×

    “Our results are in line with Gururangan et al. (2018) and Po-liak et al. (2018), who showed that the label can be identified by looking only at the hypothesis and exploiting annotation artifacts such as word choice and sentence length.”
    Result
  • Reasoning about Entailment with Neural Attention

    Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582 · cited 1×

    “…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
  • Hypothesis Only Baselines in Natural Language Inference

    Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., & Van Durme, B. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2023 · Cited by 480

  • SciTaiL: A Textual Entailment Dataset from Science Question Answering

    Khot, T., Sabharwal, A., & Clark, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12022 · Cited by 411 · cited 1×

    “The SciTail dataset (Khot et al., 2018), created from science exams, is somewhat different from the two datasets, being smaller (27,026 examples), and labeled only as entailment or neutral.”
    Method
  • Natural Language Inference by Tree-Based Convolution and Heuristic Matching

    Mou, L., Men, R., Li, G., Xu, Y., Zhang, L., Yan, R., & Jin, Z. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2022 · Cited by 332 · cited 1×

    “All models predict the label using a concatenation of features derived from the sentence representations (e.g. maximum, mean), for example as in Mou et al. (2016).”
    Method
  • Neural Natural Language Inference Models Enhanced with External Knowledge

    Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253 · cited 6×

    “E SIM (Enhanced Sequential Inference Model, Chen et al., 2017) is a hybrid TreeLSTM-based and biLSTM-based model.”
    BackgroundMethodInfluential
  • A Fast Unified Model for Parsing and Sentence Understanding

    Bowman, S. R., Gauthier, J., Rastogi, A., Gupta, R., Manning, C. D., & Potts, C. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1139 · Cited by 226 · cited 1×

    “Sentence-encoding models (e.g. Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et…”
    Method
  • Natural Language Inference over Interaction Space

    Gong, Y., Luo, H., & Zhang, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04348 · Cited by 208

  • Recognizing Textual Entailment: Models and Applications

    Dagan, I., Roth, D., Sammons, M., & Zanzotto, F. M. (2013) · Synthesis lectures on human language technologies · doi:10.2200/s00509ed1v01y201305hlt023 · Cited by 204

  • Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling

    Shen, T., Zhou, T., Long, G., Jiang, J., Wang, S., & Zhang, C. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/604 · Cited by 143

  • Shortcut-Stacked Sentence Encoders for Multi-Domain Inference

    Nie, Y., & Bansal, M. (2017) · Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP · doi:10.18653/v1/w17-5308 · Cited by 127 · cited 2×

    “Sentence-encoding models (e.g. Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.”
    BackgroundMethod

Research works printed but not linked: 2

  1. Christiane Fellbaum. 1998. WordNet . Wiley Online Library.
  2. Yixin Nie and Mohit Bansal. 2017. Shortcutstacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . · arXiv

Cited by

  • Shortcut learning in deep neural networks

    Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224

    “Feature combination crucially depends on shortcut features like word length [59, 60, 14, 61], and consequently leads to a severe lack of robustness such as an inability to generalise to more challenging test conditions [62, 63, 64, 65].”
    Background
  • HellaSwag: Can a Machine Really Finish Your Sentence?

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 35

    “This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.”
    Result
  • Multi-Task Deep Neural Networks for Natural Language Understanding

    Liu, X., He, P., Chen, W., & Gao, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1441 · Cited by 1,154

    “At last, we also would like to verify whether MT-DNN is resilience against adversarial attacks (Glockner et al., 2018; Talman and Chatzikyriakidis, 2018; Liu et al., 2019).”
    Background
  • BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10044 · Cited by 205

    “The success of MultiNLI might also be surprising given recent concerns about the generalization abilities of models trained on it (Glockner et al., 2018), particularly related to “annotation artifacts” caused by using crowd workers to write the hypothesis statements (Gururangan et al., 2018).”
    Background
  • A Survey of Data Augmentation Approaches for NLP

    Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670

See all 350

Metadata

Authors
Glockner, M., Shwartz, V., & Goldberg, Y.
Affiliations
  • Technische Universität Darmstadt (DE)
  • Bar-Ilan University (IL)
Corresponding author
Goldberg, Yoav
Venue
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Published
2018
Pages
650-655
ISSN
0736-587X (ISSN-L)
OpenAlex
W2798665661
Cited by
319
References
20
Citations by year
2026: 3 · 2025: 9 · 2024: 13 · 2023: 26 · 2022: 48 · 2021: 65 · 2020: 69 · 2019: 72 · 2018: 14
FWCI
22.45
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Deutsche Forschungsgemeinschaft (DA 1600/1-1)
  • Israel Science Foundation
  • Intel Collaboration Research Institute for Computational Intelligence
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
natural language inference, lexical inference, world knowledge, SNLI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.