Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set.Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [24]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“61 (Landis and Koch, 1977).”
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 4×
“All models are based on pre-trained GloVe embeddings (Pennington et al., 2014), which are either fine-tuned during training (RESIDUAL-STACKED-ENCODER and ESIM) or stay fixed (DECOMPOSABLE ATTENTION).”
MethodInfluentialA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 6×
“The SNLI dataset (Stanford Natural Language Inference, Bowman et al., 2015) consists of 570k sentence-pairs manually labeled as entailment, contradiction, and neutral.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 3×
“The MultiNLI dataset (Multi-Genre Natural Language Inference, Williams et al., 2018) was collected similarly to SNLI, though covering a wider range of genres, and supporting a cross-genre evaluation.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426 · cited 1×
“Premises are image captions from Young et al. (2014), while hypotheses were generated by crowd-sourced workers who were shown a premise and asked to generate entailing, contradicting, and neutral sentences.”
BackgroundAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“Inspired by the work of Jia and Liang (2017) on reading comprehension, we create a new NLI test set with examples that capture various kinds of lexical knowledge (Table 1).”
MethodA Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296 · cited 2×
“Finally, DECOMPOSABLE ATTENTION (Parikh et al., 2016) performs soft alignment of words from the premise to words in the hypothesis using attention mechanism, and decomposes the task into comparison of aligned words.”
BackgroundMethodEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113 · cited 3×
“, 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rocktäschel et al., 2016; Chen et al., 2017).”
BackgroundMethodAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“Our results are in line with Gururangan et al. (2018) and Po-liak et al. (2018), who showed that the label can be identified by looking only at the hypothesis and exploiting annotation artifacts such as word choice and sentence length.”
ResultReasoning about Entailment with Neural Attention
Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582 · cited 1×
“…Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.g. Rockt¨aschel et al., 2016; Chen et al., 2017).”
Research works printed but not linked: 2
- Christiane Fellbaum. 1998. WordNet . Wiley Online Library.
- Yixin Nie and Mohit Bansal. 2017. Shortcutstacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . · arXiv
Cited by
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
“Feature combination crucially depends on shortcut features like word length [59, 60, 14, 61], and consequently leads to a severe lack of robustness such as an inability to generalise to more challenging test conditions [62, 63, 64, 65].”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 35
“This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.”
ResultMulti-Task Deep Neural Networks for Natural Language Understanding
Liu, X., He, P., Chen, W., & Gao, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1441 · Cited by 1,154
“At last, we also would like to verify whether MT-DNN is resilience against adversarial attacks (Glockner et al., 2018; Talman and Chatzikyriakidis, 2018; Liu et al., 2019).”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10044 · Cited by 205
“The success of MultiNLI might also be surprising given recent concerns about the generalization abilities of models trained on it (Glockner et al., 2018), particularly related to “annotation artifacts” caused by using crowd workers to write the hypothesis statements (Gururangan et al., 2018).”
BackgroundA Survey of Data Augmentation Approaches for NLP
Feng, S., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.84 · Cited by 670
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
“Some of these works focus on specific semantic phenomena, such as lexical inferences (Glockner et al., 2018) and quantifiers (Geiger et al., 2018), or biases based on specific words (Sanchez et al., 2018).”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
“This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.”
ResultSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1009 · Cited by 690
“As a result, models trained on such datasets with hu-1 man biases run the risk of over-estimating the actual performance on the underlying task, and are vulnerable to adversarial or out-of-domain examples (Wang et al., 2018; Glockner et al., 2018).”
BackgroundBERT-ATTACK: Adversarial Attack Against BERT Using BERT
Li, L., Ma, R., Guo, Q., Xue, X., & Qiu, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.500 · Cited by 574
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1300 · Cited by 2,985
“The success of MultiNLI might also be surprising given recent concerns about the generalization abilities of models trained on it (Glockner et al., 2018), particularly related to “annotation artifacts” caused by using crowd workers to write the hypothesis statements (Gururangan et al., 2018).”
BackgroundAnalysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
“For instance, Sanchez et al. (2018) and Glockner et al. (2018) extracted examples from SNLI (Bow-man et al., 2015) and replaced specific words such as hypernyms, synonyms, and antonyms, followed by manual verification.”
BackgroundLearning the Difference that Makes a Difference with Counterfactually-Augmented Data
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12434 · Cited by 227
“[6] demonstrate that simply replacing words by synonyms or hypernyms, which should not alter the applicable label, nevertheless breaks ML-based NLI systems.”
BackgroundWord-level Textual Adversarial Attacking as Combinatorial Optimization
Zang, Y., Qi, F., Yang, C., Liu, Z., Zhang, M., Liu, Q., & Sun, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.540 · Cited by 195
“…methods (step 1) involve using word embeddings (Sato et al., 2018) or language model (Zhang et al., 2019a) to filter words, selecting syn-onyms as substitutes (Samanta and Mehta, 2017; Ren et al., 2019; Jin et al., 2019), and their combinations (Alzantot et al., 2018; Glockner et al., 2018).”
MethodBisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 274
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395
“There is much recent work focusing on statistical cues in datasets in vision (Jo and Bengio, 2017) and NLP (Sanchez et al., 2018; McCoy et al., 2019; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Rajpurkar et al., 2018; Jia and Liang, 2017).”
BackgroundDon’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381
“Evaluating models on out-of-domain examples built by applying minor perturbations to existing examples has also been the subject of recent study (Szegedy et al., 2014; Belinkov and Bisk, 2018; Carlini and Wagner, 2018; Glockner et al., 2018).”
Backgrounde-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367
“Glockner et al. [11] introduce a toy dataset, BreakingNLI, to test whether natural language inference models capture world knowledge and generalize beyond statistical regularities.”
BackgroundNeural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335
“(2018); Glockner et al. (2018) in natural language inference, Goyal et al.”
BackgroundEvaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 256
“The idea of creating challenging contrastive evaluation sets has a long history (Levesque et al., 2011; Ettinger et al., 2017; Glockner et al., 2018; Naik et al., 2018; Isabelle et al., 2017).”
BackgroundPAWS: Paraphrase Adversaries from Word Scrambling
Zhang, Y., Baldridge, J., & He, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1131 · Cited by 596
“A notable exception is Glockner et al. (2018): they generated both entailment and contradiction examples by replacing words with their synonyms or antonyms.”
BackgroundCombating Adversarial Misspellings with Robust Word Recognition
Pruthi, D., Dhingra, B., & Lipton, Z. C. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1561 · Cited by 275
“Inspired by this work, Glockner et al. (2018) demonstrate an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym.”
BackgroundHateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H., & Pierrehumbert, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.4 · Cited by 187
“In its use of contrastive cases for model evaluation, H ATE C HECK builds on a long history of minimally-contrastive pairs in NLP (e.g. Levesque et al., 2012; Sennrich, 2017; Glockner et al., 2018; Warstadt et al., 2020).”
MethodInfluentialPAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Yang, Y., Zhang, Y., Tar, C., & Baldridge, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1382 · Cited by 223
“, 2018), textual entailment (Zhao et al., 2018; Glockner et al., 2018), and text classification (Alzantot et al.”
BackgroundGeva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284
“In this paper, we continue recent efforts to understand biases that are introduced during the process of data creation (Levy et al., 2015; Schwartz et al., 2017; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Aharoni and Goldberg, 2018; Paun et al., 2018).”
Self-citationNeural Natural Language Inference Models Enhanced with External Knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253
“Compared with the performance on the SNLI test (Glockner et al., 2018). set, the performance of the three baseline models dropped substantially on the (Glockner et al., 2018) test set, with the differences ranging from 22.3% to 32.8% in accuracy.”
BackgroundMethodResultInfluentialRecent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 56
“SherLIiC by Schmitt and Schütze (2019) is a benchmark for lexical inference in context (LIiC), i.e., a subset of textual entailment focusing on relationships between words, which Glockner, Shwartz, and Goldberg (2018) found that previous benchmarks lacked, and state-ofthe-art systems struggled with.”
BackgroundPolyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.523 · Cited by 161
“…2020), likely because their distributions are less similar to the original SST-2 training data. challenge sets (Kim et al., 2019; Naik et al., 2018; Glockner et al., 2018; Wang et al., 2019); it exhibits comparable (or better) gains than m-CAD (manual counterfactuals) with less implementation and…”
MethodDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks
Kaushik, D., & Lipton, Z. C. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1546 · Cited by 228
PAWS: Paraphrase Adversaries from Word ScramblingPreprint
Zhang, Y., Baldridge, J., & He, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01130 · Cited by 234
“A notable exception is Glockner et al. (2018): they generated both entailment and contradiction examples by replacing words with their synonyms or antonyms.”
BackgroundImproving the Reliability of Deep Neural Networks in NLP: A Review
Alshemali, B., & Kalita, J. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105210 · Cited by 188
“[49] designed testing sets that showed the deficiency of natural language inference (NLI) models.”
KLUE: Korean Language Understanding Evaluation
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 75
“Methods like subject/object swapping or passivization is motivated by existing NLI data augmentation strategies [89, 42].”
MethodThe Importance of Being Recurrent for Modeling Hierarchical Structure
Tran, K., Bisazza, A., & Monz, C. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1503 · Cited by 164
“This shows that the SNLI task can be largely solved by exploiting shallow features without understanding the underlying linguistic structures, which has also been pointed out by recent work (Glockner et al., 2018; Gururangan et al., 2018).”
BackgroundUnlearn Dataset Bias in Natural Language Inference by Fitting the Residual
He, H., Zha, S., & Wang, H. (2019) · Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019) · doi:10.18653/v1/d19-6115 · Cited by 157
“…rules that target a specific type of bias, e.g., swapping male and female entities (Zhao et al., 2018a, 2019), synonym/antonym substitution (Glockner et al., 2018), and syntactic rules (McCoy et al., 2019; Ribeiro et al., 2018), and may require human verification (Zhang et al., 2019c; Jia…”
BackgroundLearning The Difference That Makes A Difference With Counterfactually-Augmented Data
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2020) · International Conference on Learning Representations · Cited by 511
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 120
Towards a Robust Deep Neural Network against Adversarial Texts: A Survey
Wang, W., Wang, R., Wang, L., Wang, Z., & Ye, A. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3117608 · Cited by 198
“This kind of attack usually appears in other NLP tasks like natural language inference (NLI) [63], neural machine translation (NMT) [68], reading comprehension (RC) [9], and question answering (QA) [69].”
BackgroundImproving Natural Language Inference Using External Knowledge in the Science Questions Domain
Wang, X., Kapanipathi, P., Musa, R., Yu, M., Talamadupula, K., Abdelaziz, I., Chang, M., Fokoue, A., Makni, B., Mattei, N., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017208 · Cited by 148
Robustness Gym: Unifying the NLP Evaluation Landscape
Goel, K., Rajani, N. F., Vig, J., Taschdjian, Z., Bansal, M., & Ré, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations · doi:10.18653/v1/2021.naacl-demos.6 · Cited by 98
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 241
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Wallace, E., Rodriguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00279 · Cited by 129
“Other work focuses on natural language inference, where challenge examples highlight model failures (Glockner et al., 2018; Naik et al., 2018; Wang et al., 2019).”
BackgroundAn Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
Tu, L., Lalwani, G., Gella, S., & He, H. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00335 · Cited by 122
“lifu-tu/Study-NLP-Robustness poorly on the so-called challenging or adversarial datasets where such correlations no longer hold (Glockner et al., 2018; McCoy et al., 2019; Nie et al., 2019; Zhang et al., 2019).”
BackgroundProbing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
Ravichander, A., Belinkov, Y., & Hovy, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.295 · Cited by 132
Evaluating Models' Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 40
“Challenge sets To probe a model’s ability to handle specific types of phenomena, recent work develops challenge sets (Glockner et al., 2018; Naik et al., 2018; Isabelle et al., 2017).”
BackgroundAchieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Huang, P.-S., Stanforth, R., Welbl, J., Dyer, C., Yogatama, D., Gowal, S., Dvijotham, K., & Kohli, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1419 · Cited by 113
“It is worth noting that symbol substitution is general enough to encompass other threat models such as lexical entailment perturbations (Glockner et al., 2018), and could potentially be extended to the addition of pre/postfixes (Jia and Liang, 2017; Wallace et al., 2019).”
BackgroundProbing Natural Language Inference Models through Semantic Fragments
Richardson, K., Hu, H., Moss, L., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6397 · Cited by 112
Towards Debiasing NLU Models from Unknown Biases
Utama, P. A., Moosavi, N. S., & Gurevych, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.613 · Cited by 98
A Hierarchy of Limitations in Machine Learning
Malik, M. M. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2002.05193 · Cited by 51
“Word embeddings, a representation derived from “unsupervised” learning, work fascinatingly well for many language-based prediction tasks; however, it is actually a supervised learning task of imputing a held-out word, and beyond its impressive successes, word embeddings have key limitations for what might be said to b…”
BackgroundCollecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
Poliak, A., Haldar, A., Rudinger, R., Hu, J. E., Pavlick, E., White, A. S., & Van Durme, B. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1007 · Cited by 102
“There exist more semantic phenomena integral to NLU (Allen, 1995) and we plan to include them in future versions of the DNC. Previous Recast NLI Example sentences in RTE1 (Dagan et al., 2006) were extracted from MT, IE, and QA datasets, with the process referred to as ‘recasting’ in the thesis by Glickman (2006).”
InfluentialRobust Natural Language Processing: Recent Advances, Challenges, and Future Directions
Omar, M., Choi, S., Nyang, D., & Mohaisen, D. (2022) · IEEE Access · doi:10.1109/access.2022.3197769 · Cited by 85
“The classification accuracy has been utilized by numerous research works [34], [35], [40], [41], [45], [59], [103], [105], [106].”
MethodTranslation Artifacts in Cross-lingual Transfer Learning
Artetxe, M., Labaka, G., & Agirre, E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.618 · Cited by 188
“Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark (Naik et al., 2018; Glockner et al., 2018; Nie et al., 2020).”
BackgroundAre Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Jeretic, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.768 · Cited by 75
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
BackgroundINFOTABS: Inference on Tables as Semi-structured Data
Gupta, V., Mehta, M., Nokhiz, P., & Srikumar, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.210 · Cited by 77
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
Geiger, A., Richardson, K., & Potts, C. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.16 · Cited by 44
“This data generation process is similar to that of Glockner et al. (2018), except they focus on the lexical relations of exclusion and synonymy, while we focus on entailment relations.”
BackgroundDon't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset BiasesPreprint
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03683 · Cited by 116
“Evaluating models on out-of-domain examples built by applying minor perturbations to existing examples has also been the subject of recent study (Szegedy et al., 2014; Belinkov and Bisk, 2018; Carlini and Wagner, 2018; Glockner et al., 2018).”
BackgroundStance Detection Benchmark: How Robust is Your Stance Detection?
Schiller, B., Daxenberger, J., & Gurevych, I. (2021) · Künstliche Intell. · doi:10.1007/s13218-021-00714-w · Cited by 133
“In contrast to their work, we will use the (Glockner et al., 2018; Minervini and Riedel, 2018), these stress tests have been applied to a wide range of tasks from Question-Answering (Wang and Bansal, 2018) to Natural Machine Translation (Belinkov and Bisk, 2017) and Fact Checking (Thorne et al.,…”
BackgroundMethodFrequency-Guided Word Substitutions for Detecting Textual Adversarial Examples
Mozes, M., Stenetorp, P., Kleinberg, B., & Griffin, L. D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.13 · Cited by 128
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets
Liu, N. F., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1225 · Cited by 77
“These challenges have been used as evidence that current systems are brittle (Belinkov and Bisk, 2018; Mudrakarta et al., 2018; Zhao et al., 2018; Glockner et al., 2018; Ebrahimi et al., 2018; Ribeiro et al., 2018, 1 Often referred to as “adversarial datasets” or “attacks”.”
BackgroundSemantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language
Huebner, P. A., Sulem, E., Cynthia, F., & Roth, D. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.49 · Cited by 70
“In recent years, there has been a surge of interest in linguistic evaluations of language models (Glockner et al., 2018; Ettinger, 2020; Linzen et al., 2016; Gulordava et al., 2018; Goldberg, 2019; Tran et al., 2018; Bacon and Regier, 2019; McCoy et al., 2020; Linzen, 2020).”
BackgroundAn Analysis of Natural Language Inference Benchmarks through the Lens of Negation
Hossain, M. M., Kovatchev, V., Dutta, P., Kao, T. H., Wei, E., & Blanco, E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.732 · Cited by 130
“Glockner et al. (2018) show that models trained with SNLI fail to resolve new pairs that require simple lexical substitution, e.g., holding a saxo-phone contradicts holding an electric guitar .”
BackgroundAnalyzing Compositionality-Sensitivity of NLI Models
Nie, Y., Wang, Y., & Bansal, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016867 · Cited by 76
GenAug: Data Augmentation for Finetuning Text Generators
Feng, S. Y., Gangal, V., Kang, D., Mitamura, T., & Hovy, E. H. (2020) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2020.deelio-1.4 · Cited by 77
EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
Ravichander, A., Naik, A., Rose, C., & Hovy, E. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1033 · Cited by 66
“Prior work on challenge tasks has already made headway in defining tasks for subproblems such as lexical inference with hypernymy, co-hyponymy, antonymy (Glockner et al., 2018; Naik et al., 2018).”
BackgroundCombating Adversarial Misspellings with Robust Word RecognitionPreprint
Pruthi, D., Dhingra, B., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11268 · Cited by 85
“Inspired by this work, Glockner et al. (2018) demonstrate an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym.”
BackgroundDon’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1084 · Cited by 73
Adapting natural language processing for technical text
Dima, A. A., Lukens, S., Hodkiewicz, M. R., Sexton, R., & Brundage, M. P. (2021) · Applied AI Letters · doi:10.1002/ail2.33 · Cited by 89
“1,3,6,7 There is also concern that the current NLP training and evaluation paradigm naturally favors models for which large amounts of data are available.”
BackgroundDISCO: Distilling Counterfactuals with Large Language Models
Chen, Z., Gao, Q., Bosselut, A., Sabharwal, A., & Richardson, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.302 · Cited by 160
Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets
Wu, Y., Gardner, M., Stenetorp, P., & Dasigi, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.190 · Cited by 101
A survey on Recognizing Textual Entailment as an NLP Evaluation
Poliak, A. (2020) · Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2020.eval4nlp-1.10 · Cited by 41
Increasing Robustness to Spurious Correlations using Forgettable Examples
Yaghoobzadeh, Y., Mehri, S., Combes, R. T. D., Hazen, T. J., & Sordoni, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.291 · Cited by 94
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples
Kang, D., Khot, T., Sabharwal, A., & Hovy, E. H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1225 · Cited by 84
“2A similar approach was used in a parallel work to generate an adversarial dataset from SNLI (Glockner et al., 2018).”
MethodCan Neural Networks Understand Monotonicity Reasoning?
Yanaka, H., Mineshima, K., Bekki, D., Inui, K., Sekine, S., Abzianidze, L., & Bos, J. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4804 · Cited by 99
“A number of recent works attempt to test and analyze what type of inferences an NLI model may be performing, focusing on various types of lexical inferences (Glockner et al., 2018; Naik et al., 2018; Poliak et al., 2018) and logical inferences (Bowman et al., 2015b; Evans et al., 2018).”
BackgroundEvaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 75
A Logic-Driven Framework for Consistency of Neural Models
Li, T., Gupta, V., Mehta, M., & Srikumar, V. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1405 · Cited by 104
Shifting the Baseline: Single Modality Performance on Visual Navigation &
Thomason, J., Gordon, D., & Bisk, Y. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1197 · Cited by 59
“This work directly extends observations made within both the Computer Vision (Goyal et al., 2018) and Natural Language (Glockner et al., 2018; Poliak et al., 2018; Gururangan et al., 2018; Kaushik and Lipton, 2018) communities that sophisticated models often perform well by fitting to spurious correlations in the data,…”
BackgroundInfluentialMore Bang for Your Buck: Natural Perturbation for Robust Question Answering
Khashabi, D., Khot, T., & Sabharwal, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.12 · Cited by 56
“A closely relevant line of work is adversarial perturbations to expose the weaknesses of systems upon local changes and criticize their lack robustness (Ebrahimi et al., 2018; Glockner et al., 2018; Dinan et al., 2019).”
BackgroundCan I Trust the Explainer? Verifying Post-hoc Explanatory MethodsPreprint
Camburu, O.-M., Giunchiglia, E., Foerster, J., Lukasiewicz, T., & Blunsom, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02065 · Cited by 82
“Such assumptions may be poor, since recent works show a series of surprising spurious correlations in human-annotated datasets, on which neural networks learn to heavily rely [10, 9, 5].”
BackgroundExplaining The Efficacy of Counterfactually Augmented DataPreprint
Kaushik, D., Setlur, A., Hovy, E. H., & Lipton, Z. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02114 · Cited by 66
“…shown to be vulnerable to synthetic transformations, such as distractor phrases (Jia & Liang, 2017; Wallace et al., 2019), document paraphrases (Iyyer et al., 2018; Pfeiffer et al., 2019), and meaning preserving synthetic modifications (Ribeiro et al., 2018; Glockner et al., 2018; Shen et al., 2018).”
BackgroundDynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56
“These efforts seek to determine whether models have met specific learning targets (Alzantot et al., 2018; Glockner et al., 2018; Naik et al., 2018; Nie et al., 2019), exploit relatively superficial properties of their training data, (Jia and Liang, 2017; Kaushik and Lipton, 2018; Zhang et al.”
BackgroundTrichelair, P., Emami, A., Trischler, A., Suleman, K., & Cheung, J. C. K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1335 · Cited by 66
“In similar spirit, Glockner et al. (2018) create an NLI test set specifically to show the deficiencies of stateof-the-art models in inferences that require lexical and world knowledge.”
Varshney, N., Mishra, S., & Baral, C. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.158 · Cited by 32
HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning
Yanaka, H., Mineshima, K., Bekki, D., Inui, K., Sekine, S., Abzianidze, L., & Bos, J. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1027 · Cited by 82
“There are also NLI datasets which expand lexical knowledge by re-placing words using lexical rules (Monz and de Ri-jke, 2001; Glockner et al., 2018; Naik et al., 2018; Poliak et al., 2018a).”
BackgroundSentence embeddings in NLI with iterative refinement encoders
Talman, A., Yli-Jyrä, A., & Tiedemann, J. (2019) · Natural Language Engineering · doi:10.1017/s1351324919000202 · Cited by 68
“In the second experiment we conducted testing of the proposed sentence embedding architecture using the Breaking NLI test set recently published by Glockner et al. (2018). The test set is designed to highlight the lack of lexical reasoning capability of NLI systems.”
MethodResultInfluentialCollecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
Poliak, A., Haldar, A., Rudinger, R., Hu, J. E., Pavlick, E., White, A. S., & Van Durme, B. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5441 · Cited by 42
“It includes syntactic and surface form tests but does not focus on as a wide range of semantic phenomena as in the DNC. Glockner et al. (2018) introduce a modified version of SNLI to test how well NLI models perform when requiring lexical and world knowledge.”
Influentialpair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
Joshi, M., Choi, E., Levy, O., Weld, D. S., & Zettlemoyer, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1362 · Cited by 56
“Following standard practice (Glockner et al., 2018), we train all models on a combination of SNLI (Bowman et al., 2015) and MultiNLI.”
BackgroundMethodInfluentialTasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis
Xing, X., Jin, Z., Jin, D., Wang, B., Zhang, Q., & Huang, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.292 · Cited by 72
COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences
Singh, S., Wen, N., Hou, Y., Alipoormolabashi, P., Wu, T.-L., Ma, X., & Peng, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.78 · Cited by 78
NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning
Chen, Z., Gao, Q., & Moss, L. S. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.7 · Cited by 61
“However, several experiments show that DL models lack generalization ability, adopt fallible syntactic heuristics, and show exploitation of annotation artifacts (Glockner et al., 2018; McCoy et al., 2019; Gururangan et al., 2018).”
BackgroundMining Discourse Markers for Unsupervised Sentence Representation Learning
Sileo, D., de Cruys, T. V., Pradel, C., & Müller, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1351 · Cited by 95
“Inspired by recent work (Dasgupta et al., 2018; Poliak et al., 2018; Levy et al., 2018; Glockner et al., 2018) on the unexpected properties of recent manually labelled datasets (e.”
BackgroundMethodClever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M., & Shwartz, V. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.138 · Cited by 44
Self-citationDeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking
Hidey, C., Chakrabarty, T., Alhindi, T., Varia, S., Krstovski, K., Diab, M., & Muresan, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.761 · Cited by 61
Towards Robustifying NLI Models Against Lexical Dataset Biases
Zhou, X., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.773 · Cited by 62
Elazar, Y., Zhang, H., Goldberg, Y., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.819 · Cited by 61
“Recent studies proposed to augment test instances with minimal pairs, that either change the original answer (Kaushik et al., 2019; Gardner et al., 2020), or keep it intact by using paraphrasing, synonyms, etc. (Glockner et al., 2018; Shah et al., 2019).”
BackgroundSelf-citationAdversarial attacks and defenses for large language models (LLMs): methods, frameworks & challenges
Rai, P. K. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00334-8 · Cited by 57
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 97
Testing the Generalization Power of Neural Network Models across NLI Benchmarks
Talman, A., & Chatzikyriakidis, S. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4810 · Cited by 59
“Our findings, together with the previous negative findings e.g. by Glockner et al. (2018) and Gururangan et al. (2018), indicate that the current state-of-the-art neural network models fail to capture the semantics of NLI in a way that will enable them to generalize across different NLI situations.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Glockner, M., Shwartz, V., & Goldberg, Y.
- Affiliations
- Technische Universität Darmstadt (DE)
- Bar-Ilan University (IL)
- Corresponding author
- Goldberg, Yoav
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2018
- Pages
- 650-655
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2798665661
- Cited by
- 319
- References
- 20
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 13 · 2023: 26 · 2022: 48 · 2021: 65 · 2020: 69 · 2019: 72 · 2018: 14
- FWCI
- 22.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (DA 1600/1-1)
- Israel Science Foundation
- Intel Collaboration Research Institute for Computational Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- natural language inference, lexical inference, world knowledge, SNLI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.