What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Open accessThe DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index.
Abstract
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For these kinds of tasks, we can also score the generation probability of each candidate and choose the one with the highest probability [25].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [25]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“BERT is a deep bidirectional transformer network (Vaswani et al., 2017) pre-trained on tasks of masked language modeling (predicting masked words given bidirectional context) and next-sentence prediction (binary classification of whether two sentences are a sequence).”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“As a case study, we use these three diagnostics to examine the predictive capacities of the pre-trained BERT model (Devlin et al., 2019), which has been the basis of impressive performance across a wide range of tasks.”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“, 2019) and ELMo (Peters et al., 2018a) have advanced the state of the art in a wide variety of tasks, suggesting that these models acquire valuable, generalizable linguistic competence during the pre-training process.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 2×
“Much work has attempted to evaluate systems’ overall level of ‘‘understanding’’, often with tasks such as semantic similarity and entailment (Wang et al., 2018; Bowman et al., 2015; Agirre et al., 2012; Dagan et al., 2005; Bentivogli et al., 2016), and additional work has been done to design curated versions of these…”
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
Brain potentials during reading reflect word expectancy and semantic association
Kutas, M., & Hillyard, S. A. (1984) · Nature · doi:10.1038/307161a0 · Cited by 2,218 · cited 1×
“Like cloze, the N400 can be used to gauge how expected a word w is in a context c —the amplitude of the N400 response appears to be sensitive to fit of a word in context, and has been shown to correlate with cloze in many cases (Kutas and Hillyard, 1984).”
MethodWhat Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity
Agirre, E., Cer, D. M., Diab, M., & González-Agirre, A. (2012) · Joint Conference on Lexical and Computational Semantics · Cited by 1,063
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Research works printed but not linked: 19
- Adi Yossi, 2016, International Conference on Learning Representations
- Agirre Eneko, 2012, Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, 385
- Chowdhury Shammur Absar, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 133
- Clark Kevin, 2019, arXiv preprint arXiv:1906.04341 · arXiv
- Conneau Alexis, 2018, ACL 2018-56th Annual Meeting of the Association for Computational Linguistics, 2126
- Dagan Ido, 2005, Machine Learning Challenges Workshop, 177
- Dasgupta Ishita, 2018, Proceedings of the 40th Annual Meeting of the Cognitive Science Society
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Ettinger Allyson, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 1790
- Frank Stefan L., 2013, ACL 2013-51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 2, 878
- Futrell Richard, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 32
- Giulianelli Mario, 2018, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 240, 10.18653/v1/W18-5426 · DOI
- Gulordava Kristina, 2018, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1, 1195
- Kim Najoung, 2019, Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), 235
- Peters Matthew, 2018, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1, 2227
- Tenney Ian, 2019, arXiv preprint arXiv:1905.05950 · arXiv
- Tenney Ian, 2019, Proceedings of the International Conference on Learning Representations
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Wang Alex, 2018, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 353
Cited by
Items 401–500 of 553
Diyasa, I. G. S. M., Puspaningrum, E. Y., Saputra, D., & Awang, W. S. W. (2025) · Interdisciplinary Journal of Information Knowledge and Management · doi:10.28945/5525 · Cited by 2
On Masked Language Models for Contextual Link Prediction
Brayne, A., Wiatrak, M., & Corneil, D. (2022) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2022.deelio-1.9 · Cited by 4
“Their work has been further extended to probing areas including semantic (Ettinger, 2020; Wallace et al., 2019), commonsense (Tamborrino et al., 2020; Forbes et al., 2019; Roberts et al., 2020), and linguistic (Lin et al., 2019; Tenney et al., 2019) knowledge.”
BackgroundBetz, G., & Richardson, K. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.900943 · Cited by 5
“Arguably one of their major, widely acknowledged failures consists in the fact that the output of NLMs suffers from spectacular inconsistencies (Ribeiro et al., 2019; Ettinger, 2020; Kassner and Schütze, 2020).”
BackgroundA Transformer for SAG: What Does it Grade?
Willms, N., & Padó, U. (2022) · Linköping electronic conference proceedings · doi:10.3384/ecp190012 · Cited by 4
“…with different strategies over the last years, finding that BERT learns syntactic (He-witt and Manning, 2019; Tenney et al., 2019) and semantic representations (Ettinger, 2020) that are generally preserved through fine-tuning for semantic tasks like paraphrasing (P´erez-Mayos et al., 2021).”
BackgroundMethodThe Two Sides of the Environmental Kuznets Curve: A Socio-Semantic Analysis
Menezes, T., Pottier, A., & Roth, C. (2023) · OEconomia · doi:10.4000/oeconomia.15729 · Cited by 5
“This disadvantage is accentuated by the fact that this NLP task is somewhat delicate, particularly in identifying if a result is positive or negative, and understanding negation appears to be a typical shortcoming of current deep learning models (Ettinger, 2020).”
BackgroundTesting the Effectiveness of the Diagnostic Probing Paradigm on Italian Treebanks
Miaschi, A., Alzetta, C., Brunato⋄, D., Dell’Orletta⋄, F., & Venturi⋄, G. (2023) · Information · doi:10.3390/info14030144 · Cited by 6
Huebner, P. A., & Willits, J. A. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100478 · Cited by 3
Fine-tuning language models to recognize semantic relations
Roussinov, D., Sharoff, S., & Puchnina, N. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09677-w · Cited by 2
“Similar observations were reported by Ettinger (2020), who also observed that BERT can generally discriminate between good and bad completions involving shared categories or role reversals, reliably retrieves noun hypernyms, but encounters difficulties in challenging inference and role-based event…”
ResultSieker, J., & Zarrieß, S. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.14 · Cited by 2
“However, in general, there is an increasing interest in examining the linguistic capabilities captured in LLMs, often facilitated through the utilization of linguistic test suites and experimental datasets (e.g., Belinkov and Glass (2019); Ettinger (2020)).”
BackgroundMethodResultInfluentialWang, L. Z., & Steinert‐Threlkeld, S. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.15 · Cited by 2
“Utilizing psycholinguistic tests, Ettinger (2020) have shown that the BERT model has difficulty acquiring generalizable meanings of negation quantifiers.”
MethodCalculating Semantic Frequency of GSL Words Using a BERT Model in Large Corpora
Liu, L., Tongxi, G., Jian-jun, S., & Guo, Y. (2025) · SAGE Open · doi:10.1177/21582440251333182 · Cited by 3
“Additionally, Ettinger (2020) revealed that BERT can represent semantic roles, though it struggles with more complex linguistic phenomena.”
Using Generative AI for Identifying Electoral Irregularities on Social Media
Moura, S., Santos, E., Sampaio, P. A., & Brito, K. D. S. (2025) · Annual International Conference on Digital Government Research · doi:10.59490/dgo.2025.990 · Cited by 2
“Previous studies, such as those by Ettinger (2020), have already demonstrated that language models are highly sensitive to the input they receive and that small changes in structure, tone, or vocabulary can lead to significantly different responses.”
Cavalleri, E., Soto Gomez, M., Pashaeibarough, A., Malchiodi, D., Casiraghi, E., Valentini, G., & Mesiti, M. (2024) · Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) · Cited by 11
How does BERT process disfluency?
Tian, Y., Nieradzik, T., Jalali, S., & Shiu, D.-S. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.22 · Cited by 5
Pre-trained language models evaluating themselves - A comparative study
Koch, P., Aßenmacher, M., & Heumann, C. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.25 · Cited by 2
“…based metrics, like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004), come with severe drawbacks (Belz and Reiter, 2006; Reiter and Belz, 2009) and given the the increasing versatility of modern NLG systems, they are assumed to strug-027 gle even more (Zhang et al., 2020; Sellam et al., 2020).”
BackgroundCong, Y. (2022) · Workshop on Understanding Implicit and Underspecified Language (UnImplicit) · doi:10.18653/v1/2022.unimplicit-1.1 · Cited by 2
“A lot of attention has been paid to increase LMs’ general transparency (Ettinger, 2020; Rogers et al., 2020), among which studies on LMs’ interpretation of implicitness mostly focus on scalar implicature”
BackgroundA Study on Subjective Information in Deep Neural Network Models
Kobayashi, A., Takahashi, Y., & Yamaguchi, S. (2022) · International Symposium on Computing and Networking Workshops (CANDARW) · doi:10.1109/candarw57323.2022.00077 · Cited by 2
Assessing linguistic generalisation in language models: a dataset for Brazilian Portuguese
Wilkens, R., Zilio, L., & Villavicencio, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09664-1 · Cited by 2
A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions
Berger, U., Frermann, L., Stanovsky, G., & Abend, O. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.172 · Cited by 4
BERT Shows Garden Path Effects
Irwin, T., Wilson, K., & Marantz, A. P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.235 · Cited by 14
“Roles, proto-roles, entity types, and relations are contained in token representations in addition to syntactic information (Ettinger, 2020; Tenney et al., 2019).”
BackgroundCao, B., Tang, Q., Lin, H., Han, X., & Sun, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.143 · Cited by 2
“Currently, the most popular strategy is prompt-based probing (Petroni et al., 2019; Ettinger, 2020; Srivastava et al., 2022), e.g., query BERT with “Tim Cook is the CEO of [MASK]” to determine whether it contains corresponding knowledge.”
BackgroundMadusanka, T., Zahid, I., Li, H., Pratt-Hartmann, I. E., & Batista-Navarro, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.536 · Cited by 13
“Our work follows the literature on probing how different linguistic properties affect the behaviour of neural approaches such as transformer-based language models (Madusanka et al., 2023; Clark et al., 2021; Buijtelaar and Pezzelle, 2023; Jawahar et al., 2019; Ettinger, 2020).”
BackgroundGenerating Derivational Morphology with BERT.Preprint
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2020) · arXiv (Cornell University) · Cited by 2
"Let's Eat Grandma": When Punctuation Matters in Sentence Representation for Sentiment Analysis.Preprint
Karami, M., Mosallanezhad, A., Mancenido, M. V., & Liu, H. (2021) · arXiv (Cornell University) · Cited by 2
Better than BERT but Worse than BaselinePreprint
Liu, B., Huang, J., Cai, X., & Church, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05915 · Cited by 2
“Nevertheless, there are opportunities to improve BERT on numbers (Wallace et al., 2019), negation (Ettinger, 2020), and more.”
Zou, J., & Ding, N. (2021) · Research Square · doi:10.21203/rs.3.rs-813994/v1 · Cited by 2
Acquiring and modelling abstract commonsense knowledge via conceptualization
He, M. (2022) · doi:10.14711/thesis-991013098357503412 · Cited by 5
Circling the Void: Using Heidegger and Lacan to think about Large Language ModelsPreprint
Heimann, M., & Hübener, A.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3023378/v2 · Cited by 2
The 7th Workshop on Online Abuse and Harms (WOAH)
(2023) · doi:10.18653/v1/2023.woah-1 · Cited by 2
Language-Based Depth Hints for Monocular Depth EstimationPreprint
Auty, D., & Mikolajczyk, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.15551 · Cited by 3
“BERT embeddings have been shown to encode whole syntax trees [15] and knowledge graphs [36], and to contain a degree of semantic information [9].”
BackgroundLorge, I., Joyce, D. W., Taylor, N., Nevado‐Holgado, A. J., Cipriani, A., & Kormilitzin, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4777347 · Cited by 3
Metheniti, E. (2023) · theses.fr (ABES) · doi:10.70675/2f81071bz4724z44ecz999dz1acee85d8951 · Cited by 5
Stengel-Eskin, E., & Durme, B. V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.760 · Cited by 5
Leveraging Taxonomic Information from Large Language Models for Hyponymy Prediction
Chernomorchenko, P., Panchenko, A., & Nikishina, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54534-4_4 · Cited by 1
“Specifically, Ettinger notes in [10] that model predictions highly depend on the particular input, while in [12] is noted that universal prompts marking IS-A relations do not provide enough information for the model to distinguish homonyms.”
BackgroundInfluentialYoo, M. H., & Song, S. (2025) · Lingua · doi:10.1016/j.lingua.2025.103998 · Cited by 1
Using Knowledge Induction strategies: LLMs can do better in knowledge-driven dialogue tasks
Peng, S., Zhang, W., Zhang, H., Li, S., & Qu, D. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101903 · Cited by 1
Large Language Models Are Stochastic Measuring Devices
Mallory, F. (2026) · Oxford University Press eBooks · doi:10.1093/9780191998317.003.0003 · Cited by 1
Beyond Perplexity: A Multi-Faceted Analysis of a Novel Densely Connected Transformer
Santis, E. D., Martino, A., & Rizzi, A. (2026) · Applied Sciences · doi:10.3390/app16062721 · Cited by 1
On the impact of pretraining data ordering in transformer encoder- and decoder-only language models
Dini, L., Domenichelli, L., Brunato⋄, D., & Dell’Orletta⋄, F. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115850 · Cited by 1
Zhang, E., Zhang, C., & Chen, Y. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.100898 · Cited by 1
Hsieh, T. Y. J., Eriksson, C. O., Meckler, G. D., Hansen, M. L., Bahr, N. J., Bedrick, S., Trujillo, D., Kim, K. S., Cho, B., Zhang, K., et al. (2026) · Learning Health Systems · doi:10.1002/lrh2.70088 · Cited by 2
MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling
BehnamGhader, P., Zakerinia, H., & Baghshah, M. S. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.12 · Cited by 1
You Can’t Learn What’s Not There: Self Supervised Learning and the Poverty of the Stimulus
Veres, C., & Sampson, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80599-9_1 · Cited by 1
GANDALF: a General Character Name Description Dataset for Long Fiction
Carlsson, F., Sahlgren, M., Olsson, F., & Gyllensten, A. C. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.13 · Cited by 2
“It is also straightforward to include negated statements in the character description, enabling studies of models’ sensitivity to negation (Ettinger, 2020).”
BackgroundCitiusNLP at SemEval-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization
Gamallo, P. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.34 · Cited by 1
“In fact, the predictions by BERT are not altered even by the recurrent presence of syntactic problems in the input text, such as truncated sentences, shuffled word order, or removed subjects and objects (Ettinger, 2020).”
BackgroundProSPer: Probing Human and Neural Network Language Model Understanding of Spatial Perspective
Masis, T., & Anderson, C. J. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.8 · Cited by 1
“…et al., 2018; Gulordava et al., 2018; Lin et al., 2019), question formation (Jumelet et al., 2019; McCoy et al., 2020), filler-gap dependencies (Wilcox et al., 2018), anaphora (Jumelet et al., 2019), category membership (Ettinger, 2020), and negative polarity items (Jumelet and Hupkes, 2018).”
BackgroundSemantic Analysis of Winograd Schema No. 1
Bennett, B. (2021) · Frontiers in artificial intelligence and applications · doi:10.3233/faia210369 · Cited by 2
“It does not work well on sentences involving function words such as negation [11], and lacks of robustness to with respect to semantically insignificant variations of input sentences (e.”
BackgroundSelecting Context Clozes for Lightweight Reading Compliance
Keim, G., & Littman, M. L. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.21 · Cited by 1
“More recently, this same dataset has been reused in work specifically aimed at understanding how LLMs respond to structured prompts (Ettinger, 2020), and we follow their convention in annotating the set as "CPRAG" after "Common Sense and Pragmatic Inference."”
MethodBudrionis, A., Chomutare, T., Svenning, T. O., & Dalianis, H. (2022) · Linköping electronic conference proceedings · doi:10.3384/ecp187029 · Cited by 1
“Ettinger [5] performed fine-tuning of BERT on manually annotated data sets containing both negations and non-negations.”
BackgroundUchida, T., Lair, N., Ishiguro, H., & Dominey, P. F. (2022) · Neurobiology of Language · doi:10.1162/nol_a_00090 · Cited by 2
“In certain aspects they are similar enough to human performance that specific measures of human language comprehension from psycholinguistic experiments are now being used to characterize and evaluate these LMs (Ettinger, 2020; Ettinger et al., 2016).”
BackgroundEventBERT: Incorporating Event-Based Semantics for Natural Language Understanding
Zou, A., Zhang, Z., & Zhao, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_5 · Cited by 1
“Recent studies show that current prominent pre-trained language models have already incorporated semantic information to some extent (Clark et al., 2019), yet such implicit semantic information is far from enough for comprehensive natural language understanding (Ettinger, 2020).”
BackgroundExploring Category Structure with Contextual Language Models and Lexical Semantic Networks
Renner, J., Denis, P., Gilleron, R., & Brunellière, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.167 · Cited by 1
Luo, X., Fan, C., Zhang, Y., Jiang, W.-G., Qin, B., & Xu, R.-F. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.findings-emnlp.233 · Cited by 2
“Et-9 tinger (2020) finds that BERT struggles with challenging commonsense inferences but it is robust on within-category distinctions.”
BackgroundGuiding Zero-Shot Paraphrase Generation with Fine-Grained Control Tokens
Vahtola, T., Creutz, M., & Tiedemann, J. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.29 · Cited by 1
“For instance, large language models do not reliably represent negation (Ettinger, 2020), which can be a result of not having a sufficient number of such examples in the training data.”
BackgroundIs Shortest Always Best? The Role of Brevity in Logic-to-Text Generation
Calò, E., Levy, J., Gatt, A., & van Deemter, K. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.17 · Cited by 1
“L O L A Length f is not large, f is a cube or f is a dodecahedron. nuances, such as negation (Ettinger, 2020), which is crucial for evaluating our task.”
BackgroundLiu, C., & Chersoni, E. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.28 · Cited by 3
“…the target linguistic knowledge (Tenney et al., 2019a,b; Hewitt and Liang, 2019; Wu et al., 2020; Vuli ´ c et al., 2020; Sorodoc et al., 2020; Ettinger, 2020; Geiger et al., 2021; Koto et al., 2021; Chersoni et al., 2021a; Conia and Navigli, 2022; Kim and Linzen, 2020; Arps et al., 2022;…”
BackgroundNot Wacky vs. Definitely Wacky: A Study of Scalar Adverbs in Pretrained Language Models
Lorge, I., & Pierrehumbert, J. B. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.23 · Cited by 1
Liu, X., Rzepka, R., & Araki, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111883 · Cited by 1
Nedumpozhimana, V., & Kelleher, J. D. (2024) · Natural language processing. · doi:10.1017/nlp.2024.43 · Cited by 1
Abzianidze, L., Bylinina, L., & Paperno, D. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009542340 · Cited by 1
Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learning
Hong, S. J., & Bennett, B. (2020) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.ldk.2021.41 · Cited by 3
Periñán-Pascual, C. (2022) · Oxford University Press eBooks · doi:10.1093/obo/9780199772810-0295 · Cited by 1
Circling the Void: Using Heidegger and Lacan to think about Large Language ModelsPreprint
Heimann, M., & Hübener, A.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3023378/v1 · Cited by 1
Advanced Language Understanding with Syntax-Enhanced TransformerPreprint
Rine, W., Patel, R., & Steve, N. (2023) · Preprints.org · doi:10.20944/preprints202312.1673.v1 · Cited by 1
Madrid, J., Diehl, P., Selig, M., Rolauffs, B., Hans, F. P., Busch, H.-J., Scheef, T., & Benning, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4194579/v1 · Cited by 2
Lin, J., Korpi, T., Kuukka, A., Tirkkonen, A., Kariluoto, A., Kaijansinkko, J., Satamo, M., Pajulammi, H. M., Haapanen, M. J., Häyrynen, S., et al. (2024) · medRxiv · doi:10.1101/2024.10.08.24315141 · Cited by 1
Referring expression generation in context: Combining linguistic and computational approaches
Same, F. (2024) · doi:10.5281/zenodo.11058113 · Cited by 3
Dinkar, T. (2022) · theses.fr (ABES) · doi:10.70675/15ad82a0ze517z457bzb7bfz67979f57a892 · Cited by 2
Evaluating and Improving the Reasoning Abilities of Language Models
Helwe, C. (2024) · theses.fr (ABES) · doi:10.70675/7fc70e59zdd27z4b0fza38dz03a2c67cda50 · Cited by 1
The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from HumansPreprint
Chlapanis, O. S., Mastromichalakis, O. M., & Papadimitriou, C. H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.08837 · Cited by 1
Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent SystemsPreprint
Frias, S. E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.10216 · Cited by 1
“Masked language models largely ignore negation (Kassner & Schütze, 2020; Ettinger, 2020) and sentence-embedding similarity scores negated pairs among the highest .”
Pre-trained Language Models as Symbolic Reasoners over Knowledge?Preprint
Kassner, N., Kroje, B., & Schütze, H. (2020) · arXiv (Cornell University) · Cited by 1
NwQM: A neural quality assessment framework for Wikipedia
Reddy, B. P., Bhushan, S., Sarkar, S., & Mukherjee, A. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4066404
On Learning and Applying Text Representations
Chu, Z. (2020) · Knowledge@UChicago (University of Chicago) · doi:10.6082/uchicago.2666 · Cited by 1
Is c-command Machine-learnable?
Shin, U., & Song, S. (2021) · The Linguistic Association of Korea Journal
Exploring the Use of Neural Transformers for Psycholinguistics
Laverghetta, A. (2021) · Digital Commons - University of South Florida (University of South Florida) · Cited by 1
Variation and generality in encoding of syntactic anomaly information in sentence embeddings
Wu, Q., & Ettinger, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.18 · Cited by 2
Self-citationSchmitt, S. (2021) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000139679
Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks
Kalyaeva, E., Durandin, O., & Malafeev, A. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_075 · Cited by 1
“In contrast to previous studies (Ettinger, 2020), where the focus was on psycholinguistic diagnostics and comparison between machine and human performance, we made our study more linguistically grounded by adding knowledge from theory of language.”
BackgroundResultUsing Distributional Principles for the Semantic Study of Contextual Language ModelsPreprint
Ferret, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12174
“, 2019) or tasks close to lexical substitution (Ettinger, 2020; Garí Soler et al., 2019).”
BackgroundInvestigating associative, switchable and negatable Winograd items on renewed French data sets
Wang, X., Seminck, O., & Amsili, P. (2022) · HAL · Cited by 1
Augmenting Structure with Text for Improved Graph Learning
Safavi, T. (2022) · Deep Blue (University of Michigan) · doi:10.7302/6246 · Cited by 2
Probing BERT on Syntactic Representation Paradigms: Dependency vs Constituency TreesPreprint
Alzetta, C., Miaschi, A., Dell’Orletta⋄, F., Venturi⋄, G., & Montemagni⋄, S. (2022) · Research Square · doi:10.21203/rs.3.rs-2202147/v1 · Cited by 1
(2022) · Linköping electronic conference proceedings · doi:10.3384/ecp190 · Cited by 1
Probabilistic Reasoning with Transformer Networks: An Application to Predicate JudgmentPreprint
Bhatia, S. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/s3d74
Exploring how well Large-scale Masked Language Models can Recognize Grammatical Errors
Kimura, M., Nagata, R., & Hanawa, K. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.689
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction
Yang, Y., Panagopoulou, A., Apidianaki, M., Yatskar, M., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.45 · Cited by 1
Leveraging Context for Perceptual Prediction using Word EmbeddingsPreprint
Carter, G., Keller, F., & Hoffman, P. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/34tke
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_4
Cognitive Signals of Language Processing
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_3
Circling the Void: Using Heidegger and Lacan to think about Large Language ModelsPreprint
Heimann, M., & Hübener, A.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3023378/v3
Hai-Jew, S. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch005
Disentangling the Linguistic Competence of Privacy-Preserving BERT
Arnold, S., Kemmerzell, N., & Schreiner, A. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.5
Balaur: Language Model Pretraining with Lexical Semantic Relations
Mircea, A., & Cheung, J. C. K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.674
“…lexical semantic relations (LSRs) such as hypernymy: notably cloze completions such as “ A fox is a type of ”, where the hypernym “ canine ” is a valid completion (Ettinger, 2020); and monotonicity-based inferences such as “ Bob saw a fox ” entailing “ Bob saw a canine ” (Geiger et al., 2020).”
BackgroundResultInfluentialRyu, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.27
Metadata
- Authors
- Ettinger, A.
- Affiliations
- University of Chicago (US)
- Corresponding author
- Ettinger, Allyson
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 34-48
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2965647350
- Cited by
- 511
- References
- 38
- Citations by year
- 2026: 16 · 2025: 32 · 2024: 39 · 2023: 122 · 2022: 92 · 2021: 151 · 2020: 53
- FWCI
- 41.25
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- BERT, language model pretraining, role reversal, negation, event prediction, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.