What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Open accessThe DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index.
Abstract
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second trend is that neural networks routinely fail on exactly the phenomena where such symbolic models succeed, for example, logical inference [6,9], compositionality [7,23] and out-of-distribution generalization [24–26].”
From Symbols and grounding in large language models (2023) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For these kinds of tasks, we can also score the generation probability of each candidate and choose the one with the highest probability [25].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [25]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“BERT is a deep bidirectional transformer network (Vaswani et al., 2017) pre-trained on tasks of masked language modeling (predicting masked words given bidirectional context) and next-sentence prediction (binary classification of whether two sentences are a sequence).”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“As a case study, we use these three diagnostics to examine the predictive capacities of the pre-trained BERT model (Devlin et al., 2019), which has been the basis of impressive performance across a wide range of tasks.”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“, 2019) and ELMo (Peters et al., 2018a) have advanced the state of the art in a wide variety of tasks, suggesting that these models acquire valuable, generalizable linguistic competence during the pre-training process.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 2×
“Much work has attempted to evaluate systems’ overall level of ‘‘understanding’’, often with tasks such as semantic similarity and entailment (Wang et al., 2018; Bowman et al., 2015; Agirre et al., 2012; Dagan et al., 2005; Bentivogli et al., 2016), and additional work has been done to design curated versions of these…”
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
Brain potentials during reading reflect word expectancy and semantic association
Kutas, M., & Hillyard, S. A. (1984) · Nature · doi:10.1038/307161a0 · Cited by 2,218 · cited 1×
“Like cloze, the N400 can be used to gauge how expected a word w is in a context c —the amplitude of the N400 response appears to be sensitive to fit of a word in context, and has been shown to correlate with cloze in many cases (Kutas and Hillyard, 1984).”
MethodWhat Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity
Agirre, E., Cer, D. M., Diab, M., & González-Agirre, A. (2012) · Joint Conference on Lexical and Computational Semantics · Cited by 1,063
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Research works printed but not linked: 19
- Adi Yossi, 2016, International Conference on Learning Representations
- Agirre Eneko, 2012, Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, 385
- Chowdhury Shammur Absar, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 133
- Clark Kevin, 2019, arXiv preprint arXiv:1906.04341 · arXiv
- Conneau Alexis, 2018, ACL 2018-56th Annual Meeting of the Association for Computational Linguistics, 2126
- Dagan Ido, 2005, Machine Learning Challenges Workshop, 177
- Dasgupta Ishita, 2018, Proceedings of the 40th Annual Meeting of the Cognitive Science Society
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Ettinger Allyson, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 1790
- Frank Stefan L., 2013, ACL 2013-51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 2, 878
- Futrell Richard, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 32
- Giulianelli Mario, 2018, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 240, 10.18653/v1/W18-5426 · DOI
- Gulordava Kristina, 2018, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1, 1195
- Kim Najoung, 2019, Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), 235
- Peters Matthew, 2018, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1, 2227
- Tenney Ian, 2019, arXiv preprint arXiv:1905.05950 · arXiv
- Tenney Ian, 2019, Proceedings of the International Conference on Learning Representations
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Wang Alex, 2018, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 353
Cited by
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“zed the roles of BERT’s layers in dierent tasks and found that BERT solves tasks in a similar order to that in NLP pipelines. Furthermore, knowledge of subject-verb agreement [50] and semantic roles [44] are also confirmed to exist in BERT. Besides, Hewitt and Manning [59], Jawahar et al. [72], Kim et al. [85] proposed…”
MethodA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
“However, we do have evidence from an MLM probing study that BERT has some knowledge for semantic roles (Ettinger, 2019).”
BackgroundRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“Prompts can also be used to explore the linguistic knowledge of PLMs, focusing on diferent phenomena such as analogies [39], negation [43] or semantic similarity [42].”
BackgroundExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”
BackgroundBERT applications in natural language processing: a review
Gardazi, N. M., Daud, A., Malik, M. K., Bukhari, A., Alsahfi, T., & Alshemaimri, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 251
“Negation is disregarded by the model proposed by (Ettinger 2020 ), and the integration of Condi - tional Random Fields (CRF) may be necessary to enhance its efficacy in some tasks and languages, particularly in the context of sequence classification tasks (Souza et al. 2020).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
“At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 22, 47, 85], BERT’s learned knowledge of syntax [26, 29, 30, 43, 45, 74], and BERT’s learned knowledge of semantics [23, 74].”
BackgroundPretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Kassner, N., & Schütze, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.698 · Cited by 223
“Ettinger (2019) defines and applies psycho-linguistic diagnostics for PLMs.”
BackgroundWu, T.-T., Lee, H.-Y., Li, P.-H., Huang, C.-N., & Huang, Y.-M. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331231191125 · Cited by 249
“Subsequently, two primary pre-training models emerged: the BERT model (Ettinger, 2020) and the GPT model (Radford et al., 2018).”
MethodoLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 271
“…and fine-tuning a simple, often linear, randomly initialized probe, to determine whether the representations hold relevant information (Ettinger et al., 2016; Adi et al., 2016; Belinkov and Glass, 2019; Hewitt and Manning, 2019; Wallace et al., 2019; Rozen et al., 2019; Peters et al.,…”
BackgroundHateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H., & Pierrehumbert, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.4 · Cited by 187
“, 2017; Belinkov and Bisk, 2018) and language modelling (Marvin and Linzen, 2018; Ettinger, 2020).”
BackgroundFrom Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 236
Lee, H.-Y., Chen, P.-H., Wang, W.-S., Huang, Y.-M., & Wu, T.-T. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00447-4 · Cited by 273
“This pivotal shift catalyzed the development of notable pre-trained models like BERT (Ettinger, 2020) and GPT (Radford et al., 2018).”
MethodSinha, K., Jia, R., Hupkes, D., Pineau, J., Williams, A., & Kiela, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.230 · Cited by 181
“As an alternative to parametric probes, stimulibased probing (Linzen et al., 2016; Marvin and Linzen, 2018; Gulordava et al., 2018; Goldberg, 2019; Wolf, 2019; Warstadt et al., 2019a, 2020a,b; Ettinger, 2020) has been used to show that even without a learned probing function, BERT can predict syntactic properties with…”
MethodDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
“address more mundane hazards tied to the premature deployment of a system that does not work [55, 189].”
BackgroundMethodA systematic review of natural language processing applied to radiology reports
Casey, A., Davidson, E., Poon, M., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01533-7 · Cited by 206
“Recent evidence shows that embeddings generated by BERT fail to show a generalisable understanding of negation [131], an essential factor in interpreting radiology reports effectively.”
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226
“Prior work uses LMs for zero-shot learning by choosing an arbitrary prompt [17, 79] which requires no data but is suboptimal [4].”
MethodMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“…(Rogers et al., 2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Man-ning, 2019; Tenney et al., 2019; Elazar et al., 2021), commonsense (Forbes et al., 2019; Da and Kasai, 2019; Zhang et al., 2020) and reasoning (Talmor et al., 2020; Kassner et al., 2020), usually assessed by measures…”
BackgroundQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2023) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 156
“The Cloze format is currently resurging in popularity also as away to evaluatemasked languagemodels [91, 106], as fundamentally the Cloze task is what these models are doing in pre-training.”
BackgroundWhat Happens To BERT Embeddings During Fine-tuning?
Merchant, A., Rahimtoroghi, E., Pavlick, E., & Tenney, I. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.4 · Cited by 139
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203
Cold-start Active Learning through Self-supervised Language Modeling
Yuan, M., Lin, H.-T., & Boyd-Graber, J. L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.637 · Cited by 281
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Schick, T., Schmid, H., & Schütze, H. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.488 · Cited by 184
Caufield, J. H., Hegde, H., Emonet, V., Harris, N. L., Joachimiak, M. P., Matentzoglu, N., Kim, H., Moxon, S., Reese, J. T., Haendel, M. A., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae104 · Cited by 120
“These techniques have known limitations, such as being prone to hallucinations [5] (i.e., generating incorrect statements) and insensitivity to negations [6].”
BackgroundInterpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.431 · Cited by 139
“…(2019) consider what is learned from context, Clark et al. (2019); Michel et al. (2019) look at specific attention heads, Hewitt and Manning (2019); Ettinger (2020) address linguistic understanding such as syntax and negation, and Wallace et al. (2019); Tan and Celis (2019) address ethical…”
BackgroundHow Can We Accelerate Progress Towards Human-like Linguistic Generalization?
Linzen, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.465 · Cited by 148
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131
Large language models demonstrate the potential of statistical learning in language
Contreras Kallens, P., Kristensen‐McLachlan, R. D., & Christiansen, M. H. (2023) · Cognitive Science · doi:10.1111/cogs.13256 · Cited by 130
“Some of this work has already been done in the context of encoder-only masked language models, such as BERT and its related descendants (Ettinger, 2020; Pandia & Ettinger, 2021; Rogers, Kovaleva, & Rumshisky, 2020).”
Transformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 147
McCoy, R. T., Min, J., & Linzen, T. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.21 · Cited by 98
Consistency Analysis of ChatGPT
Jang, M., & Lukasiewicz, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.991 · Cited by 67
Few-Shot Text Generation with Pattern-Exploiting Training
Schick, T., & Schütze, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11926 · Cited by 70
“The idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., Petroni et al., 2019; Wang et al., 2019; Schick and Schütze, 2020c; Ettinger, 2020).”
MethodFew-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107
“The idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., Petroni et al., 2019; Wang et al., 2019; Talmor et al., 2020; Schick and Schütze, 2020; Ettinger, 2020; Kassner and Schütze, 2020; Sakaguchi et al., 2020).”
MethodPham, T., Bui, T., Mai, L., & Nguyen, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.98 · Cited by 79
“Ettinger (2020) found that BERT heavily relies on word order when predicting missing words in masked sentences from the CPRAG-102 dataset.”
MethodAl-Garadi, M. A., Yang, Y., Cai, H., Ruan, Y., O’Connor, K., Gonzalez‐Hernandez, G., Perrone, J., & Sarker, A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01394-0 · Cited by 123
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.146 · Cited by 71
“…et al., 2019; Hewitt and Man-ning, 2019; Goldberg, 2019; Warstadt et al., 2019), semantic knowledge (Tenney et al., 2019; Wal-lace et al., 2019; Ettinger, 2020) and world knowledge (Davison et al., 2019; Bouraoui et al., 2020; Forbes et al., 2019; Zhou et al., 2019; Roberts et al., 2020; Lin…”
BackgroundSinha, K., Parthasarathi, P., Pineau, J., & Williams, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.569 · Cited by 71
“In fact, Ettinger (2020) observed that for one of three test sets, BERT loses some accuracy in word-perturbed sentences, but that there exists a subset of examples for which BERT’s accuracy remains intact.”
BackgroundHONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
Analysis and Evaluation of Language Models for Word Sense Disambiguation
Loureiro, D., Rezaee, K., Pilehvar, M. T., & Camacho-Collados, J. (2021) · Computational Linguistics · doi:10.1162/coli_a_00405 · Cited by 140
Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects
Michaelov, J. A., Bardolph, M. D., Petten, C. K. V., Bergen, B. K., & Coulson, S. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00105 · Cited by 133
“…run in R (R Core Team, 2022) using RStudio (RStudio Team, 2020) and the tidyverse (Wickham et al., 2019), lme4 (Bates et al., 2015), lmerTest (Kuznetsova et al., 2017), corrr (Kuhn et al., 2022), colorspace (Zeileis et al., 2009, 2020), gridExtra (Auguie, 2017), and cowplot (Wilke, 2020) packages.”
BackgroundMethodProbing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N., Zhao, X., Fang, T., Song, Y., & Yeung, D.-Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.329 · Cited by 69
Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for ChinesePreprint
Zhang, Z., Zhang, H., Chen, K., Guo, Y., Hua, J., Wang, Y., & Zhou, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06696 · Cited by 124
“…model effectiveness, although PLMs have shown effectiveness in capturing syntax and semantic knowledge after pre-training (Hewitt and Manning, 2019; Ettinger, 2020), recent studies show that the current models still suffer from under-fitting issues, and it remains challenging to train a powerful…”
Dimensions of commonsense knowledge
Ilievski, F., Oltramari, A., Ma, K., Zhang, B., McGuinness, D. L., & Szekely, P. A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107347 · Cited by 106
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85
“Much recent work has used probing methods to evaluate what these models have and have not learned (Belinkov and Glass, 2019; Tenney et al., 2019b; Rogers et al., 2020; Ettinger, 2020).”
MethodVisual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83
Semantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language
Huebner, P. A., Sulem, E., Cynthia, F., & Roth, D. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.49 · Cited by 70
“In recent years, there has been a surge of interest in linguistic evaluations of language models (Glock-ner et al., 2018; Ettinger, 2020; Linzen et al., 2016; Gulordava et al., 2018; Goldberg, 2019; Tran et al., 2018; Bacon and Regier, 2019; McCoy et al., 2020; Linzen, 2020).”
BackgroundEvent Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely
Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fedorenko, E., & Lenci, A. (2023) · Cognitive Science · doi:10.1111/cogs.13386 · Cited by 53
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
Ushio, A., Espinosa-Anke, L., Schockaert, S., & Camacho-Collados, J. (2021) · doi:10.18653/v1/2021.acl-long.280 · Cited by 116
“The main aspects that have been studied in recent years are syntax (Goldberg, 2019; Saphra and Lopez, 2019; Hewitt and Manning, 2019; van Schijndel et al., 2019; Jawahar et al., 2019; Tenney et al., 2019b) and semantics (Ettinger, 2019; Tenney et al., 2019a).”
BackgroundSuperbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.279 · Cited by 135
“tion and antonymy are known to be challenging for PLMs (Ettinger, 2020; Kassner and Schütze, 2020) and might be one of the reasons for the generally lower performance on Amazon.”
BackgroundLanguage models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K. M., & Cohn, T. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.10 · Cited by 130
Glavaš, G., & Vulić, I. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.270 · Cited by 85
Machine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 79
Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 104
Mallory, F. (2023) · Ergo an Open Access Journal of Philosophy · doi:10.3998/ergo.4668 · Cited by 44
Large Language Models Are Biased Because They Are Large Language Models
Resnik, P. (2025) · Computational Linguistics · doi:10.1162/coli_a_00558 · Cited by 95
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Swiatczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 54
Sinclair, A., Jumelet, J., Zuidema, W., & Fernández, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00504 · Cited by 48
“Several studies have shown that the strength of the priming effect increases after repeated exposure to a given structure (Kaschak et al., 2011; Jaeger and Snider, 2013) and tends to decay if material intervenes between prime and target (Reitter et al., 2011).”
BackgroundMethodInfluentialIs Multilingual BERT Fluent in Language Generation?
Rönnqvist, S., Kanerva, J., Salakoski, T., & Ginter, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03806 · Cited by 51
“Similarly, Ettinger (2019) evaluates the BERT model on several English psycholinguistic datasets, where the model is shown generally being able to distinguish a good completion from a bad one, while still failing in some more complex categories, for example being insensitive to negation.”
BackgroundMethodInfluentialAre Pretrained Language Models Symbolic Reasoners over Knowledge?
Kassner, N., Krojer, B., & Schütze, H. (2020) · Proceedings of the 24th Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.45 · Cited by 35
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
Exploring BERT’s Sensitivity to Lexical Cues using Tests from Semantic Priming
Misra, K., Ettinger, A., & Rayz, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.415 · Cited by 79
Self-citationDecoding EEG Brain Activity for Multi-Modal Natural Language Processing
Hollenstein, N., Renggli, C., Glaus, B., Barrett, M., Tröndle, M., Langer, N., & Zhang, C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.659410 · Cited by 67
Dynamic Contextualized Word Embeddings
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2021) · doi:10.18653/v1/2021.acl-long.542 · Cited by 75
Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-79531-8 · Cited by 54
“…have the following characteristics: (a) sentences are affirmative (i.e., contain no negation, which is known to negatively impact LLM performance 29 ), (b) they are linked through coordination, (c) either 3 or 5 human entities are featured in each prompt, all identified by proper names (i.e.,…”
Multilingual Language Models Predict Human Reading Behavior
Hollenstein, N., Pirovano, F., Zhang, C., Jäger, L. A., & Beinborn, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.10 · Cited by 55
BERT Embeddings for Automatic Readability Assessment
Imperial, J. M. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_069 · Cited by 87
“, 2019), semantic roles (Ettinger, 2019) as well as entity types and relations (Tenney et al.”
BackgroundChanging the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations
Zanzotto, F. M., Santilli, A., Ranaldi, L., Onorati, D., Tommasino, P., & Fallucchi, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.18 · Cited by 71
“…many syntactic probes (Conneau et al., 2018) for neural networks have been designed to test for specific phenomena (Kovaleva et al., 2019; Jawahar et al., 2019; Hewitt and Manning, 2019; Ettinger, 2019; Goldberg, 2019) or for full syntactic trees (Hewitt and Manning, 2019; Mareček and Rosa, 2019).”
BackgroundWhen Choosing Plausible Alternatives, Clever Hans can be Clever
Kavumba, P., Inoue, N., Heinzerling, B., Singh, K., Reisert, P., & Inui, K. (2019) · doi:10.18653/v1/d19-6004 · Cited by 74
“A second possible explanations is BERT’s insensitivity to negations (Ettinger, 2019).”
BackgroundDifferentiable Subset Pruning of Transformer Heads
Li, J., Cotterell, R., & Sachan, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00436 · Cited by 72
“In practice, each attention head works independently, which allows the heads to capture different kinds of linguistic phenomena (Clark et al., 2019; Goldberg, 2019; Ettinger, 2020; Jawahar et al., 2019).”
BackgroundMuLan: A Joint Embedding of Music Audio and Natural Language
Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. W. (2022) · arXiv · doi:10.48550/arxiv.2208.12415 · Cited by 49
“This is a result of the text encoder not adequarely modeling the meaning of these negated concepts, which is a well known problem with BERT [43, 44] (the text embedding of “not rock” is similar to “rock” and performance suffers).”
BackgroundProbing for the Usage of Grammatical Number
Lasri, K., Pimentel, T., Lenci, A., Poibeau, T., & Cotterell, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.603 · Cited by 89
“Most of this work, however, focuses on diagnostic probes (Klafka and Ettinger, 2020; Tor-roba Hennigen et al., 2020).”
BackgroundGubelmann, R. (2023) · Grazer Philosophische Studien · doi:10.1163/18756735-00000182 · Cited by 54
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
Kassner, N., Tafjord, O., Schütze, H., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.697 · Cited by 88
“, 2020), they can be inconsistent in their answers to probing questions (Ettinger, 2020; Davison et al., 2019; Ravichander et al., 2020; Elazar et al., 2021; Subramanian et al., 2020), making their “world model” unclear.”
BackgroundEmbeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning
Pilehvar, M. T., & Camacho-Collados, J. (2020) · Synthesis lectures on human language technologies · doi:10.2200/s01057ed1v01y202009hlt047 · Cited by 88
Do Neural Language Models Overcome Reporting Bias?
Shwartz, V., & Choi, Y. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.605 · Cited by 37
“On the other hand, Logan et al. (2019) have shown that LMs are limited in their ability to generate accurate factual knowledge, and Kassner and Schütze (2020) and Ettinger (2020) pointed out that LMs are not sensitive to negation, resulting in generating incorrect facts (“birds can’t fly”).”
BackgroundUsing Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks
Xia, T., Wang, Y., Tian, Y., & Chang, Y. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449988 · Cited by 64
“These include analyses of layer-wise attentions, syntactic knowledge, semantic knowledge, and world knowledge learned at each layer [8, 12, 16, 37].”
BackgroundWord Acquisition in Neural Language Models
Chang, T. A., & Bergen, B. K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00444 · Cited by 26
“, 2020); in particular, behavioral approaches from psycholinguistics and cognitive science have been applied to study language model predictions (Futrell et al., 2019; Ettinger, 2020).”
BackgroundMethodComputational Language Modeling and the Promise of In Silico Experimentation
Jain, S., Vo, V. A., Wehbe, L., & Huth, A. G. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00101 · Cited by 71
“Furthermore, while ANNs like language models have been successfully employed for a wide range of tasks, their syntactic, common sense, and logical reasoning abilities are still far from those of humans (Ettinger, 2020; Linzen, 2020; Pandia & Ettinger, 2021; Wilcox et al., 2021).”
BackgroundMethodTransformer networks of human conceptual knowledge.
Bhatia, S., & Richie, R. (2024) · Psychological Review · doi:10.1037/rev0000319 · Cited by 59
A Systematic Analysis of Morphological Content in BERT Models for Multiple LanguagesPreprint
Edmiston, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03032 · Cited by 48
“, 2020), including subject-verb agreement (Goldberg, 2019), and also various semantic aspects such as semantic role and model predictions’ correlation with human judgment (Ettinger, 2020).”
BackgroundElazar, Y., Zhang, H., Goldberg, Y., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.819 · Cited by 61
“Then, we mask this word, and compare the probabilities of the two candidates, as was done in previous work (Goldberg, 2019; Talmor et al., 2020; Ettinger, 2020).”
ResultEmergence of relational reasoning
Holyoak, K. J., & Lu, H. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.11.012 · Cited by 54
Chess as a Testbed for Language Model State Tracking
Toshniwal, S., Wiseman, S., Livescu, K., & Gimpel, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21390 · Cited by 9
“…tasks range from testing general text understanding (Paperno et al. 2016) to targeting particular aspects of natural language, such as commonsense/pragmatics (Mostafazadeh et al. 2016; Ettinger 2020), narrative understanding (Mostafazadeh et al. 2017), and factual knowledge (Petroni et al. 2019).”
BackgroundOn the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 97
Probing for idiomaticity in vector space models
García, M., Vieira, T. K., Scarton, C., Idiart, M., & Villavicencio, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.310 · Cited by 42
“Regarding semantic knowledge, the results of various experiments suggest that BERT can somewhat represent semantic roles (Ettinger, 2020).”
ResultOn Attention Redundancy: A Comprehensive Study
Bian, Y., Huang, J., Cai, X., Yuan, J., & Church, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.72 · Cited by 68
“BERT does not excel at numbers, negation, inferences and role-based event prediction (Wallace et al., 2019; Ettinger, 2020).”
BackgroundSERENGETI: Massively Multilingual Language Models for Africa
Adebara, I., Elmadany, A., Abdul-Mageed, M., & Inciarte, A. A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.97 · Cited by 71
Klafka, J., & Ettinger, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.434 · Cited by 54
Self-citationSartran, L., Barrett, S., Kuncoro, A., Stanojević, M., Blunsom, P., & Dyer, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00526 · Cited by 78
“, 2020), long-tail syntactic phenomena continue to pose a problem, and may be implicated in systematic semantic failures (Ettinger, 2020).”
Background
Metadata
- Authors
- Ettinger, A.
- Affiliations
- University of Chicago (US)
- Corresponding author
- Ettinger, Allyson
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 34-48
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2965647350
- Cited by
- 511
- References
- 38
- Citations by year
- 2026: 16 · 2025: 32 · 2024: 39 · 2023: 122 · 2022: 92 · 2021: 151 · 2020: 53
- FWCI
- 41.25
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- BERT, language model pretraining, role reversal, negation, event prediction, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.