A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 901–1,000 of 1,014
Tong, J., Wang, S., Wang, G., Wang, Y., & Moraros, J. (2026) · npj Urban Sustainability · doi:10.1038/s42949-026-00377-2 · Cited by 6
“This BERT-based approach follows established practices in natural language processing 25 , with similar methods recently applied to the classi fi cation of urban sustainability literature, achieving comparable accuracy levels 26 .”
Islam, M. S., Xiangdong, L., & Ahmed, J. (2026) · Journal of Electrical Systems and Information Technology · doi:10.1186/s43067-026-00341-1 · Cited by 3
“…and addresses a critical limitation of earlier models like ELMo and GPT [5], BERT, introduced in 2018, uses a bidirectional trans-former design that captures contextual relationships in both directions from previous and subsequent words and allows it to process text in a pre-training manner [6].”
A decade of comparing brains to DNNs: Progress and perspectives
Schmitt, L., Dong, D. T., Çelik, E., de Lange, F. P., & Toneva, M. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.06.011 · Cited by 3
How to find similar companies using websites?
Bergmann, J.-P., Amin, M., Campbell, Y., & Trela, K. (2023) · World Patent Information · doi:10.1016/j.wpi.2023.102172 · Cited by 3
TamilCogniBERT: Enhancing Tamil Language Comprehension using Self Learning
Ashwinraj, G., Sarfraz Hussain, M., & Madhu Perkin, T. (2024) · International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS) · doi:10.1109/adics58448.2024.10533623 · Cited by 3
Data augmentation using instruction-tuned models improves emotion analysis in tweets
Yandrapati, P. B., & Eswari, R. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01325-7 · Cited by 4
“Large language models, such as GPT (Radford et al. 2018) and BERT (Devlin et al. 2019b), have revolutionized the field of natural language processing, demonstrating state-of-the-art performance on a wide range of tasks (Rogers et al. 2020; Brown et al. 2020).”
FLAT: Fusing layer representations for more efficient transfer learning in NLP
Zhu, W., Li, Y., Liu, C., & Zhao, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106631 · Cited by 4
A Comprehensive Examination of ChatGPT's Contribution to the Healthcare Sector and Hepatology
Kumari, K., Pahuja, S. K., & Kumar, K. S. (2024) · Digestive Diseases and Sciences · doi:10.1007/s10620-024-08659-4 · Cited by 3
“The transformer model uses a self-attention mechanism to process input sequences of variable length, making it well-suited for NLP tasks [39].”
Kashi, M., Lahmiri, S., & Mohamed, O. A. (2025) · Expert systems with applications · doi:10.1016/j.eswa.2025.126785 · Cited by 7
Instruction Semantics Enhanced Dual-Flow Graph Model for GPU Error Resilience Prediction
Yu, P., Gu, J., Shen, D., Dong, X., Liu, Y., & Xiong, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709424 · Cited by 3
Classifying Genetic Essentialist Biases using Large Language Models
Reimann, R., Lynch, K. E., Gawronski, S. A., Chan, J., & Griffiths, P. E. (2025) · Review of Philosophy and Psychology · doi:10.1007/s13164-025-00770-3 · Cited by 3
Yu, C.-W. F., Chuang, Y., Lotsos, A. N., Meier, T., & Haase, C. M. (2025) · Journal of Language and Social Psychology · doi:10.1177/0261927x251343096 · Cited by 5
“We chose the Sentence BERT (SBERT) as an alternative model to derive turn embeddings, which was based on the BERT model (Devlin et al., 2019; Reimers & Gurevych, 2019), a model familiar to many researchers, including psychologists (Biggiogera et al., 2021; Kjell et al., 2023; Rogers et al., 2020).”
Hui, Y., Liu, J., Zhang, Q., Zhou, T., & Song, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130493 · Cited by 6
Xiong, G., Liu, S., Yan, Y., Li, Q., & Li, H. (2025) · IEEE Access · doi:10.1109/access.2025.3580641 · Cited by 7
Barchane, M., & Zahour, O. (2025) · Mathematical Modeling and Computing · doi:10.23939/mmc2025.04.1109 · Cited by 3
Basic Attention Head as a Building Block toward Understanding Transformer-based Generative AI
Restrepo, D. J., Huo, F. Y., Restrepo, N. J., & Johnson, N. F. (2025) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2025.54251 · Cited by 3
“1 (e.g. for a few key Attention heads across a few key layers [12, 13]) could in principle be derived that can describe richer AI output dynamics observed empirically in LLMs [4, 25].”
Wang, Y., Sun, G., Fu, Z., & Liang, R. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50356-6 · Cited by 4
“Papers centered on the model primarily emphasize the “ model details and principles ” [22–24] and “ model application scenarios ” [25–27], while those centered on the user focus on the “ human-in-the-loop workflow and interactivity ” [28,29] and the “ usage patterns across di ff erent user groups ”…”
Post-Editing Extractive Summaries by Definiteness Prediction
Kabbara, J., & Cheung, J. C. K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.312 · Cited by 5
“, 2019) neural model which has shown strong performance across a wide range of NLP tasks (Rogers et al., 2020).”
BackgroundMethodWei, W., Ju, T., Liao, H., Zhao, W., & Gu, H. (2021) · Asia-Pacific Workshop on Networking (APNet) · doi:10.1145/3469393.3469394 · Cited by 4
“Our encoder is a stack of Transformer encoder layers [18] which consists of multiple self-attention "heads" [14].”
MethodProbe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction
Aoyama, T., & Schneider, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.25 · Cited by 11
“With an increasing number of studies employing probing classifiers, in their comprehensive review of BERTology, Rogers et al. (2020) raise a warning that such probing may not provide us with a full picture of what BERT is: "If a more complex probe recovers more information, to what extent are we still relying on the o…”
BackgroundMethodInfluentialGarden Path Traversal in GPT-2
Jurayj, W., Rudman, W., & Eickhoff, C. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.25 · Cited by 5
“Kovaleva et al., 2019; Tenney et al., 2019a; Rogers et al., 2020).”
BackgroundMemory-efficient Temporal Moment Localization in Long Videos
Rodríguez-Opazo, C., Marrese-Taylor, E., Fernando, B., Takamura, H., & Wu, Q. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.140 · Cited by 3
“…interfering with the inductive bias that the baselines require to process the multi-modal inputs, as well as with the already acquired bias in the case of BERT, reflected in certain attention patterns for each head which have been studied and documented by Rogers et al. (2020) among others.”
BackgroundMeaning creation in novel noun-noun compounds: humans and language models
Chen, P., Poeppel, D., & Zuanazzi, A. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2254865 · Cited by 3
“…parallels to human behaviour in a variety of language tasks (Ettinger, 2020; Hofmann et al., 2022; Lin et al., 2019; Radford et al., 2019; Rogers et al., 2020; Warstadt et al., 2020; Warstadt & Bowman, 2022), however the mechanisms through which they “ interpret ” novel NNCs remain to be…”
BackgroundMethodEnsemble Approaches for Offensive Language Detection in Kannada
Monish, M., & Sooda, K. (2024) · International Conference for Innovation in Technology (INOCON) · doi:10.1109/inocon60754.2024.10511853 · Cited by 3
“Data augmentation is not restricted to NLP; it finds applications in areas like acoustic signal processing [5] and derogatory content identification [6].”
BackgroundZhu, E., Yu, Z., Huang, Y., Gao, S., & Xian, Y. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02367-1 · Cited by 3
“Fortunately, recent investigations have demonstrated that pre-trained language models implicitly capture a significant amount of linguistic knowledge from text corpora in an unsupervised manner [24–26].”
BackgroundAspect-based sentiment analysis of open-ended responses in student course evaluation surveys
Alotaibi, R. M. (2024) · Thermal Science · doi:10.2298/tsci2406037a · Cited by 5
Harrag, C., Sabil, A., Radvansky, G. A., & Conceição, M. C. (2025) · Discourse Processes · doi:10.1080/0163853x.2025.2494964 · Cited by 4
Kissane, H., Schilling, A., & Krauß, P. (2026) · Language Cognition and Neuroscience · doi:10.1080/23273798.2026.2648588 · Cited by 6
Braun, N., Goudbeek, M. B., & Krahmer, E. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3005613 · Cited by 5
Mosbach, M., Degaetano‐Ortlieb, S., Krielke, M.-P., Abdullah, B. M., & Klakow, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.67 · Cited by 7
SLICE: Supersense-based Lightweight Interpretable Contextual Embeddings
Aloui, C., Ramisch, C., Nasr, A., & Barque, L. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.298 · Cited by 7
“On the downside, they are computationally heavy and opaque (Rogers et al., 2020), requiring sophisticated techniques such as probing to interpret predictions (Jawahar et al., 2019).”
BackgroundKawabe, H., Shimomura, Y., & Seto, S. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-79203-9_32 · Cited by 3
TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021
Wiriyathammabhum, P. (2021) · Workshop on Understanding Implicit and Underspecified Language (UnImplicit) · doi:10.18653/v1/2021.unimplicit-1.8 · Cited by 3
“Many recent studies (Clark et al., 2019; Hewitt and Manning, 2019; Rogers et al., 2020) also show that BERT learns some syntactic information during its pretraining steps.”
BackgroundComparing Text Representations: A Theory-Driven Approach
Yauney, G., & Mimno, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.449 · Cited by 10
Modelling Symbolic Knowledge Using Neural Representations
Schockaert, S., & Gutiérrez-Basulto, V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-95481-9_3 · Cited by 4
“Despite the lack of any explicit supervision signal, the resulting LMs have been found to capture a wealth of syntactic and semantic knowledge [25].”
BackgroundTowards Automatic Classification of Design Decisions from Developer Conversations
Josephs, A., Gilson, F., & Galster, M. (2022) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c54293.2022.00009 · Cited by 7
A Methodology for Enabling NLP Capabilities on Edge and Low-Resource Devices
Goulas, A., Malamas, N., & Symeonidis, A. L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_18 · Cited by 4
Learning Sentence-Level Representations with Predictive Coding
Araujo, V., Moens, M., & Soto, Á. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010005 · Cited by 10
An information theoretic score for learning hierarchical concepts
Madani, O. (2023) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2023.1082502 · Cited by 4
“Transformers are the state of the art, and the progress in language models based on neural networks has substantially expanded the diversity of the applications of SLM (Dong et al., 2019; Brown et al., 2020; Rogers et al., 2020).”
BackgroundLogic-Based Inference With Phrase Abduction Using Vision-and-Language Models
Tomihari, A., & Yanaka, H. (2023) · IEEE Access · doi:10.1109/access.2023.3274489 · Cited by 3
“Previous studies have shown that norms of word vectors represent the relative informativeness of words [29], [30], [31], [32].”
BackgroundKnowledge Acquired by Foundation Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_4 · Cited by 3
Applying Natural Language Processing to Assessment
Liu, M. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197611050.003.0005 · Cited by 3
NeuroX Library for Neuron Analysis of Deep NLP Models
Dalvi, F., Sajjad, H., & Durrani, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.21 · Cited by 15
Guerin, N., & Chemla, E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.560 · Cited by 4
“NLP models have been tested for their syntactic abilities (Rogers et al., 2020; Lin et al., 2019; Wu et al., 2020; Goldberg, 2019; Warstadt et al., 2019; Jumelet and Hupkes, 2018; Marvin and Linzen, 2018) for which they obtain strong results, but to a lesser extent for their semantic abilities…”
BackgroundProbing with Noise: Unpicking the Warp and Weft of Embeddings
Klubička, F., & Kelleher, J. D. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.34 · Cited by 6
Ferraria, M. A., Ferraria, V. A., & de Castro, L. N. (2023) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2023.08.006 · Cited by 3
“SBERT is a sentence embedding representation model built on top of BERT [56] and RoBERTa models [57].”
MethodPerceptional and actional enrichment for metaphor detection with sensorimotor norms
Wan, M., Su, Q., Ahrens, K., & Huang, C. (2023) · Natural Language Engineering · doi:10.1017/s135132492300044x · Cited by 13
“…recent studies, people aim to adopt alternative techniques to compress BERT-based models for more reasonable production environments using methods such as weight pruning, matrix factorization, and knowledge distillation (Zafrir et al. 2019; Rogers, Kovaleva, and Rumshisky 2020; Zafrir et al. 2021).”
Emotional Analysis Based On LSTM-CNN Hybrid Neural Network Model
Wu, Y., Al-Daweri, M. S., & Kumar, V. D. (2023) · International Conference on Software Engineering and Computer Systems · doi:10.1109/icsecs58457.2023.10256408 · Cited by 3
Filling in the white space: Spatial interpolation with Gaussian processes and social media data
Giorgi, S., Eichstaedt, J. C., Preoţiuc-Pietro, D., Gardner, J. R., Schwartz, H. A., & Ungar, L. (2023) · Current Research in Ecological and Social Psychology · doi:10.1016/j.cresp.2023.100159 · Cited by 3
“While contextual embeddings have become standard across many computational tasks (Rogers et al., 2021), they have yet to be evaluated in the context of spatial language or community-level prediction tasks.”
BackgroundAn Attribution Method for Siamese Encoders
Moeller, L., Nikolaev, D., & Padó, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.980 · Cited by 15
INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models
Renduchintala, H. S. V. N. S. K., Killamsetty, K., Bhatia, S., Aggarwal, M., Ramakrishnan, G., Iyer, R., & Krishnamurthy, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.445 · Cited by 7
“We ex-Algorithm et al., 2020).”
BackgroundGessler, L., & Schneider, N. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.17 · Cited by 9
Modelling language using large language models
Grindrod, J. (2026) · Philosophical Studies · doi:10.1007/s11098-026-02479-0 · Cited by 7
“Regarding syntactic relations more generally, Rogers, Kovaleva, and Rumshisky (2021, 847) note that there is a consensus that in BERT models, self-attention heads are sensitive to syntactic features in the earlier layers of the model.”
Thai, T. M., & Luu, S. T. (2024) · Journal of Computer Science and Cybernetics · doi:10.15625/1813-9663/18155 · Cited by 3
“Taking advantage of natural language processing, we have several robust models for sequence-to-sequence learning tasks such as Long-short Term Memory [3], Convolutional Neural Networks for Sequence-to-sequence [9], Transformer [30] and BERTology [25].”
MethodHow well do contextual protein encodings learn structure, function, and evolutionary context?
(2025) · doi:10.1016/j.cels.2025.101201 · Cited by 3
Recurrent Neural Language Models as Probabilistic Finite-state Automata
Svete, A., & Cotterell, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.502 · Cited by 22
Bai, Y., Zhao, L., Wang, Z., Chen, J., & Lian, P. (2020) · IEEE Access · doi:10.1109/access.2020.3014185 · Cited by 4
“This approach allows for increased parallel computation and reduces the time to convergence and achieves state-of-the-art results on several natural language processing tasks [58]–[61].”
BackgroundInterpreting BERT-based stance classification: a case study about the Brazilian COVID vaccination
Sáenz, C. A. C., & Becker, K. E. (2021) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2021.17867 · Cited by 8
“However, this is still a subject of debate that deserves further investigation [Rogers et al. 2020].”
BackgroundMethodExploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering
Kim, J.-H., Kang, J., Kim, K.-M., Hong, G., & Myaeng, S.-H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.138 · Cited by 3
“This result is also in correspondence with previous works (Rogers et al., 2020; Jo and Myaeng, 2020), where the roles of heads and layers are already defined during the pre-training of the model.”
ResultFIND: Human-in-the-Loop Debugging Deep Text ClassifiersPreprint
Lertvittayakumjorn, P., Specia, L., & Toni, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04987 · Cited by 16
“…recent papers studied several types of knowledge in BERT (Devlin et al., 2019), a deep transformer-based model for language understanding, and found that syntactic information is mostly captured in the middle BERT layers while the final BERT layers are the most task-specific (Rogers et al., 2020).”
BackgroundHow Does Adversarial Fine-Tuning Benefit BERT?Preprint
Ebrahimi, J., Yang, H., & Zhang, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13602 · Cited by 6
“One class of interpretability research focuses on the direct examination of LMs, regarding what and how they learn (Rogers et al., 2020).”
BackgroundFine-Tuned Transformers Show Clusters of Similar Representations Across LayersPreprint
Phang, J., Liu, H., & Bowman, S. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08406 · Cited by 11
Proceedings of the Fifth Workshop on Teaching NLP
(2021) · doi:10.18653/v1/2021.teachingnlp-1 · Cited by 3
Improving neural machine translation of languages with little data and rich morphology
Dhar, P. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.859674171 · Cited by 3
Kewenig, V. N., Lampinen, A. K., Nastase, S. A., Edwards, C., D'Elascombe, Q., Rechardt, A., Skipper, J. I., & Vigliocco, G. (2024) · Research Square · doi:10.21203/rs.3.rs-3913308/v1 · Cited by 6
Müller-Eberstein, M., van der Goot, R., Plank, B., & Titov, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.879 · Cited by 30
Analyzing the Robustness of Vision & Language Models
Shirnin, A., Andreev, N., Potapova, S., & Artemova, E. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3399061 · Cited by 10
Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 8
Gorshkov, S., Ignatov, D. I., Chernysheva, A. Y., Goiko, V. L., & Kashpur, V. V. (2024) · IEEE Access · doi:10.1109/access.2024.3521857 · Cited by 2
Machine Learning for Econometrics
Gaillac, C., & L’Hour, J. (2025) · doi:10.1093/9780198918868.001.0001 · Cited by 2
Harinath, K. R., & Kumar, G. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1377 · Cited by 3
“The introduction of the BERT model by the Google AI team marked a significant milestone in the field of Natural Language Processing (NLP), propelling NLP research forward dramatically [41].”
InfluentialKAFA-Merge: Model Merging Method With Layer-Based Bayes+Linear Search
Yuce, M. K., & Amasyalı, M. F. (2025) · IEEE Access · doi:10.1109/access.2025.3582015 · Cited by 4
Chen, M., Wang, J., Zhao, F., & Jiang, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-16153-8 · Cited by 4
“However, its sensitivity to local sequence features and reliance on positional encoding may leave room for optimization when handling fine-grained sequence labeling tasks 36 .”
LLM Based Sentiment Analysis in Private Social Media: Case Study of My Tel-U App
Alfiansyah, F., Purbolaksono, M. D., & Gozali, A. A. (2025) · International Conference on Data Science and Its Applications (ICoDSA) · doi:10.1109/icodsa67155.2025.11157236 · Cited by 2
“This comparison underscores the limitations of traditional models in handling complex language tasks and highlights the advantage of transformer-based models for context-sensitive NLP applications [23], [24].”
Queueing for Civility: User Perspectives on Regulating Emotions in Online Conversations
Verma, A., Islam, S. N., Moghaddam, V., & Anwar, A. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2561180 · Cited by 2
“This approach allowed us to assess the emotional composition of the conversations effectively.”
Aparecido, T. D. D., Carrillo, A., Camargo, C. Q., & Stella, M. (2025) · AI · doi:10.3390/ai6100249 · Cited by 5
“Moreover, lexicon-based methods can be adapted or tuned for specific domains or topics of interest [22].”
LLM-led vision-spectral fusion: A zero-shot approach to temporal fruit image classification
Wu, H., Jia, B., & Yuan, X. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108155 · Cited by 2
Zhao, J., Chen, Y., Liu, J., & Salvadeo, P. (2025) · Land · doi:10.3390/land14102035 · Cited by 2
Prognostics of Turbofan Engines Using Self-Attention Transformers and Explainable AI Techniques
Prete, A. D., Cirillo, E., Mashaallah, Z., & Moccardi, A. (2025) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-10344-4_20 · Cited by 2
Learning mechanism underlying NLP pre-training and fine-tuning
Tzach, Y., Gross, R. D., Koresh, E., Rosner, S., Shpringer, O., Halevi, T., & Kanter, I. (2025) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2025.131113 · Cited by 6
Alhmoudi, O. K., Thameem, M., Elkamel, A., & AlHammadi, A. A. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.108390 · Cited by 2
Pütz, O. (2025) · Journal of Linguistic Anthropology · doi:10.1111/jola.70036 · Cited by 2
García, F. P., Cánovas, Ó., Clemente, F. J. G., & Lucas, A. O. (2025) · IEEE Access · doi:10.1109/access.2025.3641484 · Cited by 2
“Modern Natural Language Processing (NLP) models, particularly those based on the Transformer architecture like Bidirectional Encoder Rep-resentations from Transformers (BERT) [5], have proven indispensable for this task, setting new benchmarks in language understanding [21].”
Eckert, M., Bonial, C., Bonn, J., Hwang, J. D., & Madabushi, H. T. (2025) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-041824-061723 · Cited by 2
Kou, J., Lu, X., Yin, A., Zhang, W., Fang, X., Li, Y., Pan, T., Zhang, L., Cheng, G., Yuan, H., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189154 · Cited by 7
Savostin, A., Каипбек, Г., Koshekov, K., Savostina, G., & Wardle, K. (2025) · Naukovyi Visnyk Natsionalnoho Hirnychoho Universytetu · doi:10.33271/nvngu/2025-6/157 · Cited by 2
Multi-Hazard assessment methods for planning: A review and potential future directions
Joungwon, K. W. O. N., Yu, S., Malecha, M. L., & Berke, P. R. (2026) · Sustainable Cities and Society · doi:10.1016/j.scs.2026.107136 · Cited by 4
Raghupathi, W. (2026) · Algorithms · doi:10.3390/a19010063 · Cited by 2
Chinese Sentiment Analysis and Interpretability Research Based on BERT and its Improved Models
Xie, F., Luo, H., Chen, Y., Qiu, K., Yu, S., & Zhong, H. (2025) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic68075.2025.11324842 · Cited by 2
“Using BertViz [7], [8], we analyze attention patterns on mixed-sentiment reviews (e.g., “good service but too expen-sive”).”
Schütz, M., Losacker, S., & Hennemann, S. (2026) · Global challenges & regional science. · doi:10.1016/j.gcrs.2026.100028 · Cited by 2
SymLoc: Symbolic Localization of Hallucination across HaluEval and TruthfulQA
Lamba, N., Tiwari, S., & Gaur, M. (2025) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3799830.3799850 · Cited by 2
HiMix : Hierarchical Visual-Textual Mixing Network for Lesion Segmentation
Hwang, S., Sim, J., & Kim, W. H. (2026) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00517 · Cited by 3
Online brand advocacy and video game consumers
Pourazad, N., Abedin, E., & Burgess, J. (2026) · Journal of Product & Brand Management · doi:10.1108/jpbm-03-2025-5866 · Cited by 2
Luo, A., Phucharoen, C., & Sangkaew, N. (2026) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2026.2652041 · Cited by 2
“Following the standard paradigm for fine-tuning pre-trained language models (Rogers et al., 2020), we re-evaluated its performance on the official validation set of SemEval-2018 (Barbieri et al., 2018).”
Hartl, B., Pio‐Lopez, L., Fields, C., & Levin, M. G. (2026) · Physics of Life Reviews · doi:10.1016/j.plrev.2026.06.009 · Cited by 6
Piercing a Methodological Bubble: Large Language Models and the Social Sciences
Woods, D. (2026) · Fudan Journal of the Humanities and Social Sciences · doi:10.1007/s40647-026-00473-8 · Cited by 2
NucleicBERT interprets RNA sequence space through self-supervised language modelling
Upadhyay, U., Herold, J. M., Götz, M., & Schug, A. H. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01295-9 · Cited by 3
Probing Semantic Grounding in Language Models of Code with Representational Similarity Analysis
Naik, S., Patil, R., Agarwal, S., & Baths, V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22137-8_29 · Cited by 2
Nguyen, N. H., Vo, D. T. D., & Nguyen, K. V. (2022) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf55975.2022.10013898 · Cited by 2
“Moreover, we will analyze and research the way to effectively combine the handwriting recognition methods with pre-trained language models (like BERTology [20]) to make them infer better when making predictions for scribbled handwriting images.”
Method
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.