A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 801–900 of 1,014
How Language Models Reshape Sponsored Search: Evidence from Google’s BERT RolloutPreprint
Larsen, P., & Proserpio, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4614402 · Cited by 6
Interpreting Attention Mechanisms in Genomic Transformer Models: A Framework for Biological InsightsPreprint
Consens, M. E., Díaz‐Navarro, A., Chu, V., Stein, L. D., He, H. H., Moses, A. M., & Wang, B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.26.661544 · Cited by 8
The King Is Naked: On the Notion of Robustness for Natural Language Processing
Malfa, E. L., & Kwiatkowska, M. (2022) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v36i10.21353 · Cited by 16
Alves, L. F., Vasconcellos, F. J. S., & Nogueira, B. M. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10084-4 · Cited by 4
“As in Data Mining, the main Pattern Extraction activities are classified as predictive and descriptive (Rezende 2003).”
MethodInfluentialChen, J., Guo, J., Li, X., Wang, H., Xu, Z., Gong, Z., Zhang, L., & Leung, V. C. M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3452695 · Cited by 4
“It is a relatively new area of research to apply Transform-ers [4], [10] and GNN [2] for analyzing short texts, despite their groundbreaking achievements in natural language processing (NLP) and recommender systems.”
Practical Techniques for Theorizing from Literature Reviews
de Guinea, A. O., & Watson, R. T. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00900-8 · Cited by 4
“Altogether, BERT performs remarkably well due to its encoder-only transformer architecture, which however makes the reasons for its success unclear (Rogers et al. 2020).”
Constructing understanding: on the constructional information encoded in large language models
Bonial, C., & Madabushi, H. T. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09799-9 · Cited by 16
Lau, J. C. Y., Landau, E. B., Zeng, Q., Zhang, R., Crawford, S., Voigt, R., & Losh, M. C. (2024) · Autism · doi:10.1177/13623613241304488 · Cited by 4
“Pre-trained on billions of language tokens, PLMs generate rich semantic and pragmatic representations of their inputs required to solve complex social and pragmatic prediction tasks (Manning et al., 2020; Rogers et al., 2020; Tenney et al., 2019).”
Semantic-Weighted Word Error Rate Based on BERT for Evaluating Automatic Speech Recognition Models
Zhang, Y., & Ai, J. (2024) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa63982.2024.00034 · Cited by 4
“When text is input into the BERT model, it combines the base semantic vectors of tokens with contextual semantic representations to ultimately derive specific textual vectors for tokens [17] .”
Named Entity Recognition Method Based on Multi-Feature Fusion
Huang, W., & Yu, X. (2025) · Applied Sciences · doi:10.3390/app15010388 · Cited by 4
Integrating Language Models for Enhanced Network State Monitoring in DRL-Based SFC Provisioning
Moshiri, P. F., Önsü, M. A., Lohan, P., Kantarcı, B., & Janulewicz, E. (2025) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc65549.2025.11324453 · Cited by 7
“BERT and DistilBERT are pre-trained language models that are primarily intended for natural language interpretation rather than mathematical computations or real-time state-based decision-making, hence they cannot do numerical calculations [18].”
Where Does mBERT Understand Code-Mixing? Layer-Dependent Performance on Semantic Tasks
Somani, A., Kumar, S. R. M., & Malapati, A. (2025) · IEEE Access · doi:10.1109/access.2025.3594135 · Cited by 5
“To better understand the inner workings of large pre-trained models, the field of ‘‘BERTology’’ has emerged, which heavily relies on probing techniques [11].”
A Novel Approach for Mitigating Class Imbalance in Arabic Text Classification
Nabil, E., Nagib, A. E., Hany, M., Faizullah, S., & Gomaa, W. H. (2025) · IEEE Access · doi:10.1109/access.2025.3604427 · Cited by 4
“Originally trained on big text corpora and refined for certain tasks, BERT and its variants (e.g., RoBERTa, XLM-R) have produced state-of-the-art results in applications including question answering, text summarizing, and machine translation, so considerably improving NLP capabilities [34].”
Mischler, G., Li, Y. A., Bickel, S., Mehta, A. D., & Mesgarani, N. (2025) · Nature Communications · doi:10.1038/s41467-025-63961-7 · Cited by 5
“We appreciate the reviewer’s previous suggestions, which improved the strength of our main findings, and we hope to share our future work investigating a range of music models with the reviewer.”
Ren, F., Li, R., & Wang, Q. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.116583 · Cited by 5
Custom GPTs to aid in compliance checking for reporting standards in academic publishing
Mohapatra, R., & Härtung, T. (2026) · ALTEX · doi:10.14573/altex.2601011 · Cited by 5
BiLSTM-LIME: integrating NLP and advanced machine learning models for fake news detection
Sneha, S. G., Sen, A., Malik, S., Uddin, A., & Hossen, M. J. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-00852-w · Cited by 5
“Transformer-based architectures such as BERT [35], RoBERTa [36], and XLNet [37] have demonstrated state-of-the-art performance in natural language understanding tasks; however, these gains come at the cost of high computational and memory overhead.”
An overview of artificial intelligence approaches for automating evidence synthesis
Harasgama, S., Pearce, H., Loftus, L., Painter, H., & Ford, J. A. (2026) · Public Health · doi:10.1016/j.puhe.2026.106220 · Cited by 5
Jiang, H., & Liu, Y. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102216 · Cited by 6
Chandana, N., Rao, S. N., Kumaraswamy, M., Pallakonda, A., & Raj, R. D. A. (2026) · Next Energy · doi:10.1016/j.nxener.2026.100586 · Cited by 6
Comprehensive study of pre-trained language models: detecting humor in news headlines
Shatnawi, F., Abdullah, M., Hammad, M., & Al‐Ayyoub, M. (2022) · Soft Computing · doi:10.1007/s00500-022-07573-z · Cited by 8
Zhang, S., Wang, W., Li, H., & Zhang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3584361 · Cited by 8
“The transformer is a novel NLP model presented by the Google team in 2017, and it is currently used by the more popular BERT [3, 16, 52].”
MethodTang, F.-S. K.-B., Bukowski, M., Schmitz‐Rode, T., & Farkas, R. (2023) · Applied Sciences · doi:10.3390/app13137639 · Cited by 5
An Extensive Study of the Structure Features in Transformer-based Code Semantic Summarization
Yang, K., Mao, X., Wang, S., Qin, Y., Zhang, T., Lu, Y., & Al‐Sabahi, K. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00024 · Cited by 4
“In the natural language processing (NLP) community, the research works that attempt to demonstrate what BERT [38] learn and how it affects downstream tasks by analyzing attention and task probing have formed a “BERTology” [39] subspecialty.”
BackgroundA transformer-based deep learning model for Persian moral sentiment analysis
Karami, B., Bakouie, F., & Gharibzadeh, S. (2023) · Journal of Information Science · doi:10.1177/01655515231188344 · Cited by 8
Memorization of Named Entities in Fine-Tuned BERT Models
Diera, A., Lell, N., Garifullina, A., & Scherp, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_16 · Cited by 6
“Since its original release, there have been dozens of follow-up studies and models published [43].”
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 6
“To reach a better understanding of those black box models, new methods have been developed [78].”
MethodIdioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space
Klubička, F., Nedumpozhimana, V., & Kelleher, J. D. (2023) · Workshop on Multiword Expressions (MWE) · doi:10.18653/v1/2023.mwe-1.8 · Cited by 4
Gao, Y., Fu, X., Liu, X., & Wu, J. Y. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01228-8 · Cited by 5
“All these unsupervised pre-trained models encode contextualized information with rich semantic and syntactic knowledge [26].”
Wijnholds, G., & Moortgat, M. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.11 · Cited by 7
Quantum Transfer Learning for Sentiment Analysis: an experiment on an Italian corpus
Buonaiuto, G., Guarasci, R., & Esposito, M. (2024) · QUASAR@HPDC · doi:10.1145/3660318.3660325 · Cited by 3
A comprehensive evaluation of semantic relation knowledge of pretrained language models and humans
Cao, Z., Yamada, H. S., Teufel, S., & Tokunaga, T. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09858-9 · Cited by 7
MCVE: multimodal claim verification and explanation framework for fact-checking system
Luu, S. T., Vo, T., & Nguyen, L.-M. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01804-7 · Cited by 5
“With the power of state-of-the-art transformer models for computer vision and natural language processing such as BERT [13], BERTol-ogy [29], Vision Transformer [15], and the early fusion are a potential approach because these models have a strong associate with the data source [7].”
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
Nikoulina, V., Tezekbayev, M., Kozhakhmet, N., Babazhanova, M., Gallé, M., & Assylbekov, Z. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12788 · Cited by 9
“This success has spurred an interest in trying to understand what type of knowledge these models actually learn (Rogers et al., 2020; Petroni et al., 2019a).”
BackgroundMartínez, H. V. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.38 · Cited by 9
Proxy Model Explanations for Time Series RNNs
Wood-Doughty, Z., Cachola, I. A., & Dredze, M. H. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00117 · Cited by 7
“This popular approach is representative of the common setting where an effective model exists but is difficult to interpret; for example, the model may be a state-of-the-art neural network with billions of parameters [10], [11], [24].”
BackgroundChen, X., Chen, F., Xu, S., & Xu, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746656 · Cited by 5
Assessing the use of attention weights to interpret BERT-based stance classification
Sáenz, C. A. C., & Becker, K. E. (2021) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3486622.3493966 · Cited by 6
Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task
Tikhonova, M., Mikhailov, V., Pisarevskaya, D., Malykh, V., & Shavrina, T. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000225 · Cited by 8
“A plethora of works is devoted to the linguistic analysis of BERT, and the inspection of how fine-tuning affects the model knowledge (Rogers et al., 2020).”
BackgroundDarkness can not drive out darkness: Investigating Bias in Hate SpeechDetection Models
Elsafoury, F. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.4 · Cited by 8
“I computed the Pearson’s correlation coefficient (PCC) between the mean attention weights of fine-tuned BERT of all heads across the last layers (9-12) and the tokens’ absolute importance score, as it has been shown that fine-tuning effects mostly BERT’s last layers (9-12) (Rogers et al., 2021).”
Introduction to the special section on the interaction between formal and computational linguistics
Bernard, T., & Winterstein, G. (2022) · Journal of Language Modelling · doi:10.15398/jlm.v10i1.325 · Cited by 4
“…projection (Jiang and de Marneffe 2019) or syntactic generalisations (Hu et al. 2020) (see also the domain of “BERTology”, which seeks to study the properties of the representations manipulated by BERT-like models, though not necessarily from a linguistic angle; Ettinger 2020; Rogers et al. 2020).”
BackgroundAlhamzeh, A., Egyed‐Zsigmond, E., Mekki, D. E., Khayari, A. E., Mitrović, J., Brunie, L., & Kosch, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-662-66146-8_5 · Cited by 4
Data protections for minors with named entity recognition
Gillette, J. B., Shah, S. K., Tariq, Z., & Algamdi, S. A. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021086 · Cited by 4
“…vector, plus a vector describing the token’s position within the input sequence, and a context embedding representing the surrounding tokens, are used to give each token and sub-token within a sequence a rich contextualized meaning that addresses both language ambiguity and synonymity [19].”
BackgroundHindi Chatbot for Supporting Maternal and Child Health Related Queries in Rural India
Mishra, R., Singh, S., Kaur, J., Singh, P., & Shah, R. R. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.9 · Cited by 4
“Since Rogers et al. (Rogers et al., 2020) suggested that early layers of encoders contain more syntactic information, we froze the early layers of the encoder .”
BackgroundIntroducing Semantics into Speech Encoders
Xu, D. C., Dong, S., Wang, C., Kim, S., Lin, Z., Liu, B., Shrivastava, A., Li, S.-W. D., Tseng, L.-H., Lin, G.-T., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.639 · Cited by 11
On the Transformation of Latent Space in Fine-Tuned NLP Models
Durrani, N., Sajjad, H., Dalvi, F., & Alam, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.97 · Cited by 20
Xie, S., Qiu, J., Pasad, A., Du, L., Qu, Q., & Mei, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.422 · Cited by 19
Predicting Retrieval Performance Changes in Evolving Evaluation Environments
El-Ebshihy, A., Fink, T., González-Sáez, G., Galuščáková, P., Piroi, F., Iommi, D., Goeuriot, L., & Mulhem, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42448-9_3 · Cited by 4
Explaining Interactions Between Text Spans
Choudhury, S., Atanasova, P., & Augenstein, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.783 · Cited by 14
“Attention heads should not be aggregated as they are designed to provide different views of the data and capture different semantic and syntactic relations (Rogers et al., 2020).”
BackgroundZhang, L., Wu, N., Han, S., Xiao, J., & Ruan, S. (2023) · IEEE International Conference on Energy Internet (ICEI) · doi:10.1109/icei60179.2023.00071 · Cited by 4
“In the first BiTCN layer, there are 10 filters, a 3-unit convolution kernel, and a dilation factor array of [1, 2, 4, 8, 16].”
BackgroundMethodInfluentialCSEPrompts: A Benchmark of Introductory Computer Science Prompts
Raihan, N., Goswami, D., Puspo, S. S. C., Newman, C. D., Ranasinghe, T., & Zampieri, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-62700-2_5 · Cited by 9
Lee, M. (2024) · Fractal and Fractional · doi:10.3390/fractalfract8100552 · Cited by 5
Controlling the Imprint of Passivization and Negation in Contextualized Representations
Çelikkanat, H., Virpioja, S., Tiedemann, J., & Apidianaki, M. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.13 · Cited by 10
“The bulk of this interpretation work relies on probing tasks which serve to predict linguistic properties from the representations generated by the models (Linzen, 2018; Rogers et al., 2020).”
BackgroundText classification by untrained sentence embeddings
Sarli, D. D., Gallicchio, C., & Micheli, A. (2021) · Intelligenza Artificiale · doi:10.3233/ia-200053 · Cited by 5
“In fact, these techniques (and their tendency to require hundreds of millions of parameters) can lead to a significant cost in terms of training time [39].”
Coloring the Black Box: What Synesthesia Tells Us about Character Embeddings
Kann, K., & Monsalve‐Mercado, M. M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.230 · Cited by 4
Exploring Unsupervised Pretraining Objectives for Machine Translation
Baziotis, C., Titov, I., Birch, A., & Haddow, B. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.261 · Cited by 4
Pappas, D., & Androutsopoulos, I. (2021) · doi:10.18653/v1/2021.acl-long.301 · Cited by 4
“Previous work has reported that representations from different BERT layers may be more appropriate for different tasks (Rogers et al., 2020).”
BackgroundAutomatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning
Sathe, A., & Park, J. (2021) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/2021.fever-1.11 · Cited by 7
A Vietnamese Answer Extraction Model Based on PhoBERT
Nguyen, C. T., & Nguyen, D. T. (2021) · International Conference on Advanced Computing and Applications (ACOMP) · doi:10.1109/acomp53746.2021.00022 · Cited by 4
An ensemble multilingual model for toxic comment classification
Xie, G. (2022) · International Conference on Algorithms, Microchips, and Network Applications (AMNA) · doi:10.1117/12.2636419 · Cited by 5
“Moreover, we translated the test data to English by Google translate API for our inference BERTology[10] models as well.”
MethodFrom BERT‘s Point of View: Revealing the Prevailing Contextual Differences
Schuster, C. M., & Hegelich, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.89 · Cited by 5
“In the field of BERTology (see Rogers et al., 2020 for a general overview) much research has focused on three components; the self-attention mechanism, a key component of the transformer architecture that provides intuitive explanations (see e.g. Koval-eva et al., 2019, Manning et al., 2020, Clark…”
BackgroundWhat do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured
Henlein, A., & Mehler, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.425 · Cited by 8
Biomedical Word Sense Disambiguation with Contextualized Representation Learning
Saeidi, M., Milios, E., & Zeh, N. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524703 · Cited by 6
Berdyugina, D., & Cavallucci, D. (2022) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-17288-5_8 · Cited by 5
An Analogy based Approach for Solving Target Sense Verification
Zervakis, G., Emmanuel Vincent, A., Couceiro, M., Schoenauer, M., & Marquer, E. (2022) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3582768.3582794 · Cited by 6
“Despite their great success in a plethora of downstream tasks, the ability of these models to perform reasoning is limited and understudied [23, 24].”
BackgroundRobust hierarchical model for joint span detection and aspect-based sentiment analysis in Vietnamese
Le, A. P., Pham, T. V., Le, T.-V., Huynh, D. V., & Ngo, D. (2022) · NAFOSTED Conference on Information and Computer Science (NICS) · doi:10.1109/nics56915.2022.10013463 · Cited by 4
A Latent-Variable Model for Intrinsic Probing
Stańczak, K., Hennigen, L. T., Williams, A., Cotterell, R., & Augenstein, I. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26593 · Cited by 10
Decouple knowledge from paramters for plug-and-play language modeling
Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.901 · Cited by 21
The Architectural Bottleneck Principle
Pimentel, T., Valvoda, J., Stoehr, N., & Cotterell, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.788 · Cited by 10
PromptST: Abstract Prompt Learning for End-to-End Speech Translation
Yu, T., Ding, L., Liu, X., Chen, K., Zhang, M., Tao, D., & Zhang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.627 · Cited by 8
Glodd, A., & Hristova, D. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.679 · Cited by 4
Decomposed Meta-Learning for Few-Shot Sequence Labeling
Ma, T., Wu, Q., Jiang, H., Lin, J., Karlsson, B. F., Zhao, T., & Lin, C.-Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3372879 · Cited by 5
“To address this issue, we plug two groups of task-specific adapters into the upper layers of the shared BERT model, assuming that the lower layers are more task-agnostic [14] and can be fully shared between the two subtasks.”
MethodManual Clustering and Spatial Arrangement of Verbs for Multilingual Evaluation and Typology Analysis
Majewska, O., Vulić, I., McCarthy, D., & Korhonen, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.423 · Cited by 5
“…twofold: (i) the importance of accurate and nuanced representation of verb meaning in light of their pivotal role in sentence structure and the still subpar verbal reasoning ability of SOTA models (Rogers et al., 2020), and (ii) the scarcity of verb data in evaluation datasets currently available.”
BackgroundBERT in Negotiations: Early Prediction of Buyer-Seller Negotiation OutcomesPreprint
Chawla, K., Lucas, G., Gratch, J. S., & May, J. (2020) · arXiv (Cornell University) · Cited by 4
Demystifying Neural Language Models' Insensitivity to Word-OrderPreprint
Clouâtre, L., Parthasarathi, P., Zouaq, A., & Chandar, S. (2021) · arXiv (Cornell University) · Cited by 4
Tell me how you write and I'll tell you what you read: a study on the writing style of book reviews
Alzetta, C., Dell’Orletta⋄, F., Miaschi, A., Prat, E., & Venturi⋄, G. (2023) · Journal of Documentation · doi:10.1108/jd-04-2023-0073 · Cited by 6
Large Language Models in Psycholinguistic Studies
Günther, F., & Cassani, G. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00749-3 · Cited by 6
Enriching large language models with semantic lexicons and analogies
Zervakis, G. (2023) · theses.fr (ABES) · doi:10.70675/4f4cd0e5z5987z422az813bzc6e431d1a918 · Cited by 7
Lee, R. Y., Uyeda, A. M., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Dotolo, D. G., Engelberg, R. A., & Kross, E. K. (2025) · Journal of Palliative Medicine · doi:10.1089/jpm.2024.0496 · Cited by 3
VAST: The Valence-Assessing Semantics Test for Contextualizing Language Models
Wolfe, R., & Caliskan, A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21400 · Cited by 12
Repo, L., Hashimoto, B. J., & Laippala, V. (2023) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqad049 · Cited by 3
The Journey of Language Models in Understanding Natural Language
Liu, Y. R., Zhou, J., Sang, G., Huang, R., Zhao, X., Fang, J., Wang, T., & Li, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7707-5_29 · Cited by 3
Lázaro, E., & Moscardó, V. (2024) · Healthcare · doi:10.3390/healthcare12192008 · Cited by 5
“The pioneer and the first relevant model that used this technology was the Bert model, which was introduced in 2018 [22].”
MethodLarge Language Models for Identification of Medical Data in Unstructured Records
Petkov, P., Tomov, L., & Markov, E. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-84312-9_8 · Cited by 3
Multilingual multi-task quantum transfer learning
Buonaiuto, G., Guarasci, R., Pietro, G. D., & Esposito, M. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-025-00260-w · Cited by 5
“The second NLM taken into account is Electra (Clark et al. 2020), since it has shown a better ability to capture contextual word representations outperforming, in its downstream performance, other models, like Bert, given the same model size, data, and compute (Rogers et al. 2020).”
Dataset Alignment for Fine-Tuning Large Language Models for PJOK Educational Chatbot
Mawardi, V. C., Jonathan, J., Syahwalina, J., Monika, S., Widoatmodjo, S., & Wijaya, E. (2024) · International Conference on Informatics and Computing (ICIC) · doi:10.1109/icic64337.2024.10956758 · Cited by 3
“BERT or Bidirectional Encoder Representation from Transformers is a Machine Learning model commonly used in Natural Language Processing [17].”
AKER: Arabic Knowledge-enriched Reader for Machine Reading Comprehension
Albilali, E., Al-Twairesh, N., & Hosny, M. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3736164 · Cited by 3
“Although BERT was found to capture syntactic phenomena [66], genuine semantic comprehension remains an issue [66].”
Mono- and cross-lingual evaluation of representation language models on less-resourced languages
Ulčar, M., Žagar, A., Armendariz, C. S., Repar, A., Pollak, S., Purver, M., & Robnik‐Šikonja, M. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101852 · Cited by 4
Zhang, P., & Chaudhary, D. (2025) · Biomedicines · doi:10.3390/biomedicines13102496 · Cited by 3
“Following standard practices for extracting contextualized embeddings from transformer-based models [35,36], we extracted embeddings from the final hidden layer of the model.”
AraBERT for Arabic cyberbullying detection in Facebook comments
Hithnawi, R. I., Hamarsheh, M. M. N., & Maree, M. (2025) · Journal of Cybersecurity · doi:10.1093/cybsec/tyaf030 · Cited by 6
METAHEURISTIC-DRIVEN HYPERPARAMETER OPTIMIZATION FOR BERT IN SENTIMENT ANALYSIS
El-Demerdash, A. A., & Hamid, N. B. A. (2025) · Archives for Technical Sciences · doi:10.70102/afts.2025.1833.176 · Cited by 3
Arabic hate speech detection based on BERT models variants
Duridi, T., Jaber, A., & Martínez, P. (2025) · Egyptian Informatics Journal · doi:10.1016/j.eij.2025.100845 · Cited by 4
Mahmud, A., Rawajfih, Y., & Wu, F. (2025) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon67449.2025.11267712 · Cited by 3
“Security practitioners require reliable metrics to assess whether reported performance represents genuine analytical capability or evaluation inconsistencies stemming from data leakage, overfitting, or methodological bias [3].”
Investment portfolio optimization with supervised learning and attention mechanism
Yan, Z. (2025) · Egyptian Informatics Journal · doi:10.1016/j.eij.2025.100839 · Cited by 4
Enhancing Sentiment Classification with Machine Learning and Combinatorial Fusion
Patten, S., Chen, P.-Y., Schweikert, C., & Hsu, D. F. (2025) · IEEE Conference on Pervasive and Intelligent Computing (PICom) · doi:10.1109/picom68402.2025.00016 · Cited by 5
Manage, P. D., Wu, Y., Zhang, J., Gunathilaka, T. M. A. U., & Li, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115315 · Cited by 3
Dai, X., Zhou, X., Dong, Q., & Zhou, K. (2026) · Buildings · doi:10.3390/buildings16020437 · Cited by 3
Lv, D., Wang, S., Xiao, W., Xu, J., & Jin, Y. (2026) · Biochimica et Biophysica Acta (BBA) - Reviews on Cancer · doi:10.1016/j.bbcan.2026.189562 · Cited by 4
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.