A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 701–800 of 1,014
Li, Z., & Zhang, X. (2023) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlviii-1-w2-2023-547-2023 · Cited by 7
“The main contribution of the BERT model is the incorporation of bidirectional contextual information into the pre-training process, enabling a better understanding of semantics and grammar in natural language(Rogers, Kovaleva, & Rumshisky, 2020).”
MethodLi, J., & Armstrong, B. C. (2024) · Cognitive Science · doi:10.1111/cogs.13416 · Cited by 12
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 8
Metadata Integration for Spam Reviews Detection on Vietnamese E-Commerce Websites
Dinh, C. V., & Luu, S. T. (2024) · International Journal of Asian Language Processing · doi:10.1142/s2717554524500024 · Cited by 6
“There are many methods that have been researched for sentence embedding, and SBERT is a model proposed to utilize BERTology models (BERT / RoBERTa) 17 for generating document representations.”
MethodSocial Perception of Faces in a Vision-Language Model
Hausladen, C. I., Knott, M., Camerer, C. F., & Perona, P. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732041 · Cited by 22
“However, unlike LLMs which capture meaning from language, an already symbolic description, CLIP works with raw images where meaning is implicit in the pixel brightness values.”
Characterizing English Preposing in PP constructions
Potts, C. (2024) · Journal of Linguistics · doi:10.1017/s0022226724000227 · Cited by 9
Nyzova, M., Vysotska, V., Chyrun, L., Hu, Z., Ushenko, Y. O., & Uhryn, D. I. (2025) · International Journal of Computer Network and Information Security · doi:10.5815/ijcnis.2025.04.08 · Cited by 6
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Mikhailov, V., Serikov, O., & Artemova, E. (2021) · doi:10.18653/v1/2021.sigtyp-1.10 · Cited by 13
Universal Dependencies-Based PoS Tagging Refinement Through Linguistic Resources
Lopes, L., Duran, M. S., & Pardo, T. A. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91699-2_41 · Cited by 6
The Specificity and Helpfulness of Peer-to-Peer Feedback in Higher Education
Rietsche, R., Caines, A., Schramm, C., Pfütze, D., & Buttery, P. J. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.15 · Cited by 11
“Research has shown that one factor determining feedback helpfulness is whether the feedback points are generic or specific (Lipnevich and Smith, 2018; Shute, 2008; Hattie and Timperley, 2007).”
BackgroundSentiment Analysis on Inflation after COVID-19
Li, X., & Tang, Z. (2022) · Applied Economics and Finance · doi:10.11114/aef.v10i1.5821 · Cited by 7
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics
Lauscher, A., Bianchi, F., Bowman, S. R., & Hovy, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.539 · Cited by 23
From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?
Coil, A., & Shwartz, V. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.169 · Cited by 8
Topic Modeling With Topological Data Analysis
Byrne, C., Horak, D., Moilanen, K., & Mabona, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.792 · Cited by 8
“However, our word class clusters could motivate the application of TDA to the recent field of "BERTology" to interpret emergent linguistic structure across Transformer architectures (Rogers et al., 2020; Manning et al., 2020).”
MethodFaye, G., Ouerdane, W., Gadek, G., Gahbiche, S., & Gatepaille, S. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102230 · Cited by 6
Ormerod, M., del Rincón, J. M., & Devereux, B. J. (2023) · Computational Linguistics · doi:10.1162/coli_a_00495 · Cited by 7
“…such as BERT (Devlin et al. 2018), there has been a surge of interpretability research on this type of architecture (in particular, the study of BERT and related models has gained much popularity in a wave of research sometimes referred to as “BERTology” [Rogers, Kovaleva, and Rumshisky 2021]).”
BackgroundA Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Monea, G., Peyrard, M., Josifoski, M., Chaudhary, V., Eisner, J. M., Kıcıman, E., Palangi, H., Patra, B., & West, R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.369 · Cited by 32
Machine-Learning-Augmented Analysis of Textual Data: Application in Transit Disruption Management
Noursalehi, P., Koutsopoulos, H. N., & Zhao, J. (2020) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2020.3038395 · Cited by 7
“In this approach, the term “embedding” refers to the output vector of the final Transformer layer [21].”
BackgroundA Natural Language Processing Survey on Legislative and Greek Documents
Krasadakis, P., Sakkopoulos, E., & Verykios, V. S. (2021) · Panhellenic Conference on Informatics · doi:10.1145/3503823.3503898 · Cited by 10
Nautilus: An Optimized System for Deep Transfer Learning over Evolving Training Datasets
Nakandala, S., & Kumar, A. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3517846 · Cited by 12
“These parameters are grouped into layers , and each layer captures a different level of abstraction about the data, from generic ones to specific ones [34, 43, 62, 77].”
Word-order Typology in Multilingual BERT: A Case Study in Subordinate-Clause Detection
Nikolaev, D., & Padó, S. (2022) · Workshop on Research in Computational Linguistic Typology and Multilingual NLP (SIGTYP) · doi:10.18653/v1/2022.sigtyp-1.2 · Cited by 9
Classifying the Ideological Orientation of User-Submitted Texts in Social Media
Ravi, K., Vela, A., & Ewetz, R. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00066 · Cited by 6
DA-Parser: A Pre-trained Domain-aware Parsing Framework for Heterogeneous Log Analysis
Tao, S., Liu, Y., Meng, W., Wang, J., Zhao, Y., Su, C., Tian, W., Zhang, M., Yang, H., & Chen, X. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00050 · Cited by 8
“Research has shown that through pre-training, BERT can not only learn syntactic and semantic features within a single text sequence from a specific language or domain [25], but it can also model multiple-domain dependencies [26].”
BackgroundPlug-and-Play Document Modules for Pre-trained Models
Xiao, C., Zhang, Z., Han, X., Chan, C.-M., Lin, Y., Liu, Z., Li, X., Li, Z., Cao, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.875 · Cited by 18
Do Transformers Parse while Predicting the Masked Word?
Zhao, H., Panigrahi, A., Ge, R., & Arora, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1029 · Cited by 60
Preparing Lessons for Progressive Training on Language Models
Pan, Y., Yuan, Y., Yin, Y., Shi, J., Xu, Z., Zhang, M., Shang, L., Jiang, X., & Liu, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29851 · Cited by 18
Dumitrescu, Ș. D., Avram, A.-M., & Pyysalo, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.387 · Cited by 86
DagoBERT: Generating Derivational Morphology with a Pretrained Language Model
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.316 · Cited by 39
“What kind of linguistic knowledge is encoded by pretrained language models (PLMs) such as ELMo (Peters et al., 2018), GPT-2 (Radford et al., 2019), and BERT (Devlin et al., 2019)?”
BackgroundComparing Dependency-based Compositional Models with Contextualized Word Embeddings
Gamallo, P., Prada-Corral, M., & García, M. (2021) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0010391812581265 · Cited by 6
BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology
Gessler, L., & Schneider, N. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.43 · Cited by 9
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
Jones, A., Wang, W. Y., & Mahowald, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.471 · Cited by 12
“…models using probing tasks and downstream performance measures (Conneau et al., 2020; Wu and Dredze, 2019, 2020; Pires et al., 2019; Groenwold et al., 2020), and probed Transformer models (Wolf et al., 2020) for linguistic structure (see Rogers et al. 2020 for an overview of over 150 studies).”
MethodTowards improving coherence and diversity of slogan generation
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000474 · Cited by 14
Exploring Early Prediction of Buyer-Seller Negotiation OutcomesPreprint
Chawla, K., Lucas, G., May, J., & Gratch, J. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02363 · Cited by 8
Towards Zero-Shot Knowledge Distillation for Natural Language ProcessingPreprint
Rashid, A., Lioutas, V., Ghaddar, A., & Rezagholizadeh, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15495 · Cited by 20
Semeraro, A., Vilella, S., Mohammad, S. M., Ruffo, G., & Stella, M. (2023) · Research Square · doi:10.21203/rs.3.rs-2428155/v1 · Cited by 10
Cheok, A. D., & Zhang, E. Y. (2023) · Qeios · doi:10.32388/3ntolq.2 · Cited by 8
ChatGPT and Deepseek: Can They Predict the Stock Market and Macroeconomy?
Chen, J., Tang, G., Zhou, G., & Zhu, W. (2025) · arXiv · doi:10.48550/arxiv.2502.10008 · Cited by 6
Li, W., Ma, X., Wang, X., Wu, L., Salati, S., & Xie, Z. (2024) · Applied Computing and Geosciences · doi:10.1016/j.acags.2024.100164 · Cited by 6
Debelak, R., Koch, T. K., Aßenmacher, M., & Stachl, C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251351285 · Cited by 7
“Most recently, tailored approaches for transformer-based models based on saliency maps or inspections of the attention patterns have been proposed (Clark et al., 2019; Michel et al., 2019; Rogers et al., 2020).”
Explainable AI: Context-aware layer-wise integrated gradients for explaining transformer models
Mersha, M. A., & Kalita, J. K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133050 · Cited by 8
“A modest increase is observed at Layer 4, coinciding with an increase in the classifier contribution, suggesting the early stages of representational alignment with the decision space [80].”
Layer-dependent dynamic spectral weighting for efficient transformer models
Huang, Z., Yu, Y., Zheng, S., Chen, M., & Li, Q. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114285 · Cited by 9
Kowalska, B., & Kwaśnicka, H. (2026) · Machine Learning · doi:10.1007/s10994-026-07129-4 · Cited by 11
Improving Pretraining Techniques for Code-Switched NLP
Das, R., Ranjan, S., Pathak, S. P., & Jyothi, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.66 · Cited by 12
“Prior studies have carried out detailed investigations of how BERT works and what kind of information is encoded within representations in each of its layers (Jawahar et al., 2019; Liu et al., 2019; Rogers et al., 2020).”
BackgroundMethodLarge scale models in radiology: revolutionizing the future of medical imaging
Li, Y., Liu, J., Zhang, Y., Wei, J., Yang, L., Yang, X., Shi, Y., Zhao, L., Yang, W., Li, H., et al. (2024) · Radiology Science · doi:10.15212/radsci-2023-0018 · Cited by 6
-generAItor: Tree-in-the-loop Text Generation for Language Model Explainability and Adaptation
Spinner, T., Kehlbeck, R., Sevastjanova, R., Stähle, T., Keim, D. A., Deußen, O., & El‐Assady, M. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3652028 · Cited by 15
“Although BERT is the most widely analyzed language model so far [Rogers et al. 2020], other transformer models, such as GPT-2, and their produced embedding spaces have also attracted computational linguistics’ and visual analytics researchers’ attention [Ethayarajh 2019; Sevastjanova et al. 2022].”
DSNetax: a deep learning species annotation method based on a deep-shallow parallel framework
Zhao, H., Zhang, S.-Y., Qin, H., Liu, X., Ma, D.-N., Han, X., Mao, J., & Liu, S.-P. (2024) · Briefings Bioinform. · doi:10.1093/bib/bbae157 · Cited by 5
“While this is valuable for downstream functions that rely on sentence relationships, many studies have confirmed that for non-two-sentence forms of tasks, deleting NSP tasks will improve the performance of downstream tasks [17].”
BackgroundOptimization of AES using BERT and BiLSTM for Grading the Online Exams
(2024) · International journal of intelligent engineering and systems · doi:10.22266/ijies2024.1031.31 · Cited by 10
“This capability is enabled by transformers and is referred to as bidirectional [21].”
BackgroundMethodAnditama, M. R., & Hidayanto, A. N. (2024) · Indonesian Journal of Computer Science · doi:10.33022/ijcs.v13i4.4239 · Cited by 12
“These models utilize self-attention mechanisms to capture dependencies between words in sentences, enabling parallelization and efficient processing of extensive dependencies [18] .”
MethodAbdollahpour, N., Kakaraparthi, A., Valluru, C. S., & Artan, N. S. (2025) · IEEE Access · doi:10.1109/access.2025.3576056 · Cited by 8
“It paved the way for advanced models like BERT [29], GPT [30], and DeepSeek-R1 [31], significantly expanding machine learning applications beyond NLP into many other fields.”
Using large language models as a scalable mental status evaluation technique
Wagner, M., Stephenson, C., Jagayat, J., Kumar, A., Shirazi, A., Alavi, N., & Omrani, M. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00042-z · Cited by 7
“BERT was chosen as it has established itself as the de facto and widely adopted baseline for NLP experiments [37].”
Comparative Analysis of Transformer Models in Disaster Tweet Classification for Public Safety
Zisad, S. N., Chowdhury, N. M. I., & Hasan, R. (2025) · IEEE World Forum on Public Safety Technology (WF-PST) · doi:10.1109/wf-pst65083.2025.00038 · Cited by 5
“These models use self attention mechanisms to understand the relationships between words in a sentence, allowing them to capture the full context of the message [12].”
Studying word order through iterative shuffling
Malkin, N., Lanka, S., Goel, P., & Jojić, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.809 · Cited by 16
“Recent work on large language models (Devlin et al., 2019; Tenney et al., 2019; Rogers et al., 2021) has made this a popular hypothesis.”
A test of indirect grounding of abstract concepts using multimodal distributional semantics
Utsumi, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.906181 · Cited by 7
Ortiz-Barajas, J.-G., Bel-Enguix, G., & Gómez-Adorno, H. (2022) · Mathematics · doi:10.3390/math10193578 · Cited by 5
Predicting Item Response Theory Parameters Using Question Statements Texts
Marinho, W., Clua, E., Martí, L., & Marinho, K. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576139 · Cited by 6
Dynamic Named Entity Recognition
Luiggi, T., Guigue, V., Soulier, L., Jendoubi, S., & Baelde, A. (2023) · ACM Symposium on Applied Computing · doi:10.1145/3555776.3577603 · Cited by 6
“Those modern language models offered a lot of opportunities [31]: their extraction abilities improved transfer learning on NER [30], even in the more difficult setting in which the target domain was only associated with distant supervision [21].”
An improved sentiment classification model based on data quality and word embeddings
Siagh, A., Laallam, F. Z., Kazar, O., & Salem, H. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05099-1 · Cited by 6
“Two years later, it has become the core of NLP in capturing and learning contextual relations between words [24, 25].”
BackgroundSentiment and Objectivity in Iranian State-Sponsored Propaganda on Twitter
Barrows, M., Haig, E., & Conduit, D. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3273729 · Cited by 6
“Moreover, it is not clear why the BERT models bring improvements in performance, which limits hypothesis-driven improvements [24].”
BackgroundIs it Possible to Preserve a Language using only Data?
Bensemann, J., Brown, J., Witbrock, M., & Yogarajan, V. (2023) · Cognitive Science · doi:10.1111/cogs.13300 · Cited by 5
Oh, B.-D., & Schuler, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.562 · Cited by 15
Automatic Readability Assessment for Closely Related Languages
Imperial, J. M., & Kochmar, E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.331 · Cited by 5
Nejadgholi, I., Balkır, E., Fraser, K., & Kiritchenko, S. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.18 · Cited by 13
Multi-Source Log Parsing With Pre-Trained Domain Classifier
Liu, Y., Tao, S., Meng, W., Wang, J., Yang, H., & Jiang, Y. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3329144 · Cited by 6
“Thus, it is a common practise to add [CLS] in classification problems to quickly learn generalized features of the input sequence [17].”
BackgroundBERT for Twitter Sentiment Analysis: Achieving High Accuracy and Balanced Performance
Renuka, O., & Niranchana, R. (2024) · Journal of Trends in Computer Science and Smart Technology · doi:10.36548/jtcsst.2024.1.003 · Cited by 6
Commit-Level Software Change Intent Classification Using a Pre-Trained Transformer-Based Code Model
Heričko, T., Šumak, B., & Karakatič, S. (2024) · Mathematics · doi:10.3390/math12071012 · Cited by 10
ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models
Madasu, A., & Lal, V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00180 · Cited by 8
“Hence, there has been a growing interest to explore the morphological capabilities of these models [14, 40, 43, 49, 55].”
Collaborative Growth: When Large Language Models Meet Sociolinguistics
Nguyen, D. (2025) · Language and Linguistics Compass · doi:10.1111/lnc3.70010 · Cited by 17
Dissociable frequency effects attenuate as large language model surprisal predictors improve
Oh, B.-D., & Schuler, W. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104645 · Cited by 11
EMTeC: A corpus of eye movements on machine-generated texts
Bolliger, L. S., Haller, P., Cretton, I. C. R., Reich, D. R., Kew, T., & Jäger, L. A. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02677-4 · Cited by 15
“…allows neural models to learn the distribution of words as well as nuanced syntactic (Blevins et al., 2018; Hewitt & Manning, 2019; Manning et al., 2020; Wei et al., 2021) and semantic patterns (Merrill et al., 2024; Rogers et al., 2020; Vuli´c et al., 2020; Zhang et al., 2021) in the language.”
Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations
Manginas, N., Chalkidis, I., & Malakasiotis, P. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.7 · Cited by 11
Wang, H., Tang, X., Lai, S., Leung, K., Zhu, J., Fung, G. P. C., & Wong, K. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.47 · Cited by 5
Quranic Verses Semantic Relatedness Using AraBERT
Alsaleh, A., Atwell, E., & Altahhan, A. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 5
Learning word representation by jointly using neighbor and syntactic contexts
Tian, C., Rong, W., Zhou, S., Zhang, J., Ouyang, Y., & Xiong, Z. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.130 · Cited by 5
Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
Kim, T., Li, B., & Lee, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.41 · Cited by 7
Local and global context-based pairwise models for sentence ordering
Manku, R. R., & Paul, A. J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108453 · Cited by 5
Towards a universal continuous knowledge base
Chen, G., Sun, M., & Liu, Y. (2021) · AI Open · doi:10.1016/j.aiopen.2021.11.001 · Cited by 5
An Empirical Study on Transfer Learning for Privilege Review
Zhao, H., Ye, S., & Yang, J. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9672008 · Cited by 5
“BERT immediately acquired much traction in both academia and industry due to its effectiveness in transferring knowledge learned in general domain to specific task, replicating the achievements we have seen in computer vision to natural language processing [14].”
BackgroundBruera, A., & Poesio, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796793 · Cited by 8
“Contextual distributional models have been generally shown to improve on static models on most tasks in Natural Language Processing (Rogers et al., 2020), but it should be noticed that they are dramatically more complex than static models (Lenci et al., 2021).”
BackgroundInfluentialA COVID-19 Search Engine (CO-SE) with Transformer-based architecture
Raza, S. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100068 · Cited by 8
About Time: Do Transformers Learn Temporal Verbal Aspect?
Metheniti, E., de Cruys, T. V., & Hathout, N. (2022) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2022.cmcl-1.10 · Cited by 17
CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer
Karlapati, S., Karanasou, P., Łajszczak, M., Abbas, S. A., Moinet, A., Makarov, P., Li, R., van Korlaar, A., Slangen, S., & Drugman, T. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-367 · Cited by 7
DisCut and DiscReT: MELODI at DISRPT 2023
Metheniti, E., Braud, C., Müller, P., & Rivière, L. (2023) · doi:10.18653/v1/2023.disrpt-1.3 · Cited by 5
“It has been shown that, broadly speaking, lower layers mostly encode morpho-syntactic information, while upper contain more semantic ones (Rogers et al., 2020; Kovaleva et al., 2019; Bender and Koller, 2020).”
BackgroundChronis, G., Mahowald, K., & Erk, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.14 · Cited by 27
EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching
Whitehouse, C., Christopoulou, F., & Iacobacci, I. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.499 · Cited by 5
Exploring the Efficacy of BERT in Bengali NLP: A Study on Sentiment Analysis and Aspect Detection
Hossain, M. J., Abdullah, S. M., Barkatullah, M., & Monir, M. F. (2023) · IEEE EUROCON - International Conference on Computer as a Tool (EUROCON) · doi:10.1109/eurocon56442.2023.10199048 · Cited by 6
“BERT is a deeply bidirectional and unsupervised technique for language representation [18].”
BackgroundRevision Transformers: Instructing Language Models to Change Their Values
Friedrich, F., Stammer, W., Schramowski, P., & Kersting, K. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230341 · Cited by 9
“LLMs have been shown to possess the capabilities to perform basic reasoning and represent world knowledge [31, 33, 14].”
BackgroundExplainability of Text Processing and Retrieval Methods: A Survey
Saha, S., Majumdar, D., & Mitra, M. (2026) · ACM Computing Surveys · doi:10.1145/3801957 · Cited by 10
“As in [149] (discussed above) and similar studies conducted within the NLP community [25, 82, 159], empirical results show that BERT puts redundant attention weights on [CLS] and [SEP] tokens.”
Applying Multilingual and Monolingual Transformer-Based Models for Dialect Identification
Popa, C., & Stefanescu, V. (2020) · International Conference on Computational Linguistics · Cited by 9
Supervised Contrastive Learning Approach for Contextual Ranking
Anand, A., Leonhardt, J., Rudra, K., & Anand, A. (2022) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3539813.3545139 · Cited by 6
“Note that, variance of the ranking metrics in ne-tuning over smaller datasets using over-parameterized contextual models is a known phenomena [9, 43, 52, 63].”
BackgroundDevelopmental Negation Processing in Transformer Language Models
Laverghetta, A., & Licato, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.60 · Cited by 6
OEIS: Knowledge Graph based Intelligent Search System in Ocean Engineering
Zhu, R., Xiaoqing, W., Hu, Y., Chen, Y., & Liu, B. (2022) · International Conference on Advanced Cloud and Big Data · doi:10.1109/cbd58033.2022.00035 · Cited by 5
“an excellent performance in natural language processing [11].”
BackgroundYou Are My Type! Type Embeddings for Pre-trained Language Models
Saeed, M., & Papotti, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.336 · Cited by 10
“Being trained in a self-supervised fashion, such models convey, to a certain extent, linguistic (Puccetti et al., 2021; Lin et al., 2019) and factual knowledge (Rogers et al., 2020; Meng et al., 2022).”
BackgroundEvaluating German Transformer Language Models with Syntactic Agreement TestsPreprint
Zaczynska, K., Feldhus, N., Schwarzenberg, R., Gabryszak, A., & Möller, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03765 · Cited by 7
“With the exception of experiments targeting a multilingual BERT model (Rogers et al., 2020), most of the studies were conducted only for English, however.”
BackgroundMethodInfluentialRuSentEval: Linguistic Source, Encoder Force!Preprint
Mikhailov, V., Taktasheva, E., Sigdel, E., & Artemova, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00573 · Cited by 5
Bornheim, T., Grieger, N., & Bialonski, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03094 · Cited by 5
On the Prunability of Attention Heads in Multilingual BERTPreprint
Budhraja, A., Pande, M., Kumar, P., & Khapra, M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12683 · Cited by 6
Geometric Algebra Attention Networks for Small Point CloudsPreprint
Spellings, M. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02393 · Cited by 16
Weck, B., Favory, X., Drossos, K., & Serra, X. (2021) · Repositori digital de la UPF (Universitat Pompeu Fabra) · doi:10.48550/arxiv.2110.07410 · Cited by 10
“We decided to use the penultimate layer as embeddings as this can produce highly contextualized representations that are not too task-specific [41].”
MethodHow much pretraining data do language models need to learn syntax?Preprint
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03160 · Cited by 8
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.