A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 301–400 of 1,014
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing
Maudslay, R. H., & Cotterell, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.11 · Cited by 45
López, J. A. H., Weyssow, M., Cuadrado, J. S., & Sahraoui, H. A. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556900 · Cited by 55
“In NLP, BERT encodes the syntactic knowledge of natural languages in its middle layers [31], which lines up with our conclusion for programming languages.”
BackgroundYuan, K., Haddad, Y. K., Law, R., Shakya, I., Haileyesus, T., Navon, L., Zhang, L., Liu, Y., & Bergen, G. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2023.04.013 · Cited by 24
Quantum transfer learning for acceptability judgements
Buonaiuto, G., Guarasci, R., Minutolo, A., Pietro, G. D., & Esposito, M. (2024) · Quantum Machine Intelligence · doi:10.1007/s42484-024-00141-8 · Cited by 35
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 40
Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--HastingsPreprint
Goyal, K., Dyer, C., & Berg-Kirkpatrick, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02736 · Cited by 41
“Masked language modeling (MLM) objectives [6, 30, 15] for sequences, although recent, have become ubiquitous for many Natural Language Processing (NLP) applications [20, 31, 24] because they are easy to optimize and enable learning of highly expressive and flexible representations by the virtue of conditioning on bidi…”
BackgroundFoundation models for bioinformatics
Chen, Z., Wei, L., & Gao, G. (2024) · Quantitative Biology · doi:10.1002/qub2.69 · Cited by 29
Roumbanis, L. (2025) · Futures · doi:10.1016/j.futures.2025.103538 · Cited by 36
An evaluation of estimative uncertainty in large language models
Tang, Z., Shen, K., & Kejriwal, M. (2026) · npj Complexity · doi:10.1038/s44260-026-00070-6 · Cited by 34
“Inprinciple,our researchalignswithearliercontributionsin “ BERTology, ” 81 which sought to investigate the behavior and underlying mechanisms of the BERT language model 82 and its derivatives.”
Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval
Litschko, R., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_23 · Cited by 36
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Phang, J., Liu, H., & Bowman, S. R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.42 · Cited by 44
Gessler, L., Behzad, S., Liu, Y. J., Peng, S., Zhu, Y., & Zeldes, A. (2021) · doi:10.18653/v1/2021.disrpt-1.6 · Cited by 58
Moving beyond content‐specific computation in artificial neural networks
Shea, N. J. (2021) · Mind & Language · doi:10.1111/mila.12387 · Cited by 27
FiLMing Multimodal Sarcasm Detection with Attention
Gupta, S., Shah, A., Shah, M., Syiemlieh, L., & Maurya, C. K. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-92307-5_21 · Cited by 22
Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View
Cao, B., Lin, H., Han, X., Liu, F., & Sun, L. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.398 · Cited by 20
Broadbent, M., Grespan, M. M., Axford, K., Zhang, X., Srikumar, V., Kious, B. M., & Imel, Z. E. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1110527 · Cited by 36
van Dijk, B., Kouwenhoven, T., Spruit, M. R., & van Duijn, M. J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.779 · Cited by 54
Matsuo, H., Nishio, M., Matsunaga, T., Fujimoto, K., & Murakami, T. (2024) · Cancers · doi:10.3390/cancers16213621 · Cited by 26
“Of particular note is the development after 2017, following the introduction of the transformer, which marked a significant evolution not only in natural language but also in image processing (5–9).”
BackgroundInfluentialEnriched Pre-trained Transformers for Joint Slot Filling and Intent Detection
Hardalov, M., Koychev, I. K., & Nakov, P. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_054 · Cited by 27
GhostBERT: Generate More Features with Cheap Operations for BERT
Huang, Z., Hou, L., Shang, L., Jiang, X., Chen, X. D., & Liu, Q. (2021) · doi:10.18653/v1/2021.acl-long.509 · Cited by 24
“On the other hand, it is shown in (Voita et al., 2019; Kovaleva et al., 2019; Rogers et al., 2020) that many attention maps in pre-trained language models exhibit typical positional patterns, e.g., diagonal or vertical, which can be easily generated from other similar ones using operations like…”
BackgroundLexFit: Lexical Fine-Tuning of Pretrained Language Models
Vulić, I., Ponti, E. M., Korhonen, A., & Glavaš, G. (2021) · doi:10.18653/v1/2021.acl-long.410 · Cited by 40
“The motivation for this work largely stems from the recent work on probing and analyzing pretrained language models for various types of knowledge they might implicitly store (e.g., syntax, world knowledge) (Rogers et al., 2020).”
BackgroundFilipavicius, M., Manica, M., Cadow, J., & Martínez, M. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03084 · Cited by 27
“We can visualize the multi-head self-attention for the input sequence representation across the layers and heads [10, 36, 44], which can elucidate if the model learns the syntax and semantics necessary to perform well on prediction tasks.”
Can language representation models think in bets?
Tang, Z., & Kejriwal, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.221585 · Cited by 23
Camp: A Two-Stage Approach to Modelling Prosody in Context
Hodari, Z., Moinet, A., Karlapati, S., Lorenzo-Trueba, J., Merritt, T., Joly, A., Abbas, A., Karanasou, P., & Drugman, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414413 · Cited by 30
“CAMP is our proposed model using prosody predicted from context. tactic information [30].”
How is BERT surprised? Layerwise detection of linguistic anomalies
Li, B., Zhu, Z., Thomas, G., Xu, Y., & Rudzicz, F. (2021) · doi:10.18653/v1/2021.acl-long.325 · Cited by 40
“Soon after BERT’s release, many papers invented probing techniques to discover what linguistic knowledge it contains, and how this information is distributed between layers (e.g., Rogers et al. (2021) provides a comprehensive overview).”
BackgroundInfluentialUngoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2021) · HAL · doi:10.14618/ids-pub-10468 · Cited by 21
“With the increasing interest in language modeling in recent years in Natural Language Processing (NLP) (Rogers et al., 2020), particularly concerning contextualized word representations1 (Peters et al.”
InfluentialFine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
Al-Ghamdi, S. A., Al-Khalifa, H. S., & Al‐Salman, A. S. (2023) · Applied Sciences · doi:10.3390/app13074225 · Cited by 41
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 26
“While in this paper we focus on eliciting and evaluating semantic and commonsense knowledge, other types of knowledge exist (e.g., syntactic, linguistic) [4].”
BackgroundInfluentialNatural Language Processing for Radiation Oncology: Personalizing Treatment Pathways
Lin, H., Ni, L., Phuong, C., & Hong, J. C. (2024) · Pharmacogenomics and Personalized Medicine · doi:10.2147/pgpm.s396971 · Cited by 25
Mosbach, M., Khokhlova, A., Hedderich, M. A., & Klakow, D. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.7 · Cited by 38
The Singleton Fallacy: Why Current Critiques of Language Models Miss the Point
Sahlgren, M., & Carlsson, F. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.682578 · Cited by 34
“From the ever-growing literature on “BERTology” [25] we know that there are tasks and linguistic phenomena that current language models handle badly, if at all, and we also know that they sometimes “cheat” when solving certain types of tasks [20].”
BackgroundEdwards, C. (2021) · Communications of the ACM · doi:10.1145/3449049 · Cited by 23
Li, Q., Cheng, X., Song, C., & Liu, T. (2023) · Symmetry · doi:10.3390/sym15030731 · Cited by 20
A multi-modal deep learning solution for precise pneumonia diagnosis: the PneumoFusion-Net model
Wang, Y., Liu, C., Fan, Y., Niu, C., Huang, W., Pan, Y., Li, J., Wang, Y., & Li, J. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1512835 · Cited by 24
Long-Tail Zero and Few-Shot Learning via Contrastive Pretraining on and for Small Data
Rethmeier, N., & Augenstein, I. (2022) · doi:10.3390/cmsf2022003010 · Cited by 23
Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code
Paltenghi, M., & Pradel, M. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678712 · Cited by 39
Joint Entity and Relation Extraction Based on Table Labeling Using Convolutional Neural Networks
Ma, Y., Hiraoka, T., & Okazaki, N. (2022) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2022.spnlp-1.2 · Cited by 34
“As noted in an existing study, the bottom BERT layers encode the surface information, for example, the phrasal syntax and word order (Jawahar et al., 2019; Rogers et al., 2020).”
BackgroundResultContext Matters: A Pragmatic Study of PLMs’ Negation Understanding
Gubelmann, R., & Handschuh, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.315 · Cited by 36
“Rogers et al. (2020) provides an overview.”
BackgroundRepresentation biases in sentence transformers
Nikolaev, D., & Padó, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.268 · Cited by 38
Vagliano, I., Dormosh, N., Rios, M., Luik, T. T., Buonocore, T. M., Elbers, P., Dongelmans, D. A., Schut, M. C., & Abu–Hanna, A. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104504 · Cited by 22
Liao, H., Qi, J., Zhang, J., Zhang, C., Liu, F., & Ding, W. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102142 · Cited by 20
A matter of framing: The impact of linguistic formalism on probing results
Kuznetsov, I., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.13 · Cited by 30
Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher
Karamanolakis, G., Hsu, D., & Gravano, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.323 · Cited by 23
Deception detection in text and its relation to the cultural dimension of individualism/collectivism
Papantoniou, K., Papadakos, P., Patkos, T., Flouris, G., Androutsopoulos, I., & Plexousakis, D. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000152 · Cited by 29
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
Ji, T., Jain, S., Ferdman, M., Milder, P. A., Schwartz, H. A., & Balasubramanian, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.363 · Cited by 31
A Novel Deep Learning Approach Using Contextual Embeddings for Toponym Resolution
Cardoso, A. B., Martins, B., & Estima, J. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi11010028 · Cited by 22
“Besides ELMo and BERT, there are numerous other contextual word embedding models that could be explored [40,41,48,49], pre-trained with larger datasets and/or considering additional language modeling objectives.”
BackgroundInductive Biases and Variable Creation in Self-Attention MechanismsPreprint
Edelman, B., Goel, S., Kakade, S. M., & Zhang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10090 · Cited by 40
“A line of empirical work (“BERTology”) has made progress on understanding and interpreting state-of-the-art Transformer language models by examining the activations of their attention mechanisms (Clark et al., 2019; Rogers et al., 2020; Tenney et al., 2019).”
BackgroundInfluentialAngdresey, A., Sitanayah, L., & Tangka, I. L. H. (2025) · Journal of Computing Theories and Applications · doi:10.62411/jcta.11668 · Cited by 25
A survey of transformer architectures for autonomous driving
Chu, F., Li, H., Xie, L., & Zhao, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130338 · Cited by 28
How Can the [MASK] Know? The Sources and Limitations of Knowledge in BERT
Podkorytov, M., Biś, D., & Liu, X. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534299 · Cited by 24
“Rogers et al. [13] provide a comprehensive survey of that research direction.”
BackgroundComputational Measures of Deceptive Language: Prospects and Issues
Tomas, F., Dodier, O., & Demarchi, S. (2022) · Frontiers in Communication · doi:10.3389/fcomm.2022.792378 · Cited by 23
“A recent review regarding the properties of BERT shows that its embeddings incorporate syntactic tree-like structures, parts of speech, and semantic knowledge (Rogers et al., 2020).”
BackgroundBERT-based Topic Modeling Approach for Malaria Research Publication
Hidayat, A. A., Nirwantono, R., Budiarto, A., & Pardamean, B. (2022) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis56303.2022.10017743 · Cited by 19
“The distinction of BERT in the NLP area is mainly attributed to the use of Transformers initially proposed by Vaswani et al., which improve parallelization and the ability to learn long-range dependencies in the text sequence [23].”
MethodAllenbrand, C. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100288 · Cited by 25
The cortical representation of language timescales is shared between reading and listening
Chen, C., la Tour, T. D., Gallant, J. L., Klein, D. B., & Deniz, F. (2024) · Communications Biology · doi:10.1038/s42003-024-05909-z · Cited by 38
Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs
Li, Z. H., & Shin, D. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644946 · Cited by 39
Mosbach, M., Khokhlova, A., Hedderich, M. A., & Klakow, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.227 · Cited by 28
“One prominent technique to gain insights about the linguistic knowledge encoded in pre-trained models is probing (Rogers et al., 2020).”
MethodRicard, B. J., & Hassanpour, S. (2021) · Journal of Medical Internet Research · doi:10.2196/27314 · Cited by 21
“One recent contribution in natural language processing (NLP) is the Bidirectional Encoder Representations From Transformers (BERT) neural network, which has demonstrated superior performance on a wide variety of social media NLP tasks [57-61].”
BackgroundThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
“There are two main limitations of current featurebased probing approaches (Rogers et al., 2020; Belinkov, 2022).”
BackgroundThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
“There are two main limitations of current featurebased probing approaches (Rogers et al., 2020; Belinkov, 2022).”
BackgroundGeng, T., Xu, Z., Qu, Y., & Wong, W. E. (2026) · Computers, Materials & Continua · doi:10.32604/cmc.2025.074081 · Cited by 16
DiLBERT: Cheap Embeddings for Disease Related Medical NLP
Roitero, K., Portelli, B., Popescu, M. H., & Mea, V. D. (2021) · IEEE Access · doi:10.1109/access.2021.3131386 · Cited by 25
Geographic Adaptation of Pretrained Language Models
Hofmann, V., Glavaš, G., Ljubešić, N., Pierrehumbert, J. B., & Schütze, H. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00652 · Cited by 26
“Despite the conceptual simplicity of language modeling objectives, large-scale pretraining has been shown to incorporate complex linguistic knowledge into pretrained models, at various levels (Rogers et al., 2020): morphological (Edmiston, 2020; Hofmann et al.”
BackgroundProbing for Predicate Argument Structures in Pretrained Language Models
Conia, S., & Navigli, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.316 · Cited by 31
“This body of work quickly attracted increasing attention and grew to become a field of study with a name of its own: BERTology (Rogers et al., 2020).”
BackgroundSkipBERT: Efficient Inference with Shallow Layer Skipping
Wang, J., Chen, K., Chen, G., Shou, L., & McAuley, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.503 · Cited by 38
“As revealed by Jawa-har et al. (2019); Rogers et al. (2020), the lower layers of BERT mainly focus on short-distance context, while the higher layers are able to capture long-range dependencies.”
ResultTopoBERT: Exploring the topology of fine-tuned word representations
Rathore, A., Zhou, Y., Srikumar, V., & Wang, B. (2023) · Information Visualization · doi:10.1177/14738716231168671 · Cited by 37
“Transformer-based models are widely used in contemporary NLP 71 applications, and various studies have focused on probing the contextualized word embed-dings they construct.”
Huber, E., Sauppe, S., Isasi-Isasmendi, A., Bornkessel‐Schlesewsky, I., Merlo, P., & Bickel, B. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00121 · Cited by 34
“This challenges the opinion that transformers have “ little cognitive motivation ” (Rogers et al., 2020, p. 842).”
BackgroundText Classification Modeling Approach on Imbalanced-Unstructured Traffic Accident Descriptions Data
Seo, Y., Park, J., Oh, G., Kim, H., Hu, J. S., & So, J. J. (2023) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2023.3335817 · Cited by 19
“For data interpretation and classification, this study proposes a bidirectional encoder representation from transformer (BERT) model, which is one of the most widely used and high-performing transformer models in the field of NLP [19].”
MethodCan Language Models Understand Physical Concepts?
Li, L., Xu, J., Dong, Q., Zheng, C., Sun, X., Kong, L., & Liu, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.726 · Cited by 36
Pitfalls (and advantages) of sophisticated large language models
Strasser, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-18851-0.00007-x · Cited by 20
Brain and Cognitive Science Inspired Deep Learning: A Comprehensive Survey
Zhang, Z., Ding, X., Liang, X., Zhou, Y., Qin, B., & Liu, T. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3527551 · Cited by 28
Telling BERT’s Full Story: from Local Attention to Global Aggregation
Pascual, D., Brunner, G., & Wattenhofer, R. P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.9 · Cited by 22
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.331 · Cited by 23
“, 2018), and layer-wise interpretations of such models (Rogers et al., 2020; Tenney et al., 2019).”
BackgroundInfluentialProbing Language Models for Understanding of Temporal Expressions
Thukral, S., Kukreja, K., & Kavouras, C. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.31 · Cited by 31
Token Dropping for Efficient BERT Pretraining
Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., & Zhou, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.262 · Cited by 48
Transformers in Natural Language Processing
Yvon, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24349-3_6 · Cited by 25
“A recent biblio-graphical survey of this “Bertological” literature has no less than 110 references [Rogers et al., 2020].”
BackgroundCharacter-based Neural Semantic Parsing
van Noord, R. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.169308968 · Cited by 18
Zou, G., Li, P., Yang, S., Hu, S., Pang, S., & Gan, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104219 · Cited by 21
Does BERT Pay Attention to Cyberbullying?
Elsafoury, F., Katsigiannis, S., Wilson, S. R., & Ramzan, N. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463029 · Cited by 26
“ing the Pearson’s linear correlation between the mean attention weights of fine-tuned BERT of all heads across the last layers (9-12) and the tokens’ absolute importance score, as it has been shown that fine-tuning affects mostly BERT’s last layers (9-12) [27].”
BackgroundMethodProtein sequence profile prediction using ProtAlbert transformer
Behjati, A., Zare‐Mirakabad, F., Arab, S. S., & Nowzari-Dalini, A. (2022) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2022.107717 · Cited by 25
Review of Natural Language Processing in Pharmacology
Trajanov, D., Trajkovski, V., Dimitrieva, M., Dobreva, J., Jovanovik, M., Klemen, M., Žagar, A., & Robnik‐Šikonja, M. (2023) · Pharmacological Reviews · doi:10.1124/pharmrev.122.000715 · Cited by 25
“Transformers[232] NER, NLI, QA, SRL, Classification, Embeddings [119] [87] [237] [15] [84] [58] [123] [4] [32] [154] [131] [208] [3] [165] [53] [190] [71] [96] [170] [27] [83] [112] [7] [128] [196] [145] [120][156] [133] [166] [137] [45] [5] [249] [179] URL: https://huggingface.”
Kauk, J., Humprecht, E., Kreysa, H., & Schweinberger, S. R. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108546 · Cited by 21
MiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question Answering
Alsabbagh, A. R., Mansour, T., Al-Kharabsheh, M., Ebdah, A. S., Al-Emaryeen, R., Al-Nahhas, S., Mahafza, W. S., & Al-Kadi, O. S. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.001 · Cited by 19
Language Models Use Monotonicity to Assess NPI Licensing
Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., & Steinert‐Threlkeld, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.439 · Cited by 48
Sentence Complexity in Context
Iavarone, B., Brunato⋄, D., & Dell’Orletta⋄, F. (2021) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2021.cmcl-1.23 · Cited by 17
“Despite the remarkable performance that neural language models have achieved so far in a variety of NLP tasks (Rogers et al., 2020) – also close to ours such as the prediction of perceived sentence acceptability in context (Lau et al., 2020) – our results show that this is not the case as regards…”
ResultComparative Analysis of Word Embeddings in Assessing Semantic Similarity of Complex Sentences
Chandrasekaran, D., & Mago, V. (2021) · IEEE Access · doi:10.1109/access.2021.3135807 · Cited by 19
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
Merrill, W., Ramanujan, V., Goldberg, Y., Schwartz, R., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.133 · Cited by 13
“Despite various attempts to study these biases in transformers (Rogers et al., 2020; Lovering et al., 2021), it remains an interesting open question what they are, or even how to characterize them in a way relevant to the domain of language.”
BackgroundTowards Understanding and Answering Comparative Questions
Bondarenko, A., Ajjour, Y., Dittmar, V., Homann, N., Braslavski, P., & Hagen, M. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498534 · Cited by 19
“As representations, we used CLS -token embeddings and the mean of all token embeddings [37].”
MethodRuleBERT: Teaching Soft Rules to Pre-Trained Language Models
Saeed, M., Ahmadi, N., Nakov, P., & Papotti, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.110 · Cited by 43
“, 2020) are established tools for capturing both linguistic and factual knowledge (Clark et al., 2019b; Rogers et al., 2020).”
MethodLMdiff: A Visual Diff Tool to Compare Language Models
Strobelt, H., Hoover, B., Satyanaryan, A., & Gehrmann, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.12 · Cited by 26
“This anecdotal evidence informs hypotheses that are then rigorously studied (e.g., Tenney et al., 2019; Belinkov and Glass, 2019; Rogers et al., 2020).”
BackgroundAn ELECTRA-Based Model for Neural Coreference Resolution
Gargiulo, F., Minutolo, A., Guarasci, R., Damiano, E., Pietro, G. D., Fujita, H., & Esposito, M. (2022) · IEEE Access · doi:10.1109/access.2022.3189956 · Cited by 18
What do tokens know about their characters and how do they know it?
Kaushal, A., & Mahowald, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.179 · Cited by 57
Caines, A., Yannakoudakis, H., Allen, H., Pérez-Paredes, P., Byrne, B., & Buttery, P. J. (2022) · Linköping electronic conference proceedings · doi:10.3384/ecp190003 · Cited by 38
Meltzer, P., Lambourne, J. G., & Grandi, D. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4062454 · Cited by 18
Construction Grammar and Language Models
Madabushi, H. T., Romain, L., Milin, P., & Divjak, D. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009049139.023 · Cited by 22
Enhancing deep neural networks with morphological information
Klemen, M., Krsnik, L., & Robnik‐Šikonja, M. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000080 · Cited by 27
Disentangling Representations of Text by Masking Transformers
Zhang, X., van de Meent, J.-W., & Wallace, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.60 · Cited by 20
“The opacity of embeddings produced by models such as BERT has motivated NLP research on designing probing tasks as a means of uncovering the properties of input texts that are encoded in learned representations (Rogers et al., 2020; Linzen et al., 2019; Tenney et al., 2019).”
BackgroundClassifying Scientific Publications with BERT - Is Self-attention a Feature Selection Method?
García-Silva, A., & Gómez-Pérez, J. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_11 · Cited by 19
The Taxonomy of Writing Systems: How to Measure How Logographic a System Is
Sproat, R., & Gutkin, A. (2021) · Computational Linguistics · doi:10.1162/coli_a_00409 · Cited by 26
“…models in order to understand the linguistic function of the attention and also improve model compression schemes (Clark et al. 2019; Michel, Levy, and Neubig 2019; Vig and Belinkov 2019; Voita et al. 2019; Behnke and Heafield 2020; Wang et al. 2020; Rogers, Kovaleva, and Rumshisky 2021).”
MethodVisual Comparison of Language Model Adaptation
Sevastjanova, R., Cakmak, E., Ravfogel, S., Cotterell, R., & El‐Assady, M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209458 · Cited by 27
“It has been shown that these embeddings encode different language properties found in the training data [59].”
Background
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.