A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 101–200 of 1,014
Zhang, E. Y., Cheok, A. D., Pan, Z., Cai, J., & Yan, Y. (2023) · Sci · doi:10.3390/sci5040046 · Cited by 104
Towards a Definition of Generative Artificial Intelligence
Ronge, R., Maier, M., & Rathgeber, B. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00863-y · Cited by 101
“After its release it quickly surpassed traditional models such as LSTM models and constituted the state-of-the-art system for a variety of NLP related benchmarks (Rogers et al., 2020, 842).”
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
De Cao, N., Schlichtkrull, M. S., Aziz, W., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.262 · Cited by 75
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N., Zhao, X., Fang, T., Song, Y., & Yeung, D.-Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.329 · Cited by 69
“However, we are still investigating the reasons behind this success with the experimental setup side (Rogers et al., 2020; Prasanna et al., 2020).”
BackgroundA Primer on Pretrained Multilingual Language Models
Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A., & Kumar, P. (2025) · ACM Computing Surveys · doi:10.1145/3727339 · Cited by 108
“However, more complex tasks such as MT quality evaluation (Libovick`y et al., 2019) or language generation (Rönnqvist et al., 2019) remain outside the realm of these models currently, keeping the debate on universal patterns incomplete.”
InfluentialScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 260
“BERT is too harsh to tackle the instances that require complex reasoning [40].”
BackgroundGreco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 104
Tabi: An Efficient Multi-Level Inference System for Large Language Models
Wang, Y., Chen, K., Tan, H., & Guo, K. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3587438 · Cited by 147
“Attention layers and heads handle different aspects of the language [14, 56]; thus, accumulation makes the importance vector more reliable.”
BackgroundOn cross-lingual retrieval with multilingual text encoders
Litschko, R., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2022) · Information Retrieval · doi:10.1007/s10791-022-09406-x · Cited by 104
“This makes intuitive sense, as the parameters of higher Transformer layers encode compositional rather than lexical semantics [21,59].”
BackgroundMachine Learning Driven Mental Stress Detection on Reddit Posts Using Natural Language Processing
Inamdar, S., Chapekar, R., Gite, S., & Pradhan, B. (2023) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00020-8 · Cited by 99
“Turcan and McKeown [15] used the Dreaddit data-set with an explainable framework for stress detection, which can provide help in the form of a therapy chatbot or other deployable models for providing psychological help.”
MethodMichaelov, J. A., Coulson, S., & Bergen, B. K. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3176783 · Cited by 91
“However, previous research has shown that LMs can learn semantic relationships between words [45].”
BackgroundPrediction of RNA–protein interactions using a nucleotide language model
Yamada, K., & Hamada, M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac023 · Cited by 68
“In addition to its performance, BERT is highly interpretable and suitable for translating extended contextual information compared to conventional deep learning architectures, such as CNNs and long short-term memory (Rogers et al. , 2020).”
MethodInfluentialAnalyzing Transformers in Embedding Space
Dar, G., Geva, M., Gupta, A., & Berant, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.893 · Cited by 49
Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for ChinesePreprint
Zhang, Z., Zhang, H., Chen, K., Guo, Y., Hua, J., Wang, Y., & Zhou, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06696 · Cited by 124
“…in capturing syntax and semantic knowledge after pre-training (Hewitt and Manning, 2019; Ettinger, 2020), recent studies show that the current models still suffer from under-fitting issues, and it remains challenging to train a powerful model with less computation (Rogers et al., 2020).”
BackgroundProtein language models trained on multiple sequence alignments learn phylogenetic relationships
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2022) · Nature Communications · doi:10.1038/s41467-022-34032-y · Cited by 67
“Language models often capture important properties of the training data in their internal representations [35].”
BackgroundStorSeismic: A New Paradigm in Deep Learning for Seismic Processing
Harsuko, R., & Alkhalifah, T. A. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3216660 · Cited by 71
“As [27] highlighted, a pre-training followed by fine-tuning workflow constitute the components of BERT, where the former supposed to equip BERT with task-independent knowledge on, for example, sentences.”
MethodTransformers for Molecular Property Prediction: Lessons Learned from the Past Five Years
Sultan, A., Sieg, J., Mathea, M., & Volkamer, A. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00747 · Cited by 90
An automatic Alzheimer’s disease classifier based on spontaneous spoken English
Bertini, F., Allevi, D., Lutero, G., Calzà, L., & Montesi, D. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2021.101298 · Cited by 66
Lauscher, A., Majewska, O., Ribeiro, L. F. R., Gurevych, I., Rozanov, N., & Glavaš, G. (2020) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2020.deelio-1.5 · Cited by 102
MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction
Zhang, L., Chen, Q., Wang, W., Deng, C., Zhang, S., Li, B., Wang, W., & Cao, X. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.34 · Cited by 88
Findings of the Association for Computational Linguistics: EMNLP 2020
Dumitrescu, S., Avram, A.-M., & Pyysalo, S. (2022) · UTUPub (University of Turku) · Cited by 72
CDLM: Cross-Document Language Modeling
Caciularu, A., Cohan, A., Beltagy, I., Peters, M., Cattan, A., & Dagan, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.225 · Cited by 55
Extracting Latent Steering Vectors from Pretrained Language Models
Subramani, N., Suresh, N., & Peters, M. E. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.48 · Cited by 426
Deep learning for opinion mining and topic classification of course reviews
Koufakou, A. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11736-2 · Cited by 64
“BERT (and as a consequence any similar models) has quickly become the de facto baseline in NLP experiments (Rogers et al., 2020).”
BFCN: A Novel Classification Method of Encrypted Traffic Based on BERT and CNN
Shi, Z., Luktarhan, N., Song, Y., & Tian, G. (2023) · Electronics · doi:10.3390/electronics12030516 · Cited by 68
“1 (SQuAD) challenge [27], the top-level contest of machine reading comprehension, outperforming humans across the board in two evaluation metrics and excelling in eleven different NLP tasks.”
BackgroundMethodde Vries, W., van Cranenburgh, A., & Nissim, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.389 · Cited by 98
“…a language model based on the transformer architecture trained on predicting masked tokens and relations between sentences, has not only improved the state of the art results for many NLP tasks, but has impacted the field so much that we can already talk about ‘BERTology’ (Rogers et al., 2020).”
BackgroundInfluentialMask-guided BERT for few-shot text classification
Liao, W., Liu, Z., Dai, H., Wu, Z., Zhang, Y., Huang, X., Chen, Y., Jiang, X., Liu, D., Zhu, D., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128576 · Cited by 83
“its variants’ inherent properties and behaviors) [36]–[38], we propose the Mask-BERT framework to enhance few-shot learning capabilities of BERT-based models.”
MethodLet’s Play Mono - Poly : BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses
Soler, A. G., & Apidianaki, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00400 · Cited by 87
“The bulk of this interpretation work relies on probing tasks which serve to predict linguistic properties from the representations generated by the models (Linzen, 2018; Rogers et al., 2020).”
BackgroundGENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2023) · doi:10.1101/2023.06.12.544594 · Cited by 137
“This trend resonates with prior studies in Natural Language Processing (NLP) [41] and protein modeling [42].”
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
Ushio, A., Espinosa-Anke, L., Schockaert, S., & Camacho-Collados, J. (2021) · doi:10.18653/v1/2021.acl-long.280 · Cited by 116
“For a more complete overview on analyses of the different properties of transformer-based LMs, we refer to Rogers et al. (2021).”
BackgroundIeremie, I., Ewing, R. M., & Niranjan, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac104 · Cited by 65
Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks
Dong, H., Cheng, Z., He, X., Zhou, M., Zhou, A., Zhou, F., Liu, A., Han, S., & Zhang, D. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/761 · Cited by 93
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Swiatczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 54
“(1) is the most commonly used method with current protein LMs for explaining black-box LMs 25,40,48,56,102 , but it can only yield shallow correlational patterns that do not have a transparent and straightforward connection to biologically explanatory sequence-function rules.”
BackgroundMethodJahan, I., Laskar, M. T. R., Peng, C., & Huang, J. (2023) · arXiv · doi:10.48550/arxiv.2306.04504 · Cited by 57
“The rapid growth of language models (Rogers et al., 2021; Zhou et al., 2023) in the field of Natural Language Processing (NLP) in recent years has led to significant advancements in various domains, including the biomedical domain (Kalyan et al.”
BackgroundTR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.463 · Cited by 47
Based on billions of words on the internet, people = men
Bailey, A. H., Williams, A., & Cimpian, A. (2022) · Science Advances · doi:10.1126/sciadv.abm2463 · Cited by 53
Visualizing and Understanding Patch Interactions in Vision Transformer
Ma, J., Bai, Y., Zhong, B., Zhang, W., Yao, T., & Mei, T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270479 · Cited by 80
An Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 84
“Empirical studies show that pre-training tasks help LMs learn the linguistic knowledge such as the grammar [97] and the semantic role [99].”
BackgroundCommonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 104
A fine-grained deep learning model using embedded-CNN with BiLSTM for exploiting product sentiments
Ahmed, Z., & Wang, J. (2022) · Alexandria Engineering Journal · doi:10.1016/j.aej.2022.10.037 · Cited by 60
Mallory, F. (2023) · Ergo an Open Access Journal of Philosophy · doi:10.3998/ergo.4668 · Cited by 44
Searle, T., Ibrahim, Z. M., Teo, J., & Dobson, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104358 · Cited by 83
“, 2021) compared to only opendomain pre-training, offers improvements inline with prior research(Rogers et al., 2020).”
MethodResultEgeria: Efficient DNN Training with Knowledge-Guided Layer Freezing
Wang, Y., Sun, D., Chen, K., Lai, F., & Chowdhury, M. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3587451 · Cited by 88
“It is widely recognized that the front layers converge faster [75, 79, 95, 96], and Egeria can handle exceptions with the aforementioned module-based freezing and unfreezing mechanism.”
BackgroundDynamic Contextualized Word Embeddings
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2021) · doi:10.18653/v1/2021.acl-long.542 · Cited by 75
“…various types of dynamic word embeddings have been proposed: diachronic word embeddings for temporal semantic change (Bamler and Mandt, 2017; Rosenfeld and Erk, 2018; Rudolph and Blei, 2018; Yao et al., 2018; Gong et al., 2020) and personalized word embeddings for social semantic variation…”
BackgroundChanging the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
Self-citation(2020) · doi:10.18653/v1/2020.findings-emnlp.190 · Cited by 51
Explanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 91
“Some models, e.g., Naive Bayes, logistic regression, and decision trees, are self-explaining (SE) (Danilevsky et al., 2020), also referred to as transparent (Adadi and Berrada, 2018) or inherently interpretable (Rudin, 2019).”
MethodSinclair, A., Jumelet, J., Zuidema, W., & Fernández, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00504 · Cited by 48
“The rationale behind this paradigm is that if speakers are sensitive to sentence structure independently from sentence content, then it is reasonable to assume that such structural information is an integral part of the representations built during processing.”
BackgroundMethodSentiment analysis using deep learning techniques: a comprehensive review
Sahoo, C., Wankhade, M., & Singh, B. K. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00308-2 · Cited by 65
Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps
Barkan, O., Hauon, E., Caciularu, A., Katz, O., Malkiel, I., Armstrong, O., & Koenigstein, N. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482126 · Cited by 102
“co-occurrences) during pre-training [11, 32].”
User requirements analysis of new energy vehicles based on improved Kano model
Yang, Y., Li, Q., Li, C., & Qin, Q. (2024) · Energy · doi:10.1016/j.energy.2024.133134 · Cited by 54
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
Automated Extraction and Presentation of Data Practices in Privacy Policies
Bui, D., Shin, K. G., Choi, J., & Shin, J. (2021) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2021-0019 · Cited by 82
“The neural networks leverage the syntactical and vocabulary knowledge in word embeddings which were trained with very large datasets [52].”
WRENCH: A Comprehensive Benchmark for Weak SupervisionPreprint
Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., & Ratner, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11377 · Cited by 88
“For the end model, pre-trained language models are more suitable end models, as it can capture general semantics and syntactic information [81] which will benefit the downstream tasks.”
MethodVLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge
Ravi, S., Chinchure, A., Sigal, L., Liao, R., & Shwartz, V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00121 · Cited by 82
“Most existing methods rely on world knowledge implicitly encoded by language models, which often lacks in both accuracy and coverage [32].”
MethodImproving Neural Topic Models using Knowledge Distillation
Hoyle, A., Goel, P., & Resnik, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.137 · Cited by 58
On the Ability and Limitations of Transformers to Recognize Formal Languages
Bhattamishra, S., Ahuja, K., & Goyal, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.576 · Cited by 48
“Much effort has been devoted to understand the inner workings and intermediate representations of pre-trained models; Rogers et al. (2020) is a recent survey.”
BackgroundSmadi, B. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.107921 · Cited by 51
Hao, Y., Mendelsohn, S., Sterneck, R., Martinez, R., & Frank, R. (2020) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2020.cmcl-1.10 · Cited by 68
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
Frank, S., Bugliarello, E., & Elliott, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.775 · Cited by 35
“Significant efforts have been devoted to understanding what is learned by text-only pretrained language models (Rogers et al., 2020), which can be categorised as probing based on diagnostic classi-”
BackgroundMemorisation versus Generalisation in Pre-trained Language Models
Tänzer, M., Ruder, S., & Rei, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.521 · Cited by 98
“the types of relational or linguistic knowledge (Tenney et al., 2019; Jawahar et al., 2019; Rogers et al., 2020).”
BackgroundCoding energy knowledge in constructed responses with explainable NLP models
Gombert, S., Mitri, D. D., Karademir, O., Kubsch, M., Kolbe, H., Tautz, S., Grimm, A., Bohm, I., Neumann, K., & Drachsler, H. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12767 · Cited by 57
Gubelmann, R. (2023) · Grazer Philosophische Studien · doi:10.1163/18756735-00000182 · Cited by 54
Short-term electric vehicle battery swapping demand prediction: Deep learning methods
Wang, S., Chen, A., Wang, P., & Zhuge, C. (2023) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2023.103746 · Cited by 56
Natural language processing with transformers: a review
Țucudean, G., Bucos, M., Drăgulescu, B., & Căleanu, C. D. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2222 · Cited by 71
Codified audio language modeling learns useful representations for music information retrievalPreprint
Castellon, R., Donahue, C., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05677 · Cited by 79
“Unlike conventional pre-training, where the strongest representations for transfer learning typically lie at the end of the model [26], the strongest representations from pre-trained language models tend to lie towards the middle of the network [27–30].”
BackgroundLanguage Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46
“…many results and ideas from the inter-pretability literature (R ¨ auker et al., 2023), especially in topics related to probing (Belinkov, 2022), BERTology (Rogers et al., 2021), the linearity hypothesis and superposition (Elhage et al., 2022b), and mechanistic interpretability (Olah et al., 2020).”
BackgroundLanguage models: past, present, and future
Li, H. (2022) · Communications of the ACM · doi:10.1145/3490443 · Cited by 45
“13,16,29 Pre-trained language models (without fine-tuning), such as BERT and GPT-3, contain a large amount of factual knowledge.”
Frequency Effects on Syntactic Rule Learning in Transformers
Wei, J., Garrette, D., Linzen, T., & Pavlick, E. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.72 · Cited by 42
“While previous studies have attempted to address such questions, particularly in relation to BERT (Rogers et al., 2020), prior work has generally not analyzed the relationship between the model’s pre-training data and its behavior.”
ResultJahan, I., Laskar, M. T. R., Peng, C., & Huang, J. (2023) · The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks · doi:10.18653/v1/2023.bionlp-1.30 · Cited by 40
Merrill, W., Goldberg, Y., Schwartz, R., & Smith, N. A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00412 · Cited by 44
“Rather, analysis of massive language models has revealed that, to some degree, knowledge of syntactic and semantic dependencies can emerge without explicit supervision (Rogers et al., 2020; Tenney et al., 2019).”
BackgroundBARThez: a Skilled Pretrained French Sequence-to-Sequence Model
Eddine, M. K., Tixier, A. J. ‐., & Vazirgiannis, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.740 · Cited by 106
“The study of why and how BERT works so well has now its own dedicated research field, known as BERTology (Rogers et al., 2020).”
BackgroundUnveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Ma, W., Liu, S., Zhao, M., Xie, X., Wang, W., Hu, Q., Zhang, J., & Liu, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664606 · Cited by 74
“Probing [ 6 , 57 , 79 ] is one of the most prominent techniques widely leveraged for interpretability.”
Goldstein, A., Ham, E., Schain, M., Nastase, S. A., Aubrey, B., Zada, Z., Grinstein-Dabush, A., Gazula, H., Feder, A., Doyle, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-65518-0 · Cited by 75
ABNIRML: Analyzing the Behavior of Neural IR Models
MacAvaney, S., Feldman, S., Goharian, N., Downey, D., & Cohan, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00457 · Cited by 78
“ined what characteristics contextualized language models learn in general (Liu et al., 2019; Rogers et al., 2020; Loureiro et al., 2020), but it remains unclear if these qualities are valuable to the ad-hoc ranking task specifically.”
BackgroundActive Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates
Shelmanov, A., Puzyrev, D., Kupriyanova, L., Belyakov, D., Larionov, D., Khromov, N., Kozlova, O., Artemova, E., Dylov, D. V., & Panchenko, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.145 · Cited by 81
Using Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks
Xia, T., Wang, Y., Tian, Y., & Chang, Y. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449988 · Cited by 64
“As mentioned before, we only add such prior knowledge in the attention of the first BERT layer, as that layer’s attention is broad and uninformed [8, 33].”
MethodInfluentialBERT Embeddings for Automatic Readability Assessment
Imperial, J. M. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_069 · Cited by 87
“head attention layers are information-rich, specifically on semantic and syntactic knowledge (Rogers et al., 2020), due to the nature of its training.”
BackgroundMethodInfluentialLee, J., Soleimani, F., Irish, I., Hosmer, J., Soylu, M. Y., Finkelberg, R., & Chatterjee, S. (2022) · Online Learning · doi:10.24059/olj.v26i1.3060 · Cited by 51
“To effectively train the proposed ML model, we used a transformer-based deep learning model referred to as Bidirectional Encoder Representations from Transformers (BERT), which pre-trains and fine-tunes relevant text data (for an overview see Rogers, Kovaleva, Rumshisky, 2020).”
MethodDiet code is healthy: simplifying programs for pre-trained models of code
Zhang, Z., Zhang, H., Shen, B., & Gu, X. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549094 · Cited by 41
“Besides our work, there have been other studies that also try to explain the mechanisms of pre-trained models for code [1, 30, 32, 38].”
Result(2020) · doi:10.18653/v1/2020.conll-1.17 · Cited by 41
“The preponderance of research surrounding analysis and inter-pretability of BERT has been dubbed BERTology, after the poster-child of the contextual revolution (cf. Rogers et al., 2020 for an excellent survey).”
BackgroundOn the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Ravichander, A., Hovy, E. H., Suleman, K., Trischler, A., & Cheung, J. C. K. (2020) · Joint Conference on Lexical and Computational Semantics · Cited by 97
BERT Busters: Outlier Dimensions that Disrupt Transformers
Kovaleva, O., Kulshreshtha, S., Rogers, A., & Rumshisky, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.300 · Cited by 153
“See overviews by Ganesh et al. (2020) and Rogers et al. (2020).”
Self-citationUniTE: Unified Translation Evaluation
Wan, Y., Liu, D., Yang, B., Zhang, H., Chen, B., Wong, D. F., & Chao, L. S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.558 · Cited by 90
“First, our UniTE model can benefit from layer-coordinated semantical interactions inside every one of PLM layers, which is proven effective on capturing diverse linguistic features (He et al., 2018; Lin et al., 2019; Jawahar et al., 2019; Tenney et al., 2019; Rogers et al., 2020).”
MethodAdaptive Disentangled Transformer for Sequential Recommendation
Zhang, Y. J., Wang, X. Y., Chen, H., & Zhu, W. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599253 · Cited by 64
“…adopted multi-head attention within different layers can capture the disentangled information related to different aspects of sequence data [10, 35, 45], and lack of explicit regularization for layer-wise disentanglement, which may not take the advantage of disentangled representation in…”
BackgroundResultInfluentialPolysemy—Evidence from Linguistics, Behavioral Science, and Contextualized Language Models
Haber, J., & Poesio, M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00500 · Cited by 90
Levkovich, I., & Omar, M. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02134-3 · Cited by 57
“While the standard BERT model does not align with conventional definitions of ‘large language models,’ its advanced bidirectional architecture and applications have significantly contributed to advancements in suicide detection and risk assessment, warranting its inclusion among prominent LLMs [15].”
Transformer networks of human conceptual knowledge.
Bhatia, S., & Richie, R. (2024) · Psychological Review · doi:10.1037/rev0000319 · Cited by 59
Interpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 80
“The attention pattern behind BERT encodes meaning linguistic knowledge [154], which cannot be easily observed in LSTM based models.”
BackgroundQian, K., Liu, S., Li, T., Raković, M., Li, X., Guan, R., Molenaar, I., Nawaz, S., Swiecki, Z., Yan, L., et al. (2026) · Computers & Education · doi:10.1016/j.compedu.2025.105448 · Cited by 39
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 79
“While the progress in NLP tasks due to LMs is clear, the reasons behind this success are not as well understood (Rogers et al., 2021; McCoy et al., 2019), and there are also important downsides.”
BackgroundA Unified Framework for Multi-Domain CTR Prediction via Large Language Models
Fu, Z., Li, X., Wu, C., Wang, Y., Dong, K., Zhao, X., Zhao, M., Guo, H., & Tang, R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3698878 · Cited by 82
To BAN or Not to BAN: Bayesian Attention Networks for Reliable Hate Speech Detection
Miok, K., Škrlj, B., Zaharie, D., & Robnik‐Šikonja, M. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09826-9 · Cited by 53
Online grooming detection: A comprehensive survey of child exploitation in chat logs
Borj, P. R., Raja, K., & Bours, P. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110039 · Cited by 38
Comparative analysis of model compression techniques for achieving carbon efficient AI
Paula, E., Soni, J., Upadhyay, H., & Lagos, L. E. (2025) · Scientific Reports · doi:10.1038/s41598-025-07821-w · Cited by 53
“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks
Liu, J., Capurro, D., Nguyen, A. N., & Verspoor, K. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104149 · Cited by 61
The MultiBERTs: BERT Reproductions for Robustness Analysis
Sellam, T., Yadlowsky, S., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I., Eisenstein, J., Das, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.16163 · Cited by 37
“BERT (Devlin et al., 2019) is a particularly popular choice: it has been widely adopted in academia and industry, and aspects of its performance have been reported in thousands of research papers (see e.g., Rogers et al., 2020, for an overview).”
BackgroundWhy concepts are (probably) vectors
Piantadosi, S. T., Muller, D. C. Y., Rule, J., Kaushik, K., Gorenstein, M. I., Leib, E. R., & Sanford, E. M. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.06.011 · Cited by 87
“More recent vector models derived from text prediction, like word2vec and GloVe, learn vectors whose distance or similarity judgments replicate those of people [40 – 42].”
Method
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.