A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 201–300 of 1,014
Bevilacqua, M., Oketch, K., Qin, R., Stamey, W., Zhang, X., Gan, Y., Yang, K., & Abbasi, A. (2024) · ACM Transactions on Information Systems · doi:10.1145/3702639 · Cited by 55
“Third, it could be that because the transformer PLMs are more accurate (as per results in Table 4), they are less prone to over-fitting, and hence, are more capable of generalizing to the out-of-sample GPT text and assessing its quality.”
Dodrio: Exploring Transformer Models with Interactive Visualization
Wang, Z. J., Turko, R., & Chau, D. H. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.16 · Cited by 57
“In particular, research has shown that transformerbased models have learned to represent semantic knowledge and lexical structure in text (Rogers et al., 2020).”
BackgroundInfluentialFEEL-IT: Emotion and Sentiment Classification for the Italian Language
Bianchi, F., Nozza, D., & Hovy, D. (2021) · Archivio istituzionale della Ricerca - Bocconi · Cited by 37
Is it a good move? Mining effective tutoring strategies from human–human tutorial dialogues
Lin, J., Singh, S., Sha, L., Tan, W., Lang, D., Gašević, D., & Chen, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.09.001 · Cited by 56
Large Language Models are Not Models of Natural Language: They are Corpus Models
Veres, C. (2022) · IEEE Access · doi:10.1109/access.2022.3182505 · Cited by 50
“ing that has affectionately come to be known as Bertology [39], is attempting to do just that.”
BackgroundSelf-Explaining Structures Improve NLP ModelsPreprint
Sun, Z., Fan, C. C., Han, Q., Sun, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01786 · Cited by 62
“…al., 2015; Shi et al., 2016; Greff et al., 2016; Strobelt et al., 2016), and applying variants of the attention mechanism (Bahdanau et al., 2014) to interpret model behaviors (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Rogers et al., 2020).”
MethodDesbordes, T., Lakretz, Y., Chanoine, V., Oquab, M., Badier, J., Trébuchon, A., Carron, R., Bénar, C., Dehaene, S., & King, J.-R. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.1163-22.2023 · Cited by 40
Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA
Staliūnaitė, I., & Iacobacci, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.573 · Cited by 50
On Attention Redundancy: A Comprehensive Study
Bian, Y., Huang, J., Cai, X., Yuan, J., & Church, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.72 · Cited by 68
“Existing literature analyzes multi-layer multi-head self-attention architectures based language models (e.g., BERT) from different aspects including word/token embedding latent space, linguistic knowledge interpretability, attention mechanism, and so on (Rogers et al., 2020).”
BackgroundVietnamese hate and offensive detection using PhoBERT-CNN and social media streaming data
Tran, K. Q., Nguyen, A. T., Hoang, P. G., Luu, C. D., Do, T.-H., & Nguyen, K. V. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07745-w · Cited by 44
“BERT and its variations (BERTology) [63–65] such as RoBERTa [27], XLM-R [28], PhoBERT [36] have almost dominated and asserted their strength on natural language processing tasks, even for Vietnamese hate speech detection tasks [10, 21, 22].”
BackgroundA Systematic Analysis of Morphological Content in BERT Models for Multiple LanguagesPreprint
Edmiston, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03032 · Cited by 48
“With the recent success of Transformer-style architectures (Vaswani et al., 2017) like BERT (Devlin et al., 2018) on many natural language processing tasks, a considerable amount of research has gone into investigating the inner workings of these models, a research program sometimes dubbed “BERTology” (Rogers et al.,…”
BackgroundInfluentialAnalyzing how BERT performs entity matching
Paganelli, M., Buono, F. D., Baraldi, A., & Guerra, F. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3529337.3529356 · Cited by 60
Gaia: An AI-enabled genomic context–aware platform for protein sequence annotation
Jha, N., Kravitz, J., West-Roberts, J., Lu, C., Camargo, A. P., Roux, S., Cornman, A., & Hwang, Y. (2025) · Science Advances · doi:10.1126/sciadv.adv5109 · Cited by 47
“We use representations from the middle layer of gLM2, as previous studies have shown an improved transfer-learning ability compared to the final layer ( 36 , 37 ).”
FIND: Human-in-the-Loop Debugging Deep Text Classifiers
Lertvittayakumjorn, P., Specia, L., & Toni, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.24 · Cited by 55
Lim, S., & Lee, Y. O. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412555 · Cited by 41
“Kovaleva et al.[32] showed that most self-attention heads contain trivial information or the heads sometimes attend to the same location.”
BackgroundChen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.31.526427 · Cited by 43
“This is probably because the hidden states of the last layer are mostly related to predicting the masked tokens during pre-training 30 , while the intermediate layers preserve more contextual information.”
BackgroundKumar, B. V. P., & Sadanandam, M. (2024) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/150105 · Cited by 35
Extracting Sentence Embeddings from Pretrained Transformer Models
Stankevičius, L., & Lukoševičius, M. (2024) · Applied Sciences · doi:10.3390/app14198887 · Cited by 56
Deep Learning-Based Knowledge Graph Generation for COVID-19
Kim, T., Yun, Y. I., & Kim, N. (2021) · Sustainability · doi:10.3390/su13042276 · Cited by 43
Das, P., & Varshney, L. R. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2022.3141365 · Cited by 45
“These visualizations are examples of displaying so-called BERTology analyses [39], which try to delve into what information/knowledge is present in Transformer-type neural architectures after training.”
BackgroundGraphCode2Vec: generic code embedding via lexical and program dependence analyses
Ma, W., Zhao, M., Soremekun, E., Hu, Q., Zhang, J. M., Papadakis, M., Cordy, M., Xie, X., & Traon, Y. L. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528456 · Cited by 54
Probing BERT in Hyperbolic SpacesPreprint
Chen, B., Fu, Y., Xu, G., Xie, P., Tan, C., Chen, M., & Jing, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03869 · Cited by 58
“Probing BERT Recently, there are increasing interests in finding linguistic information encoded in BERT (Rogers et al., 2020).”
BackgroundBERTnesia: Investigating the capture and forgetting of knowledge in BERT
Wallat, J., Singh, J., & Anand, A. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.17 · Cited by 57
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Gupta, P., & Jaggi, M. (2021) · doi:10.18653/v1/2021.acl-long.408 · Cited by 51
“We also refer the reader to Mickus et al. (2020) for a discussion on the semantic properties of contextual models (primarily BERT) as well as Rogers et al. (2020), a survey on different works exploring the inner workings of BERT including its semantic properties.”
BackgroundProton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing
Wang, L., Qin, B., Hui, B., Li, B., Yang, M., Wang, B., Li, B., Sun, J., Huang, F., Si, L., et al. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539305 · Cited by 66
“The success of PLMs has led to a large number of studies investigating and interpreting the rich knowledge that PLMs learn implicitly during pre-training [21, 22, 29, 45].”
Background(2020) · doi:10.18653/v1/2020.findings-emnlp.112 · Cited by 32
Self-citationWhy Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Wei, C., Xie, S. M., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2106.09226 · Cited by 32
“, syntactic [35, 11]) by evaluating the performance of simple classifiers, or probes, on the representations [7, 12, 34, 27, 15].”
Giordano, V., Puccetti, G., Chiarello, F., Pavanello, T., & Fantoni, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120499 · Cited by 36
Asking without Telling: Exploring Latent Ontologies in Contextual Representations
Michael, J., Botha, J. A., & Tenney, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.552 · Cited by 56
Explainable Natural Language Processing
Søgaard, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01118ed1v01y202107hlt051 · Cited by 38
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.373 · Cited by 83
“As current neural-based models have an end-to-end, black-box nature, existing studies have adopted several strategies to interpret their inner workings (Car-valho et al., 2019; Rogers et al., 2020; Bra¸soveanu and Andonie, 2020).”
BackgroundMethodInfluentialKjell, O., Giorgi, S., & Schwartz, H. A. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/293kt · Cited by 39
TS-Fastformer: Fast Transformer for Time-series Forecasting
Lee, S., Hong, J., Liu, L., & Choi, W. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3630637 · Cited by 52
“BERT has become the standard for NLP tasks [ 29 ].”
MethodLarge language models and linguistic intentionality
Grindrod, J. (2024) · Synthese · doi:10.1007/s11229-024-04723-8 · Cited by 50
“We can then explore whether each self-attention head is sensitive to particular linguistic properties (for an overview, see: (Rogers et al., 2021).”
Raza, S., Schwartz, B., & Rosella, L. C. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04751-6 · Cited by 39
“Transformer-based models like Google’s BERT [41], XLNet [57], and Facebook’s BART [58] have demonstrated outstanding results in a wide range of NLP tasks [59], such as question answering, classification and related language modelling tasks.”
MethodAraLegal-BERT: A pretrained language model for Arabic Legal text
Al‐Qurishi, M., Alqaseemi, S., & Souissi, R. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.31 · Cited by 47
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model BeliefsPreprint
Hase, P., Diab, M., Çelikyılmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., & Iyer, S. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13654 · Cited by 88
“Much past work has explored how information is stored and represented in pretrained language models (Rogers et al., 2020), though few discuss what qual-ifies information as a model belief.”
BackgroundExtracting Latent Steering Vectors from Pretrained Language Models
Subramani, N., Suresh, N., & Peters, M. E. (2022) · arXiv · doi:10.48550/arxiv.2205.05124 · Cited by 31
“We suspect this is due to the fact that the embedding layer solely captures low-level information (Lin et al., 2019; Ethayarajh, 2019; Rogers et al., 2020).”
BackgroundHardalov, M., Mihaylov, T., Zlatkova, D., Dinkov, Y., Koychev, I. K., & Nakov, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.438 · Cited by 111
Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT
Papadimitriou, I., Chi, E. A., Futrell, R., & Mahowald, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.215 · Cited by 66
Multi-Task Attentive Residual Networks for Argument Mining
Galassi, A., Lippi, M., & Torroni, P. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3275040 · Cited by 37
“Looking outside the context of AM, BERT is a very popular model across all the NLP tasks [51], but it is also very resource-demanding, consisting of more than 110 millions parameters in its base implementation.”
BackgroundDrop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining
Zhang, N., Deng, S., Cheng, X., Chen, X., Zhang, Y., Zhang, W., & Chen, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/552 · Cited by 37
Word Acquisition in Neural Language Models
Chang, T. A., & Bergen, B. K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00444 · Cited by 26
“Recently, there has been substantial research evaluating language models using psycholinguistic approaches, reflecting a broader goal of interpreting language models (BERTology; Rogers et al., 2020).”
BackgroundChatGPT and Deepseek: Can They Predict the Stock Market and Macroeconomy?Preprint
Chen, J., Tang, G., Zhou, G., & Zhu, W. Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4660148 · Cited by 32
Sentence Analogies: Linguistic Regularities in Sentence Embeddings
Zhu, X., & de Melo, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.300 · Cited by 55
“Rücklé et al. (2018) improved the average pooling method by concatenating different power means of word embeddings.”
MethodMATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
Rashid, A., Lioutas, V., & Rezagholizadeh, M. (2021) · doi:10.18653/v1/2021.acl-long.86 · Cited by 49
“Increasing model parameters depends on availability of computational resources and PLMs are typically trained on unlabeled data which is cheaper to obtain.”
BackgroundInfluentialAwakening Latent Grounding from Pretrained Language Models for Semantic Parsing
Liu, Q., Yang, D., Zhang, J., Guo, J., Zhou, B., & Lou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.100 · Cited by 50
“According to prior studies (Rogers et al., 2020), most existing work focuses on syntactic structures such as part of speech (Liu et al., Figure 1: Typical scenarios for grounding, here the linguistic tokens “george washington” can be grounded into different real-world concepts.”
BackgroundKEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 43
Large language models know how the personality of public figures is perceived by the general public
Cao, X., & Kosiński, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-57271-z · Cited by 57
“They are known to encode words’ meanings 42 , including information about group stereotypes 43–45 .”
BackgroundSA2SL: From Aspect-Based Sentiment Analysis to Social Listening System for Business Intelligence
Phan, L. L., Pham, P. H., Nguyen, K. T.-T., Huynh, S. K., Nguyen, T. T., Nguyen, L. T., Huynh, T. V., & Nguyen, K. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82147-0_53 · Cited by 40
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica
Hayati, S. A., Kang, D., & Ungar, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.510 · Cited by 31
Conditional probing: measuring usable information beyond a baseline
Hewitt, J., Ethayarajh, K., Liang, P., & Manning, C. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.122 · Cited by 80
“Neural language models have become the foundation for modern NLP systems (Devlin et al., 2019; Radford et al., 2018), but what they understand about language, and how they represent that knowledge, is still poorly understood (Belinkov and Glass, 2019; Rogers et al., 2020).”
BackgroundSharing Our Concepts with Machines
Butlin, P. (2023) · Erkenntnis · doi:10.1007/s10670-021-00491-w · Cited by 22
“There is also some evidence that Transformer-based language models represent syntactic properties of inputs (Rogers et al., 2020).”
BackgroundDeep learning models to study sentence comprehension in the human brain
Arana, S., Lerousseau, J. P., & Hagoort, P. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2198245 · Cited by 37
“Since these layers are closest to the model output layer, they preserve the information that is most useful for the downstream linguistic task the model is trained on (Rogers et al., 2020).”
CLiMP: A Benchmark for Chinese Language Model Evaluation
Xiang, B., Yang, C., Li, Y., Warstadt, A., & Kann, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.242 · Cited by 85
Robust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network
Lovelace, J., Newman-Griffis, D. R., Vashishth, S., Lehman, J. F., & Rosé, C. P. (2021) · doi:10.18653/v1/2021.acl-long.82 · Cited by 43
“BERT as a Knowledge Base: Recent work (Petroni et al., 2019; Jiang et al., 2020; Rogers et al., 2020) has utilized the masked-languagemodeling (MLM) objective to probe the knowledge contained within pre-trained models using fill-in-the-blank prompts (e.g. “Dante was born in [MASK]”).”
MethodYu, S., Gu, C., Huang, K., & Li, P. (2024) · Science Advances · doi:10.1126/sciadv.adn7744 · Cited by 39
“However, NSP’s contribution to general model performance has been in question ( 30 ); many recent BERT-based models even dropped this task from pretraining ( 31 , 32 ).”
BackgroundT-cell receptor binding prediction: A machine learning revolution
Weber, A., Pélissier, A., & Martínez, M. R. (2024) · ImmunoInformatics · doi:10.1016/j.immuno.2024.100040 · Cited by 36
“For instance, the emerging field of Bertology [86] aims to unravel the workings of the BERT model [59] and to understand its effectiveness and internal mechanisms.”
BackgroundRobust Lottery Tickets for Pre-trained Language Models
Zheng, R., Rong, B., Zhou, Y., Liang, D., Wang, S., Wu, W., Gui, T., Zhang, Q., & Huang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.157 · Cited by 54
“However, the magnitude-based pruning is not suitable for robustness-aware techniques (Vemparala et al., 2021; Sehwag et al., 20).”
AI-based analysis of social media language predicts addiction treatment dropout at 90 days
Curtis, B. L., Giorgi, S., Ungar, L., Vu, H., Yaden, D. B., Liu, T., Yadeta, K., & Schwartz, H. A. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01585-5 · Cited by 32
Stop Measuring Calibration When Humans Disagree
Baan, J., Aziz, W. F., Plank, B., & Fernández, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.124 · Cited by 109
ViHOS: Hate Speech Spans Detection for Vietnamese
Hoang, P. G., Luu, C. D., Tran, K. Q., Nguyen, K. V., & Nguyen, N. L.-T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.47 · Cited by 52
Exploring the Capability of ChatGPT in Test Generation
Yi, G., Chen, Z., Chen, Z., Wong, W. E., & Chau, N. (2023) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c60940.2023.00013 · Cited by 27
Ilinykh, N., & Dobnik, S. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.767971 · Cited by 28
“Since there is no clear cognitive motivation behind tremendously successful transformers (Rogers et al., 2020), a set of sophisticated methods is required to examine how information is processed and what is learned by such models.”
BackgroundSemeraro, A., Vilella, S., Improta, R., Duro, E. S. D., Mohammad, S. M., Ruffo, G., & Stella, M. (2025) · Behavior Research Methods · doi:10.3758/s13428-024-02553-7 · Cited by 49
“Furthermore, lexicons can also be tuned according to the domain or topic of interest – e.g., finance (Loughran & McDonald, 2011) – without needing considerable training like BERT or similar methods require (Rogers et al., 2020).”
Budakoglu, G., & Emekci, H. (2025) · IEEE Access · doi:10.1109/access.2025.3542334 · Cited by 38
“These limitations have been documented in the literature, as represented by A Primer in BERTology [1], in which the operational subtleties and limitations of BERT are elaborated as a seminal LLM.”
Protein Language Models: Applications and Perspectives
Leclercq, M., & Droit, A. (2025) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c00506 · Cited by 40
Improved biomedical word embeddings in the transformer era
Noh, J., & Kavuluru, R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103867 · Cited by 34
“Base models tend to have tens of millions of parameters [11] and using them without GPUs in low-resource settings such as smart devices used in edge computing or IoT is infeasible.”
BackgroundTowards Zero-Shot Knowledge Distillation for Natural Language Processing
Rashid, A., Lioutas, V., Ghaddar, A., & Rezagholizadeh, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.526 · Cited by 34
Xue, J., Shen, Q., Li, Y., Han, S., & Chu, X. (2021) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002066 · Cited by 29
LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond
Loureiro, D., Jorge, A., & Camacho-Collados, J. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103661 · Cited by 43
“…at different tasks, with causal models excelling at language generation and masked models at language understanding.1 BERT proved highly successfully at most NLP tasks (Rogers et al., 2020), and motivated the development of numerous derivative models, many of which we also explore in this work.”
BackgroundHow much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26
Representing Context in FrameNet: A Multidimensional, Multimodal Approach
Torrent, T. T., Matos, E. E. D. S., Belcavello, F., Viridiano, M., Gamonal, M. A., da Costa, A. D., & Marim, M. C. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.838441 · Cited by 46
Ghosh, K., & Senapati, A. (2024) · Natural language processing. · doi:10.1017/nlp.2024.28 · Cited by 34
“According to Rogers et al. (2020), TLMs gather syntactic knowledge, semantic knowledge, and world knowledge during the training procedure.”
Ren, M. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/97/20241406 · Cited by 39
“The core strength of LLMs lies in their deep neural networks, which learn rich representations of language from extensive training on diverse datasets, allowing them to generalize well across different tasks [5].”
Time series transformer for tourism demand forecasting
Yi, S., Chen, X., & Tang, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-15286-0 · Cited by 29
“BERT achieves state-of-the-art performance in multiple tasks 38 .”
Li, M., & Wang, Z. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.113855 · Cited by 43
Pandia, L., & Ettinger, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 22
“A good deal of prior work has focused on testing for linguistic knowledge in language models (Rogers et al., 2020).”
BackgroundLearning to Remove: Towards Isotropic Pre-trained BERT Embedding
Liang, Y., Cao, R., Zheng, J., Ren, J., & Gao, L. C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86383-8_36 · Cited by 30
Lee, S., Lee, D., Jang, S., & Yu, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.412 · Cited by 44
Open-domain Dialogue Generation: What We Can Do, Cannot Do, And Should Do Next
Kann, K., Ebrahimi, A., Koh, J. J., Dudy, S., & Roncone, A. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.13 · Cited by 42
Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3622758.3622882 · Cited by 23
“An emerging pattern in research is to simply try a generative model and see what it can do, a kind of research sometimes referred to as “BERTology” [88].”
BackgroundTransforming Fake News: Robust Generalisable News Classification Using Transformers
Blackledge, C., & Atapour–Abarghouei, A. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671970 · Cited by 38
Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense KnowledgePreprint
Nair, S., Srinivasan, M., & Meylan, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13057 · Cited by 49
“These richer word representations from contextualized embeddings have allowed researchers to ask which aspects of linguistic knowledge are incorporated in their vector spaces (See Rogers et al. (2020) for a review; see also Ethayarajh (2019)).”
Backgroundvan Duijn, M., van Dijk, B., Kouwenhoven, T., de Valk, W., Spruit, M., & vanderPutten, P. (2023) · Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2023.conll-1.25 · Cited by 24
Gupta, R. (2024) · Информатика Экономика Управление - Informatics Economics Management · doi:10.47813/2782-5280-2024-3-1-0311-0320 · Cited by 28
A Multi-task Approach to Neural Multi-label Hierarchical Patent Classification Using Transformers
Pujari, S. C., Friedrich, A., & Strötgen, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_34 · Cited by 46
“Recently, transformer-based neural language models such as BERT [8] have been shown to be highly effective for a variety of natural language processing tasks [33], following a “pre-train and fine-tune” approach.”
BackgroundOn the Gap between Adoption and Understanding in NLP
Bianchi, F., & Hovy, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.340 · Cited by 35
Distinct but correct: generating diversified and entity-revised medical response
Li, B., Sun, B., Li, S., Chen, E., Liu, H., Weng, Y., Bai, Y., & Hu, M. (2024) · Science China Information Sciences · doi:10.1007/s11432-021-3534-9 · Cited by 56
Analyzing Encoded Concepts in Transformer Language Models
Sajjad, H., Durrani, N., Dalvi, F., Alam, F., Khan, A., & Xu, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.225 · Cited by 24
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Ruder, S., Vulić, I., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.184 · Cited by 61
“this topic; a recent example is BERTology (Rogers et al., 2020).”
BackgroundEvaluation of Fake News Detection with Knowledge-Enhanced Language Models
Whitehouse, C., Weyde, T., Madhyastha, P., & Komninos, N. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19400 · Cited by 38
Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models
Hase, P., Diab, M., Çelikyılmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., & Iyer, S. V. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.199 · Cited by 80
“Much past work has explored how information is stored and represented in pretrained language models (Rogers et al., 2020).”
BackgroundFact-Driven Logical Reasoning for Machine Reading Comprehension
Ouyang, S., Zhang, Z., & Zhao, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29850 · Cited by 29
“However, according to diagnostic tests (Ettinger, 2020; Rogers et al., 2020), PLMs like BERT (Devlin et al.”
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 34
Probing Contextual Language Models for Common Ground with Visual Representations
Ilharco, G., Zellers, R., Farhadi, A., & Hajishirzi, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.422 · Cited by 37
“We refer to Be-linkov and Glass (2019) and Rogers et al. (2020) for a more comprehensive literature review.”
BackgroundMethodELECTRA for Neural Coreference Resolution in Italian
Guarasci, R., Minutolo, A., Damiano, E., Pietro, G. D., Fujita, H., & Esposito, M. (2021) · IEEE Access · doi:10.1109/access.2021.3105278 · Cited by 24
“ELECTRA has shown a better ability to capture contextual word representations, substantially outperforming, in its downstream performance, other models, like BERT, given the same model size, data, and compute [16].”
MethodMinot, J. R., Cheney, N., Maier, M., Elbers, D. C., Danforth, C. M., & Dodds, P. S. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3524887 · Cited by 23
“In the context of deep language models this can involve interrogating the functionality of speciic model layers (e.g., BERTology [61]), or investigating the impact of perturbations in data on outputs.”
Standards for Belief Representations in LLMs
Herrmann, D. A., & Levinstein, B. A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09709-6 · Cited by 83
“We support such efforts—but we also believe that, given the pace of development of models that are not honest or interpretable by design, belief measurement can play a core role in developing safe AI.”
Edwards, A., Camacho-Collados, J., de Ribaupierre, H., & Preece, A. D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.481 · Cited by 35
“Language models pre-trained on large amounts of text corpora form the foundation of today’s NLP (Gururangan et al., 2020; Rogers et al., 2020).”
Background
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.