A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
Items 501–600 of 1,014
Automatic Difficulty Classification of Arabic SentencesPreprint
Khallaf, N., & Sharoff, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04386 · Cited by 14
“In this study we only performed some ablation analysis, but because BERT-like models are more useful as the classifiers, we want to investigate their performance via probing for linguistic features following the BERTology framework (Rogers et al., 2020; Sharoff, 2021).”
MethodDiverse Genomic Embedding Benchmark for functional evaluation across the tree of lifePreprint
West-Roberts, J. A., Kravitz, J., Jha, N., Cornman, A., & Hwang, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.10.602933 · Cited by 20
“2.1 N ATURAL LANGUAGE EMBEDDING BENCHMARKS Embedding benchmarks (e.g. SentEval Conneau & Kiela (2018); BEIR Thakur et al. (2021); MTEB Muennighoff et al. (2023)) in natural language processing (NLP) aim to evaluate how the structure of word/sentence representations match the geometric structure of their semantics.”
MethodInfluentialDetecting Online Grooming By Simple Contrastive Chat Embeddings
Rezaee Borj, P., Raja, K., & Bours, P. (2023) · Proceedings of the 9th ACM International Workshop on Security and Privacy Analytics · doi:10.1145/3579987.3586564 · Cited by 11
“A plausible approach is, therefore, to resort to domain adaptation by learning the feature extractor where data is abundantly available and transferring it to similar problems at hand [9, 12].”
BackgroundWord Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation
Pezzelle, S., Takmaz, E., & Fernández, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00443 · Cited by 13
“…a lot of interest in understanding the inner mechanisms of BERT-like models (see the interpretability line of research referred to as BERTology ; Rogers et al., 2020) and the nature of their representations (Mickus et al., 2020; Westera and Boleda, 2019), comparably less attention has been paid…”
BackgroundA multi-facet analysis of BERT-based entity matching models
Paganelli, M., Tiano, D., & Guerra, F. (2023) · The VLDB Journal · doi:10.1007/s00778-023-00824-x · Cited by 22
“…analyses typically follow two main research directions: they adopt an experimental methodology to evaluate the contribution of specific architecture components (such as contextualized embedding or attention modules) or they examine the parameters of probing classifiers trained on top of them [40].”
InfluentialSikström, S., Valavičiūtė, I., Kuusela, I., & Evors, N. (2024) · Communications Psychology · doi:10.1038/s44271-024-00097-2 · Cited by 13
“…andsystemicbiasesthatmayre fl ectdiagnosticandsocialstereotypes 25 ,ithas undeniably made a signi fi cant breakthrough in the fi eld of NLP. BERT ’ s pioneering contribution lies in its introduction of unsupervised pre-trained models,allowingthemtodrawinsightsfromextensivesetsofunlabeledtext data 26 .”
Holford, D. L., Lopez-Lopez, E., Fasce, A., Karlsson, L. C., & Lewandowsky, S. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03416-4 · Cited by 12
“…in the analysis of natural language patterns, where similarities in linguistic features (vocabulary, semantics, structures, etc.) between different types of expressions can be measured by analysing how often those expressions are confused with each other (Qiu et al., 2020; Rogers et al., 2020).”
Advancing an LDA-GMM-CorEx topic model with prior domain knowledge in information systems research
Jiang, Y., Fu, M., Fang, J., Rossi, M., Wang, Y., & Tan, C. (2024) · Information & Management · doi:10.1016/j.im.2024.104097 · Cited by 12
Mhamed, M., Sutcliffe, R. F. E., & Feng, J. (2025) · Egyptian Informatics Journal · doi:10.1016/j.eij.2024.100601 · Cited by 11
“It has become one of the essential standard benchmarks for natural language processing technology [51].”
Suong, T. P. M., Nghiem, L. H., & Tran, T. K. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2025.2520213 · Cited by 11
“However, while BERT excels at understanding context, it does not inherently explain its predictions, a limitation that has spurred further research into model interpretability (Rogers, Kovaleva, and Rumshisky 2020).”
Mapping local government priorities: a web-mining approach for regional research
Schütz, M., Kriesch, L., & Losacker, S. (2025) · Regional Science Policy & Practice · doi:10.1016/j.rspp.2025.100240 · Cited by 12
ProtRNA: A protein-derived RNA language model by cross-modality transfer learning
Zhang, R., Ma, B., Xu, G., & Ma, J. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101371 · Cited by 14
Rani, R., Kumar, M., Epiphaniou, G., & Maple, C. R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130180 · Cited by 12
A review of deep learning approaches for drug synergy prediction in cancer
Li, L., Zhang, H., Zheng, C., & Su, Y. (2025) · npj Drug Discovery. · doi:10.1038/s44386-025-00034-1 · Cited by 14
“Besides, DFFNDDS 61 uses a BERT model 62 to convert drug SMILES into vectors, which combine with drug hashed atom pair fi nger-prints to form the drug features.”
Linking Entities to Unseen Knowledge Bases with Arbitrary Schemas
Vyas, Y., & Ballesteros, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.65 · Cited by 14
Yu, H., Evans, J. A., & Varshney, L. R. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14277 · Cited by 15
“Consider studying AlphaGo or analyzing attention matrices learned from a Transformer-based NLP model like BERT or GPT (Rogers et al., 2020).”
MethodSemantic Data Set Construction from Human Clustering and Spatial Arrangement
Majewska, O., McCarthy, D., van den Bosch, J. J. F., Kriegeskorte, N., Vulić, I., & Korhonen, A. (2021) · Computational Linguistics · doi:10.1162/coli_a_00396 · Cited by 15
“Indeed, a number of works have drawn attention to BERT’s subpar verbal reasoning abilities and its reliance on shallow heuristics in natural language inference and reading comprehension (McCoy, Pavlick, and Linzen 2019; Zellers et al. 2019; Si et al. 2019; Rogers et al. 2020; Sugawara et al. 2020).”
BackgroundFayyaz, M., Aghazadeh, E., Modarressi, A., Mohebbi, H., & Pilehvar, M. T. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.29 · Cited by 28
A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models
Ye, D., Lin, Y., Li, P., Sun, M., & Liu, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.57 · Cited by 18
Machine Learning Models for Predicting, Understanding, and Influencing Health Perception
Aka, A., & Bhatia, S. (2021) · Journal of the Association for Consumer Research · doi:10.1086/718456 · Cited by 13
Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment
Matero, M., Hung, A., & Schwartz, H. A. (2022) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2022.wassa-1.9 · Cited by 16
“…human-level predictions large representations are ideal and (3) The layers that boost performance all come from the top half of RoBERTa-large likely due to them including more semantic information than syntactic (Rogers et al., 2020), which could be more informative for modeling at the human-level.”
BackgroundBranco, P., Berger, S., Abdullah, T. B., Vachon‐Presseau, É., Cecchi, G., & Apkarian, A. V. (2022) · Pain · doi:10.1097/j.pain.0000000000002808 · Cited by 14
Figueroa, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.12.003 · Cited by 11
Li, B., Wisniewski, G., & Crabbé, B. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00531 · Cited by 20
“Three kinds of approaches have been used in the literature to analyze the inner working of neural networks (Rogers et al., 2020; Belinkov et al., 2020).”
BackgroundAttention weights accurately predict language representations in the brain
Lamarre, M., Chen, C., & Deniz, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.330 · Cited by 25
Privacy-Preserving Models for Legal Natural Language Processing
Yin, Y., & Habernal, I. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.14 · Cited by 12
“Their ability to ‘learn’ certain language properties (Rogers et al., 2020) and the need of having only a small amount of labeled data in the target domain for fine-tuning makes them superior to other approaches (Brown et al., 2020).”
iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summaries
Coscia, A. J., Holmes, L., Morris, W., Choi, J. S., Crossley, S. A., & Endert, A. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645142 · Cited by 19
“This could enable engineers to identify emergent representations of semantic knowledge and lexical structure that contribute to certain model scores [42]; e.g., identifying salient interactions between specific character spans across model layers and heads.”
BackgroundRICo: Reddit ideological communities
Ravi, K., & Vela, A. (2024) · Online Social Networks and Media · doi:10.1016/j.osnem.2024.100279 · Cited by 11
Pande, M., Budhraja, A., Nema, P., Kumar, P., & Khapra, M. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17605 · Cited by 12
How Does BERT Rerank Passages? An Attribution Analysis with Information Bottlenecks
Jiang, Z., Tang, R., Xin, J., & Lin, J. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.39 · Cited by 24
“Rogers et al. (2020) provide a thorough literature survey about what we already know about how BERT works and they’ve found different probing methods sometimes lead to contradictory interpretations.”
BackgroundWidespread use of National Academies consensus reports by the American public
Hicks, D. M., Zullo, M., Doshi, A., & Asensio, O. I. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2107760119 · Cited by 14
Li, T., Rashid, A., Jafari, A., Sharma, P., Ghodsi, A., & Rezagholizadeh, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.65 · Cited by 17
Fact-driven Logical Reasoning.Preprint
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · Cited by 11
The Linear Representation Hypothesis and the Geometry of Large Language Models
Park, K., Choe, Y. J., & Veitch, V. (2023) · arXiv · doi:10.48550/arxiv.2311.03658 · Cited by 11
“…significant body of work on linear representations for interpreting (probing) (e.g., Alain & Ben-gio, 2017; Kim et al., 2018; nostalgebraist, 2020; Rogers et al., 2021; Belinkov, 2022; Li et al., 2022; Geva et al., 2022; Nanda et al., 2023) and controlling (steering) (e.g., Wang et al., 2023;…”
BackgroundTowards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Hou, Y., Li, J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.299 · Cited by 90
La Revolución en la Creación Visual
Arias, M. C., Díaz, A. P., & Lara-Martínez, M. (2024) · VISUAL REVIEW International Visual Culture Review / Revista Internacional de Cultura Visual · doi:10.62161/revvisual.v16.5304 · Cited by 12
Textual form features for text readability assessment
Pan, W., Li, X., Chen, X., & Xu, R. (2024) · Natural language processing. · doi:10.1017/nlp.2024.50 · Cited by 14
“Regarding the feature BertSim , given that text representations are rich in syntactic and discourse knowledge as noted by Rogers et al. (2020), it is capable of capturing textual form complexity at the text level and assessing readability from the perspective of syntactic.”
Mukherjee, A., Hassija, V., Chamola, V., & Gupta, K. K. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3560333 · Cited by 19
“We have examined B LEURT and BERTS CORE , both contingent on B ERT as their neural foundation [34].”
ResDecode: Accelerating Large Language Models Inference via Residual Decoding Heads
Zeng, Z., Yu, J., Pang, Q., Wang, Z., Zhuang, H., Fan, Y., Shao, H., & Zou, X. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020074 · Cited by 14
Yuan, H., Ke, R., & Xie, X. (2025) · Journal of Asian Architecture and Building Engineering · doi:10.1080/13467581.2025.2540079 · Cited by 11
“In contrast, BERT’s dynamic embeddings allow it to capture the subtle differences in meaning based on the surrounding context, making it particularly suitable for complex tasks like sentiment analysis in social media (Rogers, Kovaleva, and Rumshisky 2021).”
Natural Language Processing with Transformer-Based Models: A Meta-Analysis
Munyao, C., & Ndia, J. (2025) · Journal on artificial intelligence · doi:10.32604/jai.2025.069226 · Cited by 11
“Another study demonstrated that knowledge distillation reduced model size by 60% while retaining 95% of the original performance, enabling more efficient deployment on edge devices [29].”
Query2Prod2Vec: Grounded Word Embeddings for eCommerce
Bianchi, F., Tagliabue, J., & Yu, B. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.20 · Cited by 13
Tracking Child Language Development With Neural Network Language Models
Sagae, K. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.674402 · Cited by 13
“…not be considered uninterpretable, and a growing body of research is dedicated specifically to understanding what neural language models learn (Rogers et al., 2020), this kind of work is still in its infancy, and not yet at a stage that can provide clear information about what specific kinds…”
BackgroundFutamata, K., Park, B., Yamamoto, R., & Tachibana, K. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-252 · Cited by 20
LARGE LANGUAGE MODELS FOR CIPHERS
Noever, D. A. (2023) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2023.14301 · Cited by 13
“Large language models fall into three broad categories: autoregressive (GPT-like [3-4]), auto-encoding ([33] BERT-like), and sequence-to-sequence ([34] T5/BART-like).”
BackgroundFinding Structure in One Child's Linguistic Experience
Wang, W., Vong, W. K., Kim, N., & Lake, B. M. (2023) · Cognitive Science · doi:10.1111/cogs.13305 · Cited by 30
“SAYCam offers an unprecedented snapshot of three children’s experiences, but it captures only a small fraction of their total linguistic input, preventing us from analyzing more complex linguistic phenomena (Belinkov & Glass, 2019; Linzen & Baroni, 2021; Rogers et al., 2021).”
BackgroundEmergent Modularity in Pre-trained Transformers
Zhang, Z., Zeng, Z., Lin, Y., Xiao, C., Wang, X., Han, X., Liu, Z., Xie, R., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.250 · Cited by 40
Efficient Transformer Knowledge Distillation: A Performance Review
Brown, N., Williamson, A., Anderson, T., & Lawrence, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.6 · Cited by 15
Bridging Information-Theoretic and Geometric Compression in Language Models
Cheng, E., Kervadec, C., & Baroni, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.762 · Cited by 39
Targeted marketing on social media: utilizing text analysis to create personalized landing pages
Çetinkaya, Y. M., Külah, E., Toroslu, İ. H., & Davulcu, H. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01213-0 · Cited by 18
“While there is no standard definition of phrase quality, it is beneficial to assess it based on several criteria, such as popularity, concordance, informativeness, and completeness, as outlined by J. Liu, Shang, and Han (2017).”
Verb Knowledge Injection for Multilingual Event Processing
Majewska, O., Vulić, I., Glavaš, G., Ponti, E. M., & Korhonen, A. (2021) · doi:10.18653/v1/2021.acl-long.541 · Cited by 14
“Recent probing experiments showed that they implicitly extract a non-negligible amount of linguistic knowledge from text corpora in an un-supervised fashion (Hewitt and Manning, 2019; Vuli´c et al., 2020; Rogers et al., 2020, inter alia ).”
BackgroundWhat About the Precedent: An Information-Theoretic Analysis of Common Law
Valvoda, J., Pimentel, T., Stoehr, N., Cotterell, R., & Teufel, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.181 · Cited by 17
Unsupervised Domain Adaptation for Event Detection using Domain-specific Adapters
Ngo, N. T., Phung, D., & Nguyen, T. H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.351 · Cited by 10
“As out-lined by Rogers et al. (2021), lower-level layers of BERT contain quite broad knowledge, thus encode more random distribution when projected into label space.”
BackgroundProbing Linguistic Information for Logical Inference in Pre-trained Language Models
Chen, Z., & Gao, Q. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21294 · Cited by 17
PhraseMap: Attention-Based Keyphrases Recommendation for Information Seeking
Tu, Y., Qiu, R., Wang, Y.-S., Yen, P.-Y., & Shen, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3225114 · Cited by 13
“As indicated in recent studies [68], [69], [70], [71], the middle and the latter attention layers capture the syntactic information and high-level semantics, respectively.”
ResultSemantic Color Mapping: A Pipeline for Assigning Meaningful Colors to Text
El‐Assady, M., Kehlbeck, R., Metz, Y., Schlegel, U., Sevastjanova, R., Sperrle, F., & Spinner, T. (2022) · doi:10.1109/visguides57787.2022.00008 · Cited by 15
“Since the information that gets captured in the different layers varies [29,33], one has to decide which layer is appropriate for a given use case.”
BackgroundA Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion
Lovelace, J., & Rosé, C. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.398 · Cited by 15
“While fine-tuning language models often improves downstream performance (Rogers et al., 2020), it increases the computational overhead of computing entity representations.”
A New Sentence-Based Interpretative Topic Modeling and Automatic Topic Labeling
Kozbagarov, O., Mussabayev, R., & Mladenović, N. (2021) · Symmetry · doi:10.3390/sym13050837 · Cited by 10
“In 2018 Google developed the generative transformer BERT model based on neural networks [17,18].”
BackgroundMethodPutting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords
Karidi, T., Zhou, Y., Schneider, N., Abend, O., & Srikumar, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.806 · Cited by 18
“Attempts have been made to characterize the information captured in contextualized representations (Rogers et al., 2020; Tenney et al., 2019; Liu et al., 2019), but some of the techniques used (e.g., probing classifiers) have been subject to criticism for their indirectness.”
BackgroundToxCCIn: Toxic Content Classification with InterpretabilityPreprint
Xiang, T., MacAvaney, S., Yang, E., & Goharian, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01328 · Cited by 20
“Although reported to be highly predictive (Schmidt and Wiegand, 2017) and easily interpretable, they suffer from the problem of false positives as the presence of certain patterns could lead to misclassification (Kwok and Wang, 2013).”
BackgroundHow to Query Language Models?Preprint
Adolphs, L., Dhuliawala, S., & Hofmann, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01928 · Cited by 15
“, an Italian sounding name is Italian (Rogers et al., 2020; Poerner et al., 2020).”
BackgroundA Short Study on Compressing Decoder-Based Language ModelsPreprint
Li, T., Mesbahi, Y. E., Kobyzev, I., Rashid, A., Mahmud, A., Anchuri, N., Hajimolahoseini, H., Liu, Y., & Rezagholizadeh, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08460 · Cited by 35
“A predominant solution for fine-tuning compressed PLMs is knowledge distillation (KD) [Rogers et al., 2020].”
BackgroundModeling Emotion Dynamics in Song Lyrics with State Space Models
Song, Y., & Beck, D. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00541 · Cited by 12
“The motivation for our proposal is the hypothesis that the extension of lexicon-based information will compensate for BERT’s lack of proper representations of semantic and world knowledge (Rogers et al., 2021), making the model more stable across domains.”
BackgroundHyperRank: Hyperbolic Ranking Model for Unsupervised Keyphrase Extraction
Song, M., Liu, H., & Jing, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.997 · Cited by 17
“Previous studies (Kim et al., 2020; Rogers et al., 2020; Song et al., 2022a) have highlighted that the intermediate layers of the pre-trained language model BERT capture a diverse hierarchy of linguistic information, encompassing surface features in its lower layers, syntactic features in the…”
ResultConsumer responses to CEO activism: an impression management approach
Cabano, F. G., Li, M., & Jiménez, F. R. (2024) · European Journal of Marketing · doi:10.1108/ejm-07-2022-0553 · Cited by 13
Semantics of Multiword Expressions in Transformer-Based Models: A Survey
Miletić, F., & Walde, S. S. I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00657 · Cited by 31
“Different types of linguistic information are not represented in the same way in the transformer architecture (Rogers et al., 2020).”
BackgroundAutoIE: An Automated Framework for Information Extraction from Scientific Literature
Liu, Y., Li, S., Huang, K., & Wang, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5495-3_32 · Cited by 11
KubeAegis: A Unified Security Policy Management Framework for Containerized Environments
Kim, B., & Lee, S. (2024) · IEEE Access · doi:10.1109/access.2024.3487990 · Cited by 10
“The API recommender calculates the similarity between field descriptions and API methods using a Sentence Robustly Optimized BERT [59] 2 Pretraining Approach (SRoBERTa [19], [20]).”
Comparative analysis of NLP-driven MCQ generators from text sources
Azzi, A., Erdős, F., Németh, R., Varadarajan, V., & Afrifa, S. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100440 · Cited by 11
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 12
Evaluation of taxonomic and neural embedding methods for calculating semantic similarity
Yang, D., & Yin, Y. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000279 · Cited by 10
“…as increasing corpora size, controlling document length, and document domains, neural embeddings can be fine-tuned in many ways such as adjusting the size of a sliding window, optimizing hyperparameters in the stochastic gradient updates, and decreasing over-parameterization (Rogers et al. 2020).”
BackgroundMontenegro, J. L. Z., & da Costa, C. A. (2022) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-022-00115-0 · Cited by 12
“One reason for this is the mechanism by which multiple sentence pairs must be checked during inference, which can result in a slow process sometimes [83, 86].”
BackgroundProbing for Labeled Dependency Trees
Müller-Eberstein, M., Goot, R., & Plank, B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.532 · Cited by 22
Building a Vietnamese Dataset for Natural Language Inference Models
Nguyen, C. T., & Nguyen, D. T. (2022) · SN Computer Science · doi:10.1007/s42979-022-01267-x · Cited by 9
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
Huber, P., & Carenini, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.170 · Cited by 24
Pothina, R., & Ganguli, R. (2022) · Knowledge · doi:10.3390/knowledge2030021 · Cited by 9
XLMRQA: Open-Domain Question Answering on Vietnamese Wikipedia-Based Textual Knowledge Source
Nguyen, K. V., Nguyen-Thuan, P., Nguyen, N. D., Huynh, T. V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21743-2_30 · Cited by 11
The Analysis of Synonymy and Antonymy in Discourse Relations: An Interpretable Modeling Approach
Alamillo, A. R., Torres-Moreno, D., González, E. M., Toledo-Acosta, M., Taroni, A., & Hermosillo, J. (2023) · Computational Linguistics · doi:10.1162/coli_a_00477 · Cited by 9
“On the other hand, extracting knowledge about the linguistic functions that BERT attention heads manage to classify is an arduous and complex process, and the role of attention remains moot [40].”
InfluentialImproving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Gessler, L., & Zeldes, A. (2022) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2022.mrl-1.9 · Cited by 28
Embedding Recycling for Language Models
Saad-Falcon, J., Singh, A., Soldaini, L., D’Arcy, M., Cohan, A., & Downey, D. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.145 · Cited by 13
INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
Karmakar, A., & Robbes, R. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3341624 · Cited by 15
Finite State Automata on Multi-Word Units for Efficient Text-Mining
Postiglione, A. (2024) · Mathematics · doi:10.3390/math12040506 · Cited by 12
“An analysis published in 2020 highlighted BERT’s rapid ascent in the research world, with over 150 scientific publications within just over a year [71].”
BackgroundInvestigating the impact of pretraining corpora on the performance of Arabic BERT models
Alammary, A. S. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06698-2 · Cited by 10
“This bidirectional approach has empowered BERT to understand and generate more contextually relevant representations [4, 5].”
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Perlitz, Y., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 56
“This terminology, including probing and probe , is commonly found in influential literature surveys (Rogers et al., 2020; Belinkov, 2022) and diverse investigation settings, such as analyzing internal representations using linear classifiers (Tenney et al., 2019b; Conneau et al., 2018; Elazar et…”
Model Compression for Domain Adaptation through Causal Effect Estimation
Rotman, G., Feder, A., & Reichart, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00431 · Cited by 9
“…what gives large models the advantage over their smaller competitors, recent probing efforts have discovered that commonly used models such as BERT (Devlin et al., 2019), learn to capture semantic and syntactic information in different layers and neurons across the network (Rogers et al., 2021).”
MethodOn the evolution of syntactic information encoded by BERT’s contextualized representations
Pérez-Mayos, L., Carlini, R., Ballesteros, M., & Wanner, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.191 · Cited by 14
Fighting the COVID-19 Infodemic with a Holistic BERT Ensemble
Tziafas, G., Kogkalidis, K., & Caselli, T. (2021) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/2021.nlp4if-1.18 · Cited by 9
Soni, N., Matero, M., Balasubramanian, N., & Schwartz, H. A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.52 · Cited by 31
Self-supervised Representation Learning for Speech Processing
Lee, H.-Y., Mohamed, A., Watanabe, S., Sainath, T. N., Livescu, K., Li, S.-W. D., Yang, S.-W., & Kirchhoff, K. (2022) · doi:10.18653/v1/2022.naacl-tutorials.2 · Cited by 14
Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Grounding
Shimomoto, E. K., Marrese-Taylor, E., Takamura, H., Kobayashi, I., Nakayama, H., & Miyao, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.829 · Cited by 11
Attention And Positional Encoding Are (Almost) All You Need For Shape Matching
Raganato, A., Pasi, G., & Melzi, S. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14912 · Cited by 14
Wave to Syntax: Probing spoken language models for syntax
Shen, G., Alishahi, A., Bisazza, A., & Chrupała, G. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-679 · Cited by 27
“C L ] 30 M a y 2023 tations emerging within text-based language models: surveys of such work include [3, 4, 12].”
BackgroundMethodOf Non-Linearity and Commutativity in BERT
Zhao, S., Pascual, D., Brunner, G., & Wattenhofer, R. P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533563 · Cited by 22
Investigating Negation in Pre-trained Vision-and-language Models
Dobreva, R., & Keller, F. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.27 · Cited by 10
“In order to investigate the performance of these models on examples that contain negation in a controlled manner, we annotated a portion of the original NLVR2 test set to create minimally differing instances.”
InfluentialUnderstanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural NetworksPreprint
Dhar, P., & Bisazza, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.14056 · Cited by 15
“Extensive overviews of this body of work are presented in (Belinkov and Glass, 2019) and (Rogers et al., 2020).”
Interpreting Language Models Through Knowledge Graph ExtractionPreprint
Swamy, V., Romanou, A., & Jaggi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08546 · Cited by 15
“BERT is successful at encoding information at the syntactic and semantic level [14].”
BackgroundEvaluation of Contrastive Learning with Various Code Representations for Code Clone DetectionPreprint
Zubkov, M., Spirin, E., Bogomolov, E., & Bryksin, T. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4159812 · Cited by 14
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.