A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 213 of the 213 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Research works printed but not linked: 76
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“6 1 The desing of this table is borrowed from [196, 144].”
BackgroundA Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 994
FinBERT: A Large Language Model for Extracting Information from Financial Text*
Huang, A. H., Wang, H., & Yang, Y. (2023) · Contemporary Accounting Research · doi:10.1111/1911-3846.12832 · Cited by 778
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., De Almeida, B. P., Sirelkhatim, H., et al. (2023) · doi:10.1101/2023.01.11.523679 · Cited by 680
“1), which aligns with recent work in computational biology [33] and common practice in NLP [47].”
BackgroundInfluentialNucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
“These hyper-parameters were selected as they led to promising results in the field of NLP 35 .”
InfluentialDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
“Work on BERT such as “A Primer on BERTology” [41] and “BERT Relearns the Classical NLP Pipeline” [44] have probed what BERT learns across its layers and what world knowledge it holds within.”
BackgroundDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635
“Work on BERT such as “A Primer on BERTology” [41] and “BERT Relearns the Classical NLP Pipeline” [44] have probed what BERT learns across its layers and what world knowledge it holds within.”
BackgroundARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.551 · Cited by 397
“More information about BERT-inspired LMs can be found in Rogers et al. (2020).”
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundInfluentialBeneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 321
“al., 2019; Raffel et al., 2019). Despite this progress, it is still not clear as to whether these new models learn the composition semantics associated to sentiment or simply learn surface patterns (Rogers et al., 2020). Sentiment-Aware Word Embeddings: One of the critical building blocks of a text-processing deep-lea…”
BackgroundLearning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342
“…simply training a system to complete sentences—language modeling—causes it to implicitly acquire non-linguistic abilities from its training corpora (Rogers et al., 2020), including factual knowledge (Petroni et al., 2019; Jiang et al., 2020), common sense (Bisk et al., 2019), reasoning (Talmor et…”
BackgroundBERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
“, the subspecialty of “BERTology” [63], which studies the BERT [18] Transformer model specifically.”
BackgroundInfluentialSurvey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 215
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2022) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 249
“For recent analyses focusing on the BERT model (Devlin et al. 2019), see Rogers, Kovaleva, and Rumshisky (2020).”
MethodPre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
Bianchi, F., Terragni, S., & Hovy, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.96 · Cited by 269
“Consequently, BERT representations are used in a diverse set of NLP applications (Rogers et al., 2020; Nozza et al., 2020).”
BackgroundA Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.201 · Cited by 260
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
A Survey on Model Compression for Large Language Models
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00704 · Cited by 273
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269
“Although the standard BERT model, with only 0.34 billion parameters [29], does not meet the traditional criteria for “large,” its sophisticated bidirectional design and pivotal role in establishing new natural language processing (NLP) benchmarks justify its inclusion among notable LLMs [30].”
BackgroundInfluentialTransfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning
Prottasha, N. J., Sami, A. A., Kowsher, M., Murad, S. A., Bairagi, A. K., Masud, M., & Baz, M. (2022) · Sensors · doi:10.3390/s22114157 · Cited by 248
“However, the Bidirectional Encoder Representations from Transformers’s (BERT) recent advent phenomenon tremendously amplifies the contextualization strategy [3].”
BackgroundFrom Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 236
Post-hoc Interpretability for Neural NLP: A Survey
Madsen, A., Reddy, S., & Chandar, S. (2023) · ACM Computing Surveys · doi:10.1145/3546577 · Cited by 207
“BERT’s popularity and effectiveness have resulted in countless papers in this category [28, 30, 76, 98, 116], hence the name BERTology .”
BackgroundMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“There has been significant interest in analyzing how well PLMs (Rogers et al., 2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Manning, 2019; Tenney et al.”
BackgroundARBERT & MARBERT: Deep Bidirectional Transformers for ArabicPreprint
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01785 · Cited by 260
“More information about BERT-inspired LMs can be found in Rogers et al. (2020).”
Sinha, K., Jia, R., Hupkes, D., Pineau, J., Williams, A., & Kiela, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.230 · Cited by 181
“tains a surprising amount of syntactic, semantic and world knowledge (Giulianelli et al., 2018; Rogers et al., 2020).”
BackgroundSyntactic Structure from Deep Learning
Linzen, T., & Baroni, M. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-032020-051035 · Cited by 202
“…in NLP systems such as BERT (Devlin et al. 2019); however, a clear picture of the differences between their linguistic abilities and those of RNNs, especially when both are exposed to the same amount of training data, has yet to emerge (Y. Goldberg 2019, Rogers et al. 2020, Tran et al. 2018).”
BackgroundCompressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
“Several recent surveys have focused on pre-trained representations and large-scale Transformer-based models e.g. , Qiu et al. (2020); Rogers et al. (2020); Wang et al. (2020a).”
Probing Pretrained Language Models for Lexical Semantics
Vulić, I., Ponti, E. M., Litschko, R., Glavaš, G., & Korhonen, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.586 · Cited by 180
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.574 · Cited by 374
“Many studies have probed their inner workings to understand the mechanisms underlying their success (Rogers et al., 2020; Clark et al., 2019).”
BackgroundBERT Models for Arabic Text Classification: A Systematic Review
Alammary, A. S. (2022) · Applied Sciences · doi:10.3390/app12115720 · Cited by 166
“Since its introduction, it has become a ubiquitous baseline in natural language-processing research [4].”
BackgroundCross-lingual Contextualized Topic Models with Zero-shot Learning
Bianchi, F., Terragni, S., Hovy, D., Nozza, D., & Fersini, E. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.143 · Cited by 280
What Happens To BERT Embeddings During Fine-tuning?
Merchant, A., Rahimtoroghi, E., Pavlick, E., & Tenney, I. (2020) · Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2020.blackboxnlp-1.4 · Cited by 139
(2021) · doi:10.1145/3462757.3466088 · Cited by 151
Should You Fine-Tune BERT for Automated Essay Scoring?
Mayfield, E., & Black, A. W. (2020) · Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.15 · Cited by 137
“More broadly, the natural language processing (NLP) research community in recent years has been dominated by deep neural network research, in particular, the Transformer architecture popularized by BERT (Devlin et al., 2019).”
BackgroundInfluentialInterpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.431 · Cited by 139
URLTran: Improving Phishing URL Detection Using Transformers
Maneriker, P., Stokes, J. W., Lazo, E. G., Carutasu, D., Tajaddodianfar, F., & Gururajan, A. (2021) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom52596.2021.9653028 · Cited by 191
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2004.04037 · Cited by 143
“” In [25, 11, 20], the attention heads for single-sentence task are found to mainly play “positional”, “syntactic/semantic” functions.”
BackgroundGENA-LM: a family of open-source foundational DNA language models for long sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkae1310 · Cited by 135
Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
Weikum, G., Dong, X. L., Razniewski, S., & Suchanek, F. M. (2021) · Foundations and Trends in Databases · doi:10.1561/1900000064 · Cited by 202
When BERT Plays the Lottery, All Tickets Are Winning
Prasanna, S., Rogers, A., & Rumshisky, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.259 · Cited by 131
Self-citationTransformer-Based Language Models for Software Vulnerability Detection
Thapa, C., Jang, S. I., Ahmed, M. E., Camtepe, S., Pieprzyk, J., & Nepal, S. (2022) · Proceedings of the 38th Annual Computer Security Applications Conference · doi:10.1145/3564625.3567985 · Cited by 122
Hayes, A. S. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251322346 · Cited by 184
“Through their deep learning architecture and extensive pre-training, LLMs develop what appears to be an intuitive grasp of language in much the same way humans learn to understand nuance and context — through massive and repetitive exposure to examples (Rogers et al., 2021; Wei et al., 2022).”
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131
Large language models demonstrate the potential of statistical learning in language
Contreras Kallens, P., Kristensen‐McLachlan, R. D., & Christiansen, M. H. (2023) · Cognitive Science · doi:10.1111/cogs.13256 · Cited by 130
“Some of this work has already been done in the context of encoder-only masked language models, such as BERT and its related descendants (Ettinger, 2020; Pandia & Ettinger, 2021; Rogers, Kovaleva, & Rumshisky, 2020).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
“15 For a thorough summary of these advancements we refer the reader to a recent primer on the subject (Rogers et al., 2020).”
BackgroundOn the Explainability of Natural Language Processing Deep Models
Zini, J. E., & Awad, M. (2023) · ACM Computing Surveys · doi:10.1145/3529755 · Cited by 120
“Such dissection aids in understanding BERT [48, 108], its weights [19, 87] and limitations [93].”
BackgroundTable-GPT: Table Fine-tuned GPT for Diverse Table Tasks
Li, P., He, Y., Yashar, D., Cui, W., Song, G., Zhang, H., Fainman, D. R., Zhang, D., & Chaudhuri, S. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654979 · Cited by 278
“For example, suppose the downstream task is sentiment analysis of Yelp restaurant reviews, then labels in the form of (“ The food is amazing ”, “ positive ”), (“ The service is slow ”, “ negative ”), are needed to fine-tune BERT-like models for the desired outcome [22, 60].”
What the [MASK]? Making Sense of Language-Specific BERT Models
Nozza, D., Bianchi, F., & Hovy, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02912 · Cited by 84
“Both the Transformer and BERT have gathered much attention, and there is now a wealth of research articles and blog posts describing the inner workings of these models (Rogers et al., 2020, among others).”
BackgroundQuantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Mireshghallah, F., Goyal, K., Uniyal, A., Berg-Kirkpatrick, T., & Shokri, R. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.570 · Cited by 92
Lewis, M., Ghazvininejad, M., Ghosh, G., Aghajanyan, A., Wang, S., & Zettlemoyer, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15020 · Cited by 113
“Masked language models, in contrast, are less directly related to target fine tuning tasks and significant ongoing research focuses on understanding why they work so well, see Rogers et al. [2020] for a survey.”
BackgroundAkbar, R., Bashour, H., Rawat, P., Robert, P. A., Smorodina, E., Cotet, T.-S., Flem-Karlsen, K., Frank, R., Mehta, B. B., Vu, M. H., et al. (2022) · mAbs · doi:10.1080/19420862.2021.2008790 · Cited by 113
A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 106
Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
DeRose, J. F., Wang, J., & Berger, M. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028976 · Cited by 109
“The successes of BERT have led to numerous works in attempting to understand why the model – specifically the Transformer model and its pre-training objectives – performs so well [44].”
BackgroundWhat Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
He, Y., Zhu, Z., Zhang, Y., Chen, Q., & Caverlee, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.372 · Cited by 103
Ilie, V.-I., Truică, C., Apostol, E., & Paschke, A. (2021) · IEEE Access · doi:10.1109/access.2021.3132502 · Cited by 111
“…are a few fake news detection models that use dynamic word embeddings methods, i.e., Bidirectional Encoder Representations from Transformers (BERT) [39] [30], they use such methods as pretrained language model, i.e., without a detailed evaluation and discussion of the effects of the training on…”
MethodLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Xia, Y., Xiao, Z., Jazdi, N., & Weyrich, M. (2024) · IEEE Access · doi:10.1109/access.2024.3415470 · Cited by 129
“…interesting insights: in a LLM, the basic syntactic rules are learned in lower layer while high-level semantic concepts are learned in higher layer [29]; Moreover, ‘poly-semantic neurons’, which handle multiple unrelated concepts, enable the reuse of neurons in processing semantic through…”
BackgroundDEPTWEET: A typology for social media texts to detect depression severities
Kabir, M., Ahmed, T., Hasan, M. B., Laskar, M. T. R., Joarder, T. K., Mahmud, H., & Hasan, K. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107503 · Cited by 123
Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language Understanding
Kowsher, M., Sami, A. A., Prottasha, N. J., Arefin, M. S., Dhar, P. K., & Koshiba, T. (2022) · IEEE Access · doi:10.1109/access.2022.3197662 · Cited by 106
“BERT, a renowned transformer-based technique, brought a great revolution that had huge impacts on the evolution of NLP [2].”
BackgroundGenomic language model predicts protein co-regulation and function
Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R. (2024) · Nature Communications · doi:10.1038/s41467-024-46947-9 · Cited by 99
Annotating Columns with Pre-trained Language Models
Suhara, Y., Li, J., Li, Y., Zhang, D., Demiralp, Ç., Chen, C., & Tan, W.-C. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3517906 · Cited by 204
Transformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 147
Yun, Z., Chen, Y., Olshausen, B., & LeCun, Y. (2021) · Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2021.deelio-1.1 · Cited by 98
Comparing Pre-trained and Feature-Based Models for Prediction of Alzheimer's Disease Based on Speech
Balagopalan, A., Eyre, B., Robin, J., Rudzicz, F., & Novikova, J. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.635945 · Cited by 96
“…dependencies in text and as such, has achieved state-of-the-art performance on a variety of tasks in NLP. Previous research (Jawahar et al., 2019; Rogers et al., 2021) has suggested that it encodes language information (lexical, syntactic etc.) that is known to be important for performing…”
BackgroundShared functional specialization in transformer-based language models and the human brain
Kumar, S., Sumers, T. R., Yamakoshi, T., Goldstein, A., Hasson, U., Norman, K. A., Griffiths, T. L., Hawkins, R. D., & Nastase, S. A. (2024) · Nature Communications · doi:10.1038/s41467-024-49173-5 · Cited by 86
“Critically, BERT does not just learn to approximate certain classical syntactic operations; it learns a rich multiplicity of linguistic and contextual relations from natural language, a subset of which can be said to approximate classical syntactic labels 58 .”
BackgroundMethodInfluentialM2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 91
“They have been widely used as a probe for natural language models [56, 58, 66].”
BackgroundLiu, F., Vulić, I., Korhonen, A., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.109 · Cited by 162
“Despite the wealth of semantic knowledge stored in the MLMs (Rogers et al., 2020), they do not produce high-quality lexical and sentence Assumption”
What do they capture?: a structural analysis of pre-trained language models for source code
Wan, Y., Zhao, W., Zhang, H., Sui, Y., Xu, G., & Jin, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510050 · Cited by 73
“This kind of research has become a subspecialty of “BERTology” [31], which focuses on studying the inner-mechanism of BERT model [8].”
BackgroundProbing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arora, A., Kaffee, L.-A., & Augenstein, I. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.12 · Cited by 57
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
Raganato, A., Scherrer, Y., & Tiedemann, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.49 · Cited by 86
Pham, T., Bui, T., Mai, L., & Nguyen, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.98 · Cited by 79
“Pretrained BERT is known to capture various types of syntactic, semantic, and world knowledge (Rogers et al., 2020).”
BackgroundGeneralization in NLI: Ways (Not) To Go Beyond Simple Heuristics
Bhargava, P., Валентинович, Д. А., & Rogers, A. (2021) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2021.insights-1.18 · Cited by 120
Self-citationIteratively Prompt Pre-trained Language Models for Chain of Thought
Wang, B., Deng, X., & Sun, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.174 · Cited by 73
“With the recent success of large-scale pre-trained language models (PLMs), there has been growing interest in investigating what is captured by these PLMs during pre-training (Talmor et al., 2020a; Rogers et al., 2020; Kassner et al., 2020).”
BackgroundA Survey of Knowledge Enhanced Pre-trained Language Models
Yang, J., Hu, X., Xiao, G., & Shen, Y. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3631392 · Cited by 47
“Much effort has been devoted to examining the knowledge encoded in PLMs by different probing ways [86].”
BackgroundScaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
Lialin, V., Deshpande, V., Yao, X., & Rumshisky, A. (2023) · arXiv · doi:10.48550/arxiv.2303.15647 · Cited by 88
Self-citationAn Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels
Chalkidis, I., Fergadiotis, M., Kotitsas, S., Malakasiotis, P., Αλέτρας, Ν., & Androutsopoulos, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.607 · Cited by 126
Toward Green AI: A Methodological Survey of the Scientific Literature
Barbierato, E., & Gatti, A. (2024) · IEEE Access · doi:10.1109/access.2024.3360705 · Cited by 116
“Bidirectional Encoder Representations from Transformers (BERT) is a language representation model introduced by Google (a discussion of the principles ruling BERT training can be found in [69]).”
MethodWhen Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85
“Much recent work has used probing methods to evaluate what these models have and have not learned (Belinkov and Glass, 2019; Tenney et al., 2019b; Rogers et al., 2020; Ettinger, 2020).”
BackgroundMethodBERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweets
Joloudari, J. H., Hussain, S., Nematollahi, M. A., Bagheri, R., Fazl, F., Alizadehsani, R., Lashgari, R., & Talukder, A. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01102-y · Cited by 98
Hanna, M., Liu, O., & Variengien, A. (2023) · arXiv · doi:10.48550/arxiv.2305.00586 · Cited by 84
“Yet despite heavy study of these models [41, 42, 26], how LMs implement these abilities is still poorly understood.”
BackgroundUsing sequences of life-events to predict human lives
Savcisens, G., Eliassi‐Rad, T., Hansen, L. K., Mortensen, L. H., Lilleholt, L., Rogers, A., Zettler, I., & Lehmann, S. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00573-5 · Cited by 130
“Due to their structural similarity to written language, transformer-based architectures [2] have also shown promise as tools to make sense of a range of multi-variate sequences from protein-structures [3, 4], music [5, 6], electronic health records [7] to weather-forecasts [8, 9].”
Self-citationQuantum Natural Language Processing: Challenges and Opportunities
Guarasci, R., Pietro, G. D., & Esposito, M. (2022) · Applied Sciences · doi:10.3390/app12115651 · Cited by 104
Artificial intelligence: reflecting on the past and looking towards the next paradigm shift
Radanliev, P. (2024) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2024.2323042 · Cited by 128
“The model has been analysed and improved in over 150 research publications (Rogers et al., 2020).”
What do pre-trained code models know about code?
Karmakar, A., & Robbes, R. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678927 · Cited by 69
“Studies of the BERT models alone have spawned a subfield known as BERTology with over 150 studies surveyed [25].”
BackgroundChalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., & Malakasiotis, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.22 · Cited by 78
Analysis and Evaluation of Language Models for Word Sense Disambiguation
Loureiro, D., Rezaee, K., Pilehvar, M. T., & Camacho-Collados, J. (2021) · Computational Linguistics · doi:10.1162/coli_a_00405 · Cited by 140
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
Ostendorff, M., Rethmeier, N., Augenstein, I., Gipp, B., & Rehm, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.802 · Cited by 56
On Position Embeddings in BERT
Wang, B., Shang, L., Lioma, C., Jiang, X., Yang, H., Liu, Q., & Simonsen, J. G. (2021) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 133
Social Media Toxicity Classification Using Deep Learning: Real-World Application UK Brexit
Fan, H. M., Du, W., Dahou, A., Ewees, A. A., Yousri, D., Elaziz, M. A., Elsheikh, A. H., Abualigah, L., & Al‐qaness, M. A. A. (2021) · Electronics · doi:10.3390/electronics10111332 · Cited by 93
“Transformer-based models are dominating a wide variety of NLP tasks leading to many state-of-the-art results [70].”
BackgroundChen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae163 · Cited by 99
“This is probably because the hidden states of the last layer are mostly related to predicting the masked tokens during pretraining [ 46], while the intermediate layers preserve more contextual information.”
BackgroundSinha, K., Parthasarathi, P., Pineau, J., & Williams, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.569 · Cited by 71
“A simple demo is provided in an associated Google Co-lab notebook. tion (Hewitt and Manning, 2019; Jawahar et al., 2019; Warstadt and Bowman, 2020; Wu et al., 2020), with their self-attention layers being capable of surprisingly effective learning (Rogers et al., 2020).”
BackgroundMethod
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.