A Primer in BERTology: What We Know About How BERT Works
Open accessThe DOI matches its Crossref record; 208 of the 208 references it cites were found in a registry or an index.
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It is advised to apply Bidirectional Encoder Representations from Transformers (BERT) [95,144] to cope with the transferability in cross-domain for cyber grooming detection as it has profoundly bidirectional contextualization and allows the model to gain information from various representations with different positions [146].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Contextual models such as BERT 9 can represent each term based on the chat context and lead to a more profound knowledge for semantic analysis [144].”
From Online grooming detection: A comprehensive survey of child exploitation in chat logs (2023) · cites this paper as [144]
The cited source says what the claim attributes to it.
In the citing paper
“Before the emergence of ChatGPT, BERT is a commonly used pre-trained machine learning model for NLP tasks (Wolf et al., 2020), and a ubiquitous baseline in text classification experiments of research in recent years (Rogers et al., 2021).”
From Towards reliable generative AI-driven scaffolding: Reducing hallucinations and enhancing quality in self-regulated learning support (2026) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 5×
“…search-language-understanding-bert 2 Overview of BERT architecture Fundamentally, BERT is a stack of Transformer encoder layers (Vaswani et al., 2017) which consist of multiple “heads”, i.e., fully-connected neural networks augmented with a self-attention mechanism.”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The input representations are computed as fol-Figure 1: BERT fine-tuning (Devlin et al., 2019). lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Unlike the conventional static embeddings (Mikolov et al., 2013a; Pennington et al., 2014), BERT’s representations are contextualized, i.e., every input token is represented by a vector dependent on the particular context of occurrence.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 11×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundMethodResultInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The studies in the knowledge distillation framework (Hinton et al., 2015) use a smaller student-network that is trained to mimic the behavior of a larger teacher-network (BERT-large or BERT-base).”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“(2020) show the same for open-domain QA using T5 model (Raffel et al., 2019).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“, 2020), predicting how many are missing (Lewis et al., 2019).”
BackgroundXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“Yang et al. (2019) replace MLM with training on different permutations of word order in the input sequence, maximizing the probability of the original word order.”
MethodUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“C L ] 2 7 Fe b 20 20 lows: BERT first tokenizes the given word into wordpieces (Wu et al., 2016b), and then combines three embedding layers (token, position, and segment) to obtain a fixed-length vector.”
BackgroundMethodDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
(2006) · Cited by 3,931
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998) · Proceedings of the 36th annual meeting on Association for Computational Linguistics - · doi:10.3115/980845.980860 · Cited by 2,518 · cited 1×
“Koval-eva et al. (2019) identified two heads of base BERT, in which self-attention maps were closely aligned with annotations of core frame semantic relations (Baker et al., 1998).”
BackgroundParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1×
“With larger and larger models even fine-tuning becomes expensive, but Houlsby et al. (2019) show that it can be successfully approximated by inserting adapter modules.”
BackgroundJain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,975 · cited 2×
“This view is currently debated (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020), and in a multi-layer model where attention is followed by non-linear transformations, the patterns in individual heads do not provide a full picture.”
BackgroundMethodLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · cited 3×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10529 · Cited by 1,827 · cited 3×
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundTinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020) · Communications of the ACM · doi:10.1145/3381831 · Cited by 1,647
(2018) · International Conference on Learning Representations · Cited by 1,621
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · cited 2×
“xamples, rather than sentences from different documents. Dynamic masking (Liu et al.,2019b) improves on BERT’s MLM by using diverse masks for training examples within an epoch; Beyond-sentence MLM. Lample and Conneau (2019) replace sentence pairs with arbitrary text streams, and subsample frequent outputs similarly to…”
BackgroundERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 4×
“Liu et al. (2019c) explore the benefits 4New token-level tasks in ERNIE include prediction whether a token is capitalized and whether it occurs in other segments of the same document.”
BackgroundMethodInfluentialWhat Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
How Multilingual is Multilingual BERT?
Pires, T., Schlinger, E., & Garrette, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1493 · Cited by 1,351 · cited 5×
“Languages with a lot of data were subsampled, and some were super-sampled using exponential smoothing. mBERT performs surprisingly well in zero-shot transfer on many tasks (Wu and Dredze, 2019; Pires et al., 2019), although not in language generation (R¨onnqvist et al., 2019).”
BackgroundMethodInfluentialBERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 5×
“Jawahar et al. (2019) include a brief illustration of a dependency tree extracted directly from self-attention weights, but provide no quantitative evaluation.”
BackgroundInfluentialPre-Training With Whole Word Masking for Chinese BERT
Cui, Y., Che, W., Liu, T., Qin, B., & Yang, Z. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3124365 · Cited by 1,181
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 5×
“To mitigate that, Michel et al. (2019) prune heads in the order set by a proxy importance score, and Voita et al. (2019) fine-tune the pretrained Transformer with a regularized objective that has the head-disabling effect.”
BackgroundResultInfluentialHow Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.Preprint
Frankle, J., & Carbin, M. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 838
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 2×
“In particular, it was shown to rely on shallow heuristics in both natural language inference (McCoy et al., 2019b; Zellers et al., 2019) and reading comprehension (Si et al., 2019; Rogers et al., 2020; Sugawara et al., 2020).”
BackgroundERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., & Wang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6428 · Cited by 766
ERNIE: Enhanced Representation through Knowledge Integration
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 721 · cited 3×
“Masking phrases and named entities (Sun et al., 2019b) improves representation of structured knowledge.”
BackgroundInfluentialMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · cited 1×
“4 93% ×9 BERT8 †‖ SQuAD MiniLM (Wang et al., 2020b) ×1.”
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · cited 1×
“However, there is ongoing debate on the merits of attention as a tool for interpreting deep learning models (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Brunner et al., 2020).”
MethodKEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676
Knowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661
K-Adapter: Infusing Knowledge into Pre-Trained Models with AdaptersPreprint
Wang, R., Tang, D.-Y., Duan, N., Wei, Z., Huang, X., Ji, J.-S., Cao, G.-H., Jiang, D.-X., & Zhou, M. (2020) · Findings · doi:10.48550/arxiv.2002.01808 · Cited by 657
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1441 · Cited by 631
Linguistic Knowledge and Transferability of Contextual Representations
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., & Smith, N. A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1112 · Cited by 621 · cited 16×
“, 2020; Rosa and Mareček, 2019), although probing classifiers could not recover the labels of distant parent nodes in the syntactic tree (Liu et al., 2019a).”
BackgroundMethodResultInfluential(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · cited 2×
“Hewitt and Manning (2019) had the most success reconstructing syntactic tree depth from the middle BERT layers (6-9 for base-BERT, 14-19 for BERT-large).”
BackgroundOn the Cross-lingual Transferability of Monolingual Representations
Artetxe, M., Ruder, S., & Yogatama, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.421 · Cited by 598 · cited 4×
“Artetxe et al. (2019) also show cross-lingual transfer is possible by swapping the model Figure 7: Language centroids of the mean-pooled mBERT representations (Libovick´y et al., 2019) vocabulary, without any shared word-pieces.”
BackgroundMethodInfluentialREALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
MPNet: Masked and Permuted Pre-training for Language Understanding
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09297 · Cited by 514
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511 · cited 2×
“However, we do have evidence from an MLM probing study that BERT has some knowledge for semantic roles (Ettinger, 2019).”
BackgroundRevealing the Dark Secrets of BERT
Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1445 · Cited by 495 · cited 18×
“At the moment, we know that the knowledge represented in BERT does not necessarily get used in down-stream tasks (Kovaleva et al., 2019).”
BackgroundMethodResultInfluentialSelf-citationTaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · cited 1×
“"This is <>") (May et al., 2019), and even using contextualized embeddings to train static embeddings (Wang et al.”
BackgroundK-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G., Jiang, D., & Zhou, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.121 · Cited by 456
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442 · cited 1×
“The quantization approach aims to decrease BERT’s memory footprint through lowering the precision of its weights (Shen et al., 2019; Zafrir et al., 2019).”
MethodQ-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6409 · Cited by 437
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
Peters, M. E., Ruder, S., & Smith, N. A. (2019) · Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) · doi:10.18653/v1/w19-4302 · Cited by 432
The Lottery Ticket Hypothesis for Pre-trained BERT NetworksPreprint
Chen, T., Frankle, J., Chang, S.-Y., Liu, S., Zhang, Y., Wang, Z.-Y., & Carbin, M. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2007.12223 · Cited by 432
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., & Smith, N. (2020) · arXiv · doi:10.48550/arxiv.2002.06305 · Cited by 412 · cited 1×
“Dodge et al. (2020) report significant variation for BERT fine-tuned on GLUE tasks, where both weight initialization and training data order contribute to the variation.”
MethodWell-Read Students Learn Better: On the Importance of Pre-training Compact Models
Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08962 · Cited by 398 · cited 3×
“3 100% ×4 BERT24 No WNLI; SQuAD PD (Turc et al., 2019) ×1.”
BackgroundProbing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395
Reducing Transformer Depth on Demand with Structured Dropout
Fan, A., Grave, E., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1909.11556 · Cited by 387 · cited 1×
“In structured pruning, architecture blocks are dropped, as in LayerDrop (Fan et al., 2019).”
BackgroundAssessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385 · cited 4×
“In particular, the opposite was observed for subject-verb agreement (Goldberg, 2019) and sentence subject detection Lin et al. (2019).”
BackgroundMethodInfluentialCross-Lingual Ability of Multilingual BERT: An Empirical StudyPreprint
Karthikeyan, K., Wang, Z., Mayhew, S., & Roth, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07840 · Cited by 383 · cited 7×
“To date, the most systematic study of BERT architecture was performed by Wang et al. (2019b).”
BackgroundMethodInfluentialSemantics-Aware BERT for Language Understanding
Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., & Zhou, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6510 · Cited by 364
Emergent linguistic structure in artificial neural networks trained by self-supervision
Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., & Levy, O. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907367117 · Cited by 339
Conditional BERT Contextual Augmentation
Wu, X., Lv, S., Zang, L., Han, J., & Hu, S. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-22747-0_7 · Cited by 336
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Gordon, M., Duh, K., & Andrews, N. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.18 · Cited by 303
Analyzing the Structure of Attention in a Transformer Language Model
Vig, J., & Belinkov, Y. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4808 · Cited by 301
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 297 · cited 1×
“, 2019) ×6§ 98% ×27§ mBERT3 CoNLL-2018 POS and morphology BiLSTM soft (Tang et al., 2019) ×110 91% ×434‡ BiLSTM1 MNLI, QQP, SST-2”
Data Augmentation using Pre-trained Transformer Models
Kumar, V., Choudhary, A., & Cho, E. (2020) · Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems · doi:10.18653/v1/2020.lifelongnlp-1.3 · Cited by 294
Pooled Contextualized Embeddings for Named Entity Recognition
Akbik, A., Bergmann, T., & Vollgraf, R. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1078 · Cited by 286 · cited 1×
“The methods to distill a contextualized representation into static include aggregating the information across multiple contexts (Akbik et al., 2019; Bommasani et al., 2020), encoding "semantically bleached" sentences that rely almost exclusively on the meaning of a given word (e.”
MethodAn Analysis of Encoder Representations in Transformer-Based Machine Translation
Raganato, A., & Tiedemann, J. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5431 · Cited by 277
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 271
Pay Less Attention with Lightweight and Dynamic Convolutions
Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10430 · Cited by 270 · cited 2×
“This raises concerns about computational complexity of self-attention (Wu et al., 2019a), environmental issues (Strubell et al., 2019; Schwartz et al., 2019), as well as re-producibility and access to research resources in academia vs. industry.”
BackgroundMethodFreeLB: Enhanced Adversarial Training for Natural Language Understanding
Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., & Liu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11764 · Cited by 268 · cited 1×
“• Adversarial token perturbations improve robustness of the model (Zhu et al., 2019).”
BackgroundCommonsense Knowledge Mining from Pretrained Models
Davison, J., Feldman, J., & Rush, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1109 · Cited by 264 · cited 1×
“Davison et al. (2019) suggest that it generalizes better to unseen data.”
BackgroundInterpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.431 · Cited by 256
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12804 · Cited by 254
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 251
Open Sesame: Getting inside BERT’s Linguistic Knowledge
Lin, Y., Tan, Y. C., & Frank, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4825 · Cited by 237
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · cited 2×
“This is in line with the recent findings on adversarial attacks, with models disturbed by nonsensical inputs (Wallace et al., 2019a), and suggests that BERT’s encoding of syntactic structure does not indicate that it actually relies on that knowledge .”
BackgroundResult75 Languages, 1 Model: Parsing Universal Dependencies Universally
Kondratyuk, D., & Straka, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1279 · Cited by 225 · cited 1×
“• Taking more layers into account: learning a complementary representation of the information in deep and output layers (Yang and Zhao, 2019), using a weighted combination of all layers instead of the final one (Su and Cheng, 2019; Kondratyuk and Straka, 2019), and layer dropout (Kondratyuk and Straka, 2019).”
BackgroundMovement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218
Cloze-driven Pretraining of Self-attention Networks
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1539 · Cited by 215 · cited 1×
“Several studies explored the benefits of increasing the corpus volume (Liu et al., 2019b; Conneau et al., 2019; Baevski et al., 2019) and longer training (Liu et al.”
BackgroundDoes BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized EmbeddingsPreprint
Wiedemann, G., Remus, S., Chawla, A., & Biemann, C. (2019) · Conference on Natural Language Processing · doi:10.48550/arxiv.1909.10430 · Cited by 206
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1223 · Cited by 198
Voita, E., Sennrich, R., & Titov, I. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1448 · Cited by 193
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190 · cited 2×
“In unstructured, the weights in the entire model are pruned irrespective of their location, as in magnitude pruning (Chen et al., 2020) or movement pruning (Sanh et al.”
MethodTANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection
Garg, S., Vu, T., & Moschitti, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6282 · Cited by 177
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Xu, C., Zhou, W., Ge, T., Wei, F., & Zhou, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.633 · Cited by 168
Visualizing and Understanding the Effectiveness of BERT
Hao, Y., Dong, L., Wei, F., & Xu, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1424 · Cited by 166 · cited 4×
“Figure 5: Pre-trained weights help BERT find wider optima in fine-tuning on MRPC (right) than training from scratch (left) (Hao et al., 2019)”
BackgroundInfluentialBERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task LearningPreprint
Stickland, A. C., & Murray, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02671 · Cited by 163
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., & Bowman, S. R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.467 · Cited by 162
Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Zhao, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.197 · Cited by 162
Knowledge Distillation from Internal Representations
Aguilar, G., Ling, Y., Zhang, Y., Yao, B., Fan, X., & Guo, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6229 · Cited by 159
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
Wu, Z., Chen, Y., Kao, B., & Liu, Q. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.383 · Cited by 152
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.22 · Cited by 151
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146
Learning and Evaluating General Linguistic Intelligence
Yogatama, D., d’Autume, C. D. M., Connor, J. T., Kočiský, T., Chrzanowski, M., Kong, L., Lazaridou, A., Wang, L., Yu, L., Dyer, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11373 · Cited by 144
Small and Practical BERT Models for Sequence Labeling
Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., & Archer, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1374 · Cited by 143 · cited 1×
“53 BERT6 No WNLI, CoLA and STS-B MiniBERT(Tsai et al., 2019) ×6§ 98% ×27§ mBERT3 CoNLL-2018 POS and morphology BiLSTM soft (Tang et al.”
BackgroundAll-but-the-Top: Simple and Effective Postprocessing for Word RepresentationsPreprint
Mu, J., & Viswanath, P. (2018) · arXiv (Cornell University) · Cited by 141
Inducing Relational Knowledge from BERT
Bouraoui, Z., Camacho-Collados, J., & Schockaert, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6242 · Cited by 137
Data Augmentation using Pre-trained Transformer ModelsPreprint
Kumar, V., Choudhary, A., & Cho, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02245 · Cited by 137
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11794 · Cited by 137
How multilingual is Multilingual BERT?Preprint
Pires, T., Schlinger, E., & Garrette, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01502 · Cited by 135 · cited 5×
“Languages with a lot of data were subsampled, and some were super-sampled using exponential smoothing. mBERT performs surprisingly well in zero-shot transfer on many tasks (Wu and Dredze, 2019; Pires et al., 2019), although not in language generation (R¨onnqvist et al., 2019).”
BackgroundMethodInfluentialTinyBERT: Distilling BERT for Natural Language UnderstandingPreprint
Jiao, X., Yin, Y., Shang, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10351 · Cited by 133 · cited 2×
“This is achieved through experiments with loss functions (Sanh et al., 2019; Jiao et al., 2019), mimicking the activation patterns of individual portions of the teacher network (Sun et al., 2019a), and knowledge transfer at different stages at the pre-training (Turc et al., 2019; Jiao et al., 2019;…”
BackgroundMethodWhen BERT Plays the Lottery, All Tickets Are Winning
Prasanna, S., Rogers, A., & Rumshisky, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.259 · Cited by 131
Self-citationAmnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · International Conference on Machine Learning · Cited by 125
Visualizing Attention in Transformer-Based Language Representation ModelsPreprint
Vig, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02679 · Cited by 123 · cited 1×
“Many papers are accompanied by attention visualizations, with a growing number of visualization tools (Vig, 2019; Hoover et al., 2019).”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge DistillationPreprint
Turc, I., Chang, M., Lee, K., & Toutanova, K. (2019) · arXiv (Cornell University) · Cited by 120
Probing Natural Language Inference Models through Semantic Fragments
Richardson, K., Hu, H., Moss, L., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6397 · Cited by 112
Thieves on Sesame Street! Model Extraction of BERT-based APIs
Krishna, K., Tomar, G. S., Parikh, A. P., Papernot, N., & Iyyer, M. (2019) · arXiv · doi:10.48550/arxiv.1910.12366 · Cited by 111 · cited 1×
“Such heuristics can even be used to reconstruct a non-publiclyavailable model (Krishna et al., 2020).”
BackgroundHow Language-Neutral is Multilingual BERT?Preprint
Libovický, J., Rosa, R., & Fraser, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03310 · Cited by 109 · cited 8×
“It is also aware of at least some typological language features (Libovický et al., 2019; Singh et al., 2019), and transfer between structurally similar languages works better (Wang et al., 2019b; Pires et al., 2019).”
BackgroundInfluentialOn the use of BERT for Neural Machine Translation
Clinchant, S., Jung, K. W., & Nikoulina, V. (2019) · Proceedings of the 3rd Workshop on Neural Generation and Translation · doi:10.18653/v1/d19-5611 · Cited by 103
On Identifiability in Transformers
Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., & Wattenhofer, R. P. (2020) · International Conference on Learning Representations · Cited by 94
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Wang, W., Bi, B., Yan, M., Wu, C., Bao, Z., Xia, J., Peng, L., & Si, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04577 · Cited by 91 · cited 1×
“See also the n-gram word order reconstruction task (Wang et al., 2019a).”
MethodUniversal Text Representation from BERT: An Empirical StudyPreprint
Ma, X., Wang, Z., Ng, P., Nallapati, R. M., & Xiang, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07973 · Cited by 90
Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
Broscheit, S. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1063 · Cited by 89
What do you mean, BERT? Assessing BERT as a Distributional Semantics ModelPreprint
Mickus, T., Paperno, D., Constant, M., & van Deemter, K. (2019) · arXiv (Cornell University) · doi:10.7275/t778-ja71 · Cited by 88 · cited 1×
“However, Mickus et al. (2019) note that representations of the same word varies depending on position of the sentence in which it occurs, likely due to NSP objective.”
BackgroundProducts of Random Latent Variable Grammars
Petrov, S. (2010) · Cited by 87
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Huang, H., Liang, Y., Duan, N., Gong, M., Shou, L., Jiang, D., & Zhou, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00964 · Cited by 87 · cited 1×
“…pre-training objective where words are masked in parallel sentence pairs (the model can attend to one or both sentences to solve the prediction task); • Huang et al. (2019) combine 5 pre-training tasks (monolingual and cross-lingual MLM, translation language modeling, cross-lingual word…”
BackgroundFixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
Raganato, A., Scherrer, Y., & Tiedemann, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.49 · Cited by 86
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs
Warstadt, A., Cao, Y., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1286 · Cited by 85 · cited 3×
“Furthermore, different probing methods may reveal complementary or even contradictory information, in which case a single test (as done in most studies) would not be sufficient (Warstadt et al., 2019).”
BackgroundInfluentialDo Attention Heads in BERT Track Syntactic Dependencies?
Htut, P. M., Phang, J., Bordia, S., & Bowman, S. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.12246 · Cited by 84 · cited 8×
“Htut et al. (2019) and Clark et al. (2019) report that there are BERT heads that attended significantly more than a random baseline to words in certain syntactic positions.”
BackgroundMethodInfluentialKEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language RepresentationPreprint
Wang, X., Gao, T., Zhu, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06136 · Cited by 79
PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models
Ben-David, E., Rabinovitz, C., & Reichart, R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00328 · Cited by 77
Efficient Training of BERT by Progressively Stacking
Gong, L., He, D., Li, Z., Qin, T., Wang, L., & Liu, T. (2019) · International Conference on Machine Learning · Cited by 76
ERNIE 2.0: A Continual Pre-training Framework for Language UnderstandingPreprint
Sun, Y., Wang, S., Li, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12412 · Cited by 75
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10597 · Cited by 73 · cited 1×
“This raises concerns about computational complexity of self-attention (Wu et al., 2019a), environmental issues (Strubell et al., 2019; Schwartz et al., 2019), as well as re-producibility and access to research resources in academia vs. industry.”
BackgroundContextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation
Miaschi, A., & Dell’Orletta, F. (2020) · Proceedings of the 5th Workshop on Representation Learning for NLP · doi:10.18653/v1/2020.repl4nlp-1.15 · Cited by 72
Crane, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00018 · Cited by 70 · cited 1×
“However, variation across initializations is not often reported, although the performance improvements claimed in many NLP modeling papers may be within the range of that variation (Crane, 2018).”
BackgroundAnalyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., et al. (2019) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.1905.09418 · Cited by 69 · cited 5×
“To mitigate that, Michel et al. (2019) prune heads in the order set by a proxy importance score, and Voita et al. (2019) fine-tune the pretrained Transformer with a regularized objective that has the head-disabling effect.”
BackgroundResultInfluentialQuantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65
Improving Transformer Models by Reordering their Sublayers
Press, O., Smith, N. A., & Levy, O. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.270 · Cited by 64
Structured Pruning of a BERT-based Question Answering Model
McCarley, J. S., Chakravarti, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06360 · Cited by 64
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Sugawara, S., Stenetorp, P., Inui, K., & Aizawa, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6422 · Cited by 63
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 63
Self-citationSpan Selection Pre-training for Question Answering
Glass, M., Gliozzo, A., Chakravarti, R., Ferritto, A., Pan, L., Bhargav, G. P. S., Garg, D., & Sil, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.247 · Cited by 62
BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
Schick, T., & Schütze, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.368 · Cited by 62
MasakhaNER: Named entity recognition for African languages
(2021) · HAL · doi:10.1162/tacl · Cited by 61
What do you mean, BERT? Assessing BERT as a Distributional Semantics ModelPreprint
Mickus, T., Paperno, D., Constant, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05758 · Cited by 61 · cited 1×
“However, Mickus et al. (2019) note that representations of the same word varies depending on position of the sentence in which it occurs, likely due to NSP objective.”
BackgroundWhat does BERT Learn from Multiple-Choice Reading Comprehension Datasets?Preprint
Si, C., Wang, S., Kan, M., & Jiang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12391 · Cited by 58 · cited 1×
“In particular, it was shown to rely on shallow heuristics in both natural language inference (McCoy et al., 2019b; Zellers et al., 2019) and reading comprehension (Si et al., 2019; Rogers et al., 2020; Sugawara et al., 2020).”
BackgroundTransfer Fine-Tuning: A BERT Case Study
Arase, Y., & Tsujii, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1542 · Cited by 57 · cited 2×
“, 2020; Arase and Tsujii, 2019; Pruksachatkun et al., 2020; Glavaš and Vulić, 2020). Ben-David et al. (2020) propose a pivot-based variant of MLM to fine-tune BERT for domain adaptation.”
BackgroundFrom English To Foreign Languages: Transferring Pre-trained Language ModelsPreprint
Tran, K. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07306 · Cited by 57 · cited 2×
“To date, the following proposals were made for improving mBERT: • fine-tuning on multilingual datasets is improved by freezing the bottom layers (Wu and Dredze, 2019); • improving word alignment in fine-tuning (Cao et al., 2019); • translation language modeling (Lample and Conneau, 2019) (2019) independently showed that…”
BackgroundBeto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTPreprint
Wu, S., & Dredze, M. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09077 · Cited by 56 · cited 4×
“Pires et al. (2019) and Wu and Dredze (2019) hypothesize that shared word-pieces help mBERT, based on experiments where the task performance correlated with the amount of shared vocabulary between languages.”
BackgroundMethodInfluentialSundararaman, D., Subramanian, V., Wang, G., Si, S., Shen, D., Wang, D., & Carin, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06156 · Cited by 54 · cited 1×
“In particular, it was shown to rely on shallow heuristics in both natural language inference (McCoy et al., 2019b; Zellers et al., 2019) and reading comprehension (Si et al., 2019; Rogers et al., 2020; Sugawara et al., 2020).”
BackgroundQ-BERT: Hessian Based Ultra Low Precision Quantization of BERTPreprint
Shen, S., Dong, Z., Ye, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05840 · Cited by 52
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 48
Zhao, Y., & Bethard, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.429 · Cited by 47
RNNs Implicitly Implement Tensor Product Representations
McCoy, R. T., Linzen, T., Dunbar, E., & Smolensky, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08718 · Cited by 45
Vilares, D., Strzyz, M., Søgaard, A., & Gómez‐Rodríguez, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6446 · Cited by 45
A Cross-Task Analysis of Text Span Representations
Toshniwal, S., Shi, H., Shi, B., Gao, L., Livescu, K., & Gimpel, K. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.20 · Cited by 45
Reweighted Proximal Pruning for Large-Scale Language Representation
Guo, F.-M., Liu, S., Mungall, F. S., Lin, X., & Wang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12486 · Cited by 41
Information-Theoretic Probing for Linguistic Structure
Pimentel, T., Valvoda, J., Hall Maudslay, R., Zmigrod, R., Williams, A., & Cotterell, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.420 · Cited by 41
Small and Practical BERT Models for Sequence LabelingPreprint
Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., & Archer, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00100 · Cited by 40 · cited 1×
“53 BERT6 No WNLI, CoLA and STS-B MiniBERT(Tsai et al., 2019) ×6§ 98% ×27§ mBERT3 CoNLL-2018 POS and morphology BiLSTM soft (Tang et al.”
BackgroundexBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05276 · Cited by 40 · cited 2×
“Hoover et al. (2019) hypothesize that even complex dependencies like dobj are encoded by a combination of heads rather than a single head, but this work is limited to qualitative analysis.”
BackgroundUnderstanding Multi-Head Attention in Abstractive SummarizationPreprint
Baan, J., ter Hoeve, M., van der Wees, M., Schuth, A., & de Rijke, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03898 · Cited by 36 · cited 2×
“, 2019), abstractive summarization (Baan et al., 2019), and GLUE tasks (Kovaleva et al.”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 35 · cited 2×
“In particular, it was shown to rely on shallow heuristics in both natural language inference (McCoy et al., 2019b; Zellers et al., 2019) and reading comprehension (Si et al., 2019; Rogers et al., 2020; Sugawara et al., 2020).”
BackgroundCan neural networks acquire a structural bias from raw linguistic data?
Warstadt, A., & Bowman, S. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06761 · Cited by 33
BERT-of-Theseus: Compressing BERT by Progressive Module ReplacingPreprint
Xu, C., Zhou, W., Ge, T., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02925 · Cited by 33 · cited 1×
“Other techniques include decomposing BERT’s embedding matrix into smaller matrices (Lan et al., 2019) and progressive model replacing (Xu et al., 2020).”
MethodAre Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
Kim, T., Choi, J., Edmiston, D., & Lee, S.-G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00737 · Cited by 32
A matter of framing: The impact of linguistic formalism on probing results
Kuznetsov, I., & Gurevych, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.13 · Cited by 30
Inducing Syntactic Trees from BERT RepresentationsPreprint
Rosa, R., & Mareček, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.11511 · Cited by 28
What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?
Balasubramanian, S., Jain, N., Jindal, G., Awasthi, A., & Sarawagi, S. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.24 · Cited by 27
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
Li, C., & Choi, J. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.505 · Cited by 25
Extremely Small BERT Models from Mixed-Vocabulary Training
Zhao, S., Gupta, R., Song, Y., & Zhou, D. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.238 · Cited by 25
On the comparability of Pre-trained Language ModelsPreprint
Aßenmacher, M., & Heumann, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00781 · Cited by 23
Knowledge Distillation from Internal RepresentationsPreprint
Aguilar, G., Ling, Y., Zhang, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03723 · Cited by 23
Understanding Commonsense Inference Aptitude of Deep Contextual Representations
Da, J., & Kasai, J. (2019) · Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing · doi:10.18653/v1/d19-6001 · Cited by 22
Does BERT Solve Commonsense Task via Commonsense KnowledgePreprint
Cui, L., Cheng, S., Wu, Y., & Zhang, Y. (2020) · arXiv (Cornell University) · Cited by 21
Kao, W.-T., Wu, T.-H., Chi, P.-H., Hsieh, C.-C., & Lee, H.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09309 · Cited by 21 · cited 1×
“Depending on the task, some BERT heads/ layers are not only redundant (Kao et al., 2020), but also harmful to the downstream task performance.”
BackgroundDoes BERT agree? Evaluating knowledge of structure dependence through agreement relationsPreprint
Bacon, G., & Regier, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.09892 · Cited by 20 · cited 1×
“At least some of the syntactic properties of English BERT hold for mBERT: its MLM is aware of 4 types of agreement in 26 languages (Bacon and Regier, 2019), and main auxiliary of the sentence can be detected in German and Nordic languages R¨onnqvist et al. (2019).”
BackgroundComputational Science – ICCS 2019
(2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22747-0 · Cited by 18
Document Classification by Word Embeddings of BERT
Tanaka, H., Shinnou, H., Cao, R., Bai, J., & Ma, W. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-6168-9_13 · Cited by 17
Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
Xu, W., Cheng, X., Chen, K., & Wang, T. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401309 · Cited by 17
Deepening Hidden Representations from Pre-trained Language ModelsPreprint
Yang, J., & Zhao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01940 · Cited by 16 · cited 1×
“Yang and Zhao (2019) learn a complementary representation of the information in the deeper layers that is combined with the output layer.”
BackgroundAttention is Not Only a Weight: Analyzing Transformers with Vector NormsPreprint
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10102 · Cited by 15
Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical InvestigationPreprint
Glavaš, G., & Vulić, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06788 · Cited by 14
How Can BERT Help Lexical Semantics Tasks?Preprint
Wang, Y., Cui, L., & Zhang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02929 · Cited by 14
SesameBERT: Attention for Anywhere
Su, T.-C., & Cheng, H.-C. (2020) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa49011.2020.00050 · Cited by 12 · cited 1×
“Su and Cheng (2019) propose using a weighted representation of all layers instead of the final layer output.”
BackgroundWhatcha lookin' at? DeepLIFTing BERT's Attention in Question AnsweringPreprint
Arkhangelskaia, E., & Dutta, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06431 · Cited by 10 · cited 1×
“As starting points for work in this direction, we also have other head ablation studies (Voita et al., 2019; Michel et al., 2019) and studies of how BERT behaves in reading comprehension task (van Aken et al., 2019; Arkhangelskaia and Dutta, 2019).”
BERT as a Teacher: Contextual Embeddings for Sequence-Level RewardPreprint
Schmidt, F., & Hofmann, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02738 · Cited by 10
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Zadeh, A. H., Edo, I., Awad, O. M., & Moshovos, A. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00071 · Cited by 10 · cited 1×
“, 2019) ×4 99% − BERT12 No WNLI, MNLI; SQuAD GOBO (Zadeh and Moshovos, 2020) ×9.”
Improving BERT Fine-tuning with Embedding NormalizationPreprint
Zhou, W., Du, J., & Ren, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03918 · Cited by 9 · cited 1×
“Zhou et al. (2019) observe that the embedding values of the trained [CLS] token are not centered around zero, their normalization stabilizes the training leading to a slight performance gain on text classification tasks.”
BackgroundYang, J., & Zhao, H. (2019) · arXiv (Cornell University) · Cited by 8
SegaBERT: Pre-training of Segment-aware BERT for Language Understanding
Bai, H., Shi, P., Lin, J., Tan, L., Xiong, K., Gao, W., & Li, M. (2020) · Cited by 7
Symmetric Regularization based BERT for Pair-wise Semantic ReasoningPreprint
Xu, W., Cheng, X., Chen, K., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03405 · Cited by 6
Research works printed but not linked: 77
- Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
- Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
- He Bai, 2020, arXiv:2004. 14996 [cs]
- Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
- Brunner Gino, 2020, International Conference on Learning Representations
- Clark Kevin, 2020, International Conference on Learning Representations
- Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
- Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
- Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
- Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
- Fan Angela, 2019, International Conference on Learning Representations
- Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
- Frankle Jonathan, 2019, International Conference on Learning Representations
- Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
- Garg Siddhant, 2020, AAAI
- Gong Linyuan, 2019, International Conference on Machine Learning, 2337
- Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
- Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
- Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
- Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
- Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
- Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
- Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
- Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
- Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
- Di Jin, 2020, AAAI 2020
- Kim Taeuk, 2020, ICLR 2020
- Kong Lingpeng, 2019, International Conference on Learning Representations
- Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
- Krishna Kalpesh, 2020, ICLR 2020
- Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
- Lan Zhenzhong, 2020, ICLR
- Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
- Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
- Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
- Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
- May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
- Thomas McCoy R., 2019, International Conference on Learning Representations
- Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
- Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
- Jiaqi Mu, 2018, International Conference on Learning Representations
- Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
- Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
- Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
- Richardson Kyle, 2020, AAAI 2020
- Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
- Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
- Rogers Anna, 2020, AAAI, 11
- Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
- Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
- Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
- Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
- Strubell Emma, 2019, ACL 2019
- Sugawara Saku, 2020, AAAI
- Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
- Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
- Tenney Ian, 2019, International Conference on Learning Representations
- Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
- Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
- Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
- Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
- Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
- Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
- Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
- Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
- Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
- Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
- Felix Wu, 2019, International Conference on Learning Representations
- Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
- Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
- Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
- Zafrir Ofir, 2019, arXiv preprint arXiv:1910.06188 · arXiv
- Zhang Zhuosheng, 2020, AAAI 2020
- Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
- Zhou Xuhui, 2020, AAAI 2020
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“6 1 The desing of this table is borrowed from [196, 144].”
BackgroundA Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Metadata
- Authors
- Rogers, A., Kovaleva, O., & Rumshisky, A.
- Affiliations
- University of Copenhagen (DK)
- University of Massachusetts Lowell (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 842-866
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3006881356
- Cited by
- 1,552
- References
- 182
- Citations by year
- 2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
- FWCI
- 93.20
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1844740)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.