SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
“However, some recent work has questioned the necessity of the NSP loss (Lample and Conneau, 2019; Yang et al., 2019; Joshi et al., 2019).”
BackgroundMethodSelf-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
“4, which was loosely inspired by SpanBERT [Joshi et al., 2019].”
BackgroundMethodInfluentialLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Self-citationLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
“Recent work has shown that very strong performance can be achieved by training for longer (Liu et al., 2019), by tying parameters across layers (Lan et al., 2019), and by masking spans instead of words (Joshi et al., 2019).”
BackgroundMethodInfluentialSelf-citationTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
“…classification (Yang et al., 2019), language understanding (Liu et al., 2019b; Wang et al., 2018, 2019), machine translation (Lample and Conneau, 2019a), coreference resolution (Joshi et al., 2019), commonsense inference (Bosselut et al., 2019), and summarization (Lewis et al., 2019) among others.”
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
“, SpanBERT [63], RoBERTa [82]) are a series of PTMs built on the multi-layer Transformer encoder architecture.”
BackgroundALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
“, 2019) up to co-reference resolution (Joshi et al., 2019) and commonsense inference (Bosselut et al.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Deep Learning--based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2022) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,538
“SpanBERT [149] extends BERT to better represent and predict text spans.”
BackgroundSelf-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,740
“SpanBERT [118], for example, shows that instead of masking out single tokens for language modeling, masking out multiple tokens at a time, known as spans, results in better downstream performance.”
BackgroundTinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Pre-Training With Whole Word Masking for Chinese BERT
Cui, Y., Che, W., Liu, T., Qin, B., & Yang, Z. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3124365 · Cited by 1,181
“Phrase-level masking is to mask consecutive words, which is similar to the N-gram masking strategy [2], [14], [19].”
BackgroundMethodPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
“• Removing NSP does not hurt or slightly improves task performance (Liu et al., 2019b; Joshi et al., 2020; Clinchant et al., 2019), especially in cross-lingual setting (Wang et al., 2019b).”
BackgroundBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
“[DBK+20] also report that pixel-level auto-encoding performs worse, although it is the most straightforward translation of BERT from NLP to CV.”
MethodInfluentialRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“Text inilling, which is closely related to the span-based MLM used in SpanBERT [76], along with sentence permutation turn out to be the best objectives for BART.”
MethodInfluentialUnifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
“This work is followed by the introduction of SpanBERT [155] which is pre-trained on BERT architecture with a span-based masked language model (MLM).”
BackgroundMethodBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
“Most prominently, SpanBERT [42], ORQA [54], REALM [34], RAG [57] have achieved strong performance for different tasks.”
MethodDeBERTa: Decoding-enhanced BERT with Disentangled Attention
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 420
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206
“Transformer-based sequence models has led to a dramatic improvement in performance when finetuned for both natural language understanding and text generation tasks (Radford et al., 2018a; Devlin et al., 2019; Rothe et al., 2019; Yang et al., 2019; Joshi et al., 2019; Song et al., 2019; Dong et al., 2019; Lewis et al.,…”
BackgroundDEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
He, P., Liu, X., Gao, J., & Chen, W. (2021) · International Conference on Learning Representations · Cited by 2,445
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,037
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
“For example, XL-Net (Yang et al., 2019) encodes the original position so that it can perceive the number of missing tokens, and SpanBERT (Joshi et al., 2020) replaces the span with multiple [MASK] tokens and keeps the length unchanged.”
BackgroundPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Self-citationCamemBERT: a Tasty French Language Model
Martin, L., Muller, B., Ortiz Suárez, P. J., Dupont, Y., Romary, L., de la Clergerie, É., Seddah, D., & Sagot, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.645 · Cited by 754
Revisiting Pre-Trained Models for Chinese Natural Language Processing
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., & Hu, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.58 · Cited by 578
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
“In addition, we replace the next sentence prediction head with sentence order prediction as suggested by (Lan et al., 2019) and use whole word n-gram masking of (Joshi et al., 2019).”
MethodPoint-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01871 · Cited by 801
“Transformers [49] have become the dominant framework in NLP [3, 8, 18, 25, 34] due to its salient benefits, including massively parallel computing, longdistance characteristics, and minimal inductive bias.”
BackgroundMethodInfluentialK-BERT: Enabling Language Representation with Knowledge Graph
Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5681 · Cited by 894
“SpanBERT (Joshi et al. 2019) extended BERT by masking contiguous random spans and proposed a span boundary objective.”
BackgroundBitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.195 · Cited by 757
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Huang, Y., Lv, T., Cui, L., Lu, Y., & Wei, F. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548112 · Cited by 719
“We mask 30% of text tokens with a span masking strategy with span lengths drawn from a Poisson distribution (λ = 3) [20, 24].”
MethodMPNet: Masked and Permuted Pre-training for Language Understanding
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09297 · Cited by 514
“Under this unified view, we propose a new pre-training method, masked and permuted language modeling (MPNet for short), which addresses the issues in both MLM and PLM while inherits their advantages: 1) It takes the dependency among the predicted tokens into consideration through permuted language modeling and thus av…”
MethodA comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
Self-citationMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
“…2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019; Song et al., 2019; Dong et al., 2019; Yang et al., 2019; Joshi et al., 2019; Liu et al., 2019; Lewis et al., 2019a; Raffel et al., 2019) has achieved significant improvements for a wide range of NLP tasks.”
BackgroundMethodA Survey on Text Classification: From Traditional to Deep Learning
Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P. S., & He, L. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3495162 · Cited by 494
“The architecture of BART [202] and SpanBERT [203].”
MethodLUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
Yamada, I., Asai, A., Shindo, H., Takeda, H., & Matsumoto, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.523 · Cited by 590
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
“While such salient span masking has not been shown to be impactful in previous work with standard BERT training (Joshi et al., 2019), it is crucial for REALM.”
BackgroundInfluentialPTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2022) · AI Open · doi:10.1016/j.aiopen.2022.11.003 · Cited by 467
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478
“For those knowledge-enhanced PLMs, we select SPANBERT (Joshi et al., 2020), KNOWBERT (Peters et al.”
MethodCao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
Self-citationChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Attention mechanism in neural networks: where it comes and where it goes
Soydaner, D. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07366-3 · Cited by 414
“In order to obtain better representations, SpanBERT is proposed as a pre-training method [151].”
MethodLearning inverse folding from millions of predicted structures
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., & Rives, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 416
“Such a span masking scheme has shown to improve performance on natural language processing benchmarks (Joshi et al., 2020).”
BackgroundDeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.72 · Cited by 350
“They propose swapping NSP for a Sentence-Order Prediction objective (SOP), which they suggest better models inter-sentence coherence.”
BackgroundInfluentialA Robustly Optimized BERT Pre-training Approach with Post-training
Liu, Z., Lin, W., Shi, Y., & Zhao, J. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-84186-7_31 · Cited by 403
“Other approach is to optimize pre-training, like GP-T 2.0 (Radford et al., 2018), MT-DNN (Liu et al., 2020a), SpanBERT (Joshi et al., 2019), and AL-BERT (Lan et al., 2020).”
MethodInfluentialvq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 388
“We then train a Deep Bidirectional Transformer (BERT; Devlin et al., 2018; Liu et al., 2019) on the discretized unlabeled speech data and input these representations to a standard acoustic model (Figure 1b; §4).”
MethodSpan-Based Joint Entity and Relation Extraction with Transformer Pre-Training
Markus, E., & Adrian, U. (2020) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia200321 · Cited by 168
“Applications include coreference resolution [14, 15], semantic role labeling [25, 12], and the improvement of language modeling by learning to predict spans instead of single words [13].”
Background12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01045 · Cited by 410
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 385
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 374
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389
Deep Learning Based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Asgari-Chenaghlu, M., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03705 · Cited by 381
“SpanBERT [149] extends BERT to better represent and predict text spans.”
BackgroundLinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.551 · Cited by 327
Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Zhao, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.197 · Cited by 162
“BERT motivated many follow-up works to further improve the pre-training by introducing new unsupervised learning tasks (Yang et al., 2019; Dong et al., 2019; Joshi et al., 2020), enlarging model size (Lan et al.”
BackgroundFinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining
Liu, Z., Huang, D., Huang, K., Li, Z., & Zhao, J. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/622 · Cited by 288
BERT for Coreference Resolution: Baselines and Analysis
Joshi, M., Levy, O., Zettlemoyer, L., & Weld, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1588 · Cited by 312
“Also concurrent, SpanBERT (Joshi et al., 2019), another self-supervised method, pretrains span representations achieving state of the art results (Avg.”
BackgroundMethodInfluentialSelf-citationCAT: Cross Attention in Vision Transformer
Lin, H., Cheng, X., Wu, X., & Shen, D. (2022) · 2022 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859720 · Cited by 269
A transformer-based approach to irony and sarcasm detection
Potamias, R. A., Siolas, G., & Stafylopatis, A. G. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05102-3 · Cited by 278
“studies that question the NSP loss term [44, 55, 101].”
BackgroundEvidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H., & Wang, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.202 · Cited by 231
“We sample the span length from a geometric distribution l ∼ Geo ( p ) , where p is set as 0.2, similar to SpanBERT (Joshi et al., 2020).”
MethodInfluentialBERT applications in natural language processing: a review
Gardazi, N. M., Daud, A., Malik, M. K., Bukhari, A., Alsahfi, T., & Alshemaimri, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 251
“Chen et al. ( 2020 ) and Chinnalagu and Durairaj ( 2022) further support this, with the former proposing a fine-grained sentiment analysis model based on BERT and the latter finding BERT to outperform other deep learning models.”
Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Karthikeyan, K., Wang, Z., Mayhew, S., & Roth, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07840 · Cited by 148
“We study the effect of NSP as recent works (Conneau & Lample, 2019; Joshi et al., 2019; Liu et al., 2019) show that the NSP objective hurts the performance on several monolingual tasks.”
BackgroundResultSKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis
Tian, H., Gao, C., Xiao, X., Liu, H., He, B., Wu, H., Wang, H., & wu, F. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.374 · Cited by 272
Kostas, D., Aroca-Ouellette, S., & Rudzicz, F. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.653659 · Cited by 272
“Masking a contiguous span of tokens instead, which is beneficial in NLP [42, 34], could avoid simply learning to interpolate missing samples, but the reconstruction of time-series data is difficult, due to the difficulty (among other things) of capturing the degree of error in time (within contiguous…”
BackgroundSentiment analysis methods, applications, and challenges: A systematic literature review
Mao, Y., Liu, Q., & Zhang, Y. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102048 · Cited by 206
“There are many improved version of BERT were proposed, such as RoBERTa (Liu et al., 2019), SpanBERT (Joshi et al., 2020), KBERT (Liu et al., 2020), ELECTRA (Clark et al., 2020).”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wang, W., Bao, H., Huang, S., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.188 · Cited by 257
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
“Span Boundary Objective (SBO) [49] .”
BackgroundMethodInfluentialLarge language models for human–robot interaction: A review
Zhang, C., Chen, J., Li, J., Peng, Y., & Mao, Z. (2023) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2023.100131 · Cited by 254
RobBERT: a Dutch RoBERTa-based Language Model
Delobelle, P., Winters, T., & Berendt, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.292 · Cited by 235
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12804 · Cited by 254
“For the i-th factorization step, the masked position set Mi contains one token, or a continuous text span (Joshi et al., 2019).”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Wang, W., Bi, B., Yan, M., Wu, C., Bao, Z., Xia, J., Peng, L., & Si, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04577 · Cited by 91
“0 SpanBERT [12] 64.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 426
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2023) · arXiv · doi:10.48550/arxiv.2306.11489 · Cited by 239
“The training objective of the SBO task can be expressed as the subsequent equation: LSBO = − T∑ t=1 log p(xi|yi), (6) where yi is token xi’s representation in the span.”
InfluentialYang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
“Although there exist a few surveys [17], [18], [19] of knowledge-enhanced PLMs, 1041-4347 © 2024 IEEE.”
BackgroundRetrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 143
“After that, many OpenQA systems [17], [28], [34], [35], [78], [81] develop their readers by applying this mechanism based on original MRC models like BiDAF [23], BERT [26] and SpanBERT [102].”
Dash, A., Ye, J., & Wang, G. (2024) · IEEE Access · doi:10.1109/access.2023.3346273 · Cited by 211
“The model comprises of three loss functions: adversarial loss, cycle-consistent loss, and generation-consistent loss.”
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.55 · Cited by 230
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2021) · arXiv · doi:10.48550/arxiv.2112.07916 · Cited by 230
“Language model pre-training followed by task specific fine-tuning has proven to be a powerful tool for numerous NLP tasks (Devlin et al., 2019; Liu et al., 2019; Zhang et al., 2019b; Radford et al., 2019; Raffel et al., 2019a; Lewis et al., 2020; Joshi et al., 2020).”
MERLOT: Multimodal Neural Script Knowledge Models
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 53
“We then apply SpanBERT masking [45], randomly corrupting the following or preceding tokens with an average length of 0.”
BackgroundMethodSelf-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 189
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
Sun, Z., Li, X., Sun, X., Meng, Y., Ao, X., He, Q., Wu, F., & Li, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.161 · Cited by 203
Aspect-based Sentiment Analysis with Type-aware Graph Convolutional Networks and Layer Ensemble
Tian, Y., Chen, G., & Song, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.231 · Cited by 187
“…Song et al., 2018), we employ BERT (Devlin et al., 2019) as the context encoder, which and whose variants (Diao et al., 2020; Dai et al., 2019; Joshi et al., 2020) have demonstrated their effectiveness to encode context information and achieved state-of-the-art performance in many NLP tasks…”
MethodBhattacharjee, A., Hasan, T., Ahmad, W., Mubasshir, K. S., Islam, M. S., Iqbal, A., Rahman, M. S., & Shahriyar, R. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.98 · Cited by 182
“Joshi et al. (2020b) categorized Bangla in the language group that lacks efforts in labeled data collection and relies on self-supervised pretraining (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019) to boost the natural language understanding (NLU) task performances.”
BackgroundMethodBridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
Lin, X. V., Socher, R., & Xiong, C. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.438 · Cited by 181
“For example, modeling its span structure (Joshi et al., 2019; Herzig and Berant, 2020) and constructing interpretable grounding with the DB schema.”
MethodTowards Natural Language Interfaces for Data Visualization: A Survey
Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., & Wang, J. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3148007 · Cited by 181
“DataBase (NLI4DB) A/B/E/F/G [6], [290] [13], [14], [15], [19], [22], [55], [72], [73], [74], [80], [87], [92], [95], [118], [135], [136], [137],”
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194
“SpanBERT [62] generalized ERNIE to mask random spans, without indicating to external knowledge.”
BackgroundGeneral Facial Representation Learning in a Visual-Linguistic Manner
Zheng, Y., Yang, H., Zhang, T., Bao, J., Chen, D., Huang, Y., Yuan, L., Chen, D., Zeng, M., & Wen, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01814 · Cited by 182
“sion appears to be paved by a popular learning paradigm, pre-training, due to the remarkable success of the groundbreaking models in Natural Language Processing such as BERT [24] and GPT-series [5, 74, 75], followed by a wide variety of multiple techniques [25, 45, 52, 71, 76].”
BackgroundSelf-citationERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., Lu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.02137 · Cited by 186
“0 and SpanBERT [32], in which the base/large model size represent 12/24 layers Transformer respectively.”
BackgroundInfluentialEnd-to-End Transformer-Based Models in Textual-Based NLP
Rahali, A., & Akhloufi, M. A. (2023) · AI · doi:10.3390/ai4010004 · Cited by 164
A Complete Process of Text Classification System Using State-of-the-Art NLP Models
Dogra, V., Verma, S., Kavita, Chatterjee, P., Shafi, J., Choi, J., & Ijaz, M. F. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/1883698 · Cited by 164
“Another variant of BERT, SpanBERT [158], is a pretraining method designed to accurately represent and forecast spans of text.”
MethodBlockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 161
Self-citationTransformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. (2023) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 164
“Another remarkable variation of BERT is spanBERT [147].”
BackgroundA Survey on Contextual Embeddings
Coelho, J., Neto, A., Tavares, M., Coutinho, C., Ribeiro, R., & Batista, F. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.12 · Cited by 109
Coreferential Reasoning Learning for Language Representation
Ye, D., Lin, Y., Du, J., Liu, Z., Li, P., Sun, M., & Liu, Z. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.582 · Cited by 163
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.