SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 201–300 of 1,012
A Perspective on Foundation Models in Chemistry
Choi, J., Nam, G., Choi, J., & Jung, Y. (2025) · JACS Au · doi:10.1021/jacsau.4c01160 · Cited by 106
VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
Pârcălăbescu, L., Cafagna, M., Muradjan, L., Frank, A., Calixto, I., & Gatt, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.567 · Cited by 197
“To ensure that foiled and correct captions are similarly plausible, we use language models such as BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2020) to suggest replacements in our foiling functions.”
MethodERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding
Peng, Q., Pan, Y., Wang, W., Luo, B., Zhang, Z., Huang, Z., Cao, Y., Yin, W., Chen, Y., Zhang⋆, Y., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.274 · Cited by 102
Tang, N., Fan, J., Li, F., Tu, J., Du, X., Li, G., Madden, S. R., & Ouzzani, M. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3457390.3457391 · Cited by 88
Enhancing large language model capabilities for rumor detection with Knowledge-Powered Prompting
Yan, Y., Zheng, P., & Wang, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108259 · Cited by 80
Deep Neural Approaches to Relation Triplets Extraction: a Comprehensive Survey
Nayak, T. K., Majumder, N., Goyal, P., & Poria, S. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09917-7 · Cited by 83
“Contextualized word embeddings based model such as ELMo [49], BERT [50], RoBERTa [51], and SpanBERT [52] can also be used for named entity recognition.”
BackgroundMethodInfluentialBeyond word embeddings: A survey
Incitti, F., Urli, F., & Snidaro, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.08.024 · Cited by 97
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
Wang, S., Thompson, L., & Iyyer, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.846 · Cited by 106
“Previously proposed methods to address these issues include predicting spans during pretraining instead of words (Joshi et al., 2019), fine-tuning BERT on shorter texts (Reimers and Gurevych, 2019), and adding an explicit postprocessing step to induce a continuous and isotropic semantic space (Li et…”
InfluentialPALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
Bi, B., Li, C., Wu, C., Yan, M., Wang, W., Huang, S., Huang, F., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.700 · Cited by 60
An Improved Baseline for Sentence-level Relation Extraction
Zhou, W., & Chen, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01373 · Cited by 47
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-TrainingPreprint
Bapna, A., Chung, Y.-A., Wu, N., Gulati, A., Ye, J., Clark, J. H., Johnson, M., Riesa, J., Conneau, A., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10329 · Cited by 111
“We pre-train the model with four objectives: SpanBERT (Joshi et al., 2020) on unlabeled text, w2v-BERT (Chung et al., 2021) on unlabeled speech, Translation Language Modeling (Conneau & Lample, 2019; Zheng et al., 2021) on paired speech and text data, and Speech-Text Matching (Li et al., 2021) on…”
BackgroundMethodInfluentialExploiting Structured Knowledge in Text via Graph-Guided Representation Learning
Shen, T., Mao, Y., He, P., Long, G., Trischler, A., & Chen, W. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.722 · Cited by 63
A Report on the 2020 Sarcasm Detection Shared Task
Ghosh, D., Vajpayee, A., & Muresan, S. (2020) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2020.figlang-1.1 · Cited by 90
AUGER: automatically generating review comments with pre-training models
Li, L., Yang, L., Jiang, H., Yan, J., Luo, T., Hua, Z., Liang, G., & Zuo, C. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549099 · Cited by 57
“Devlin et al. introduced a simple but powerful model named BERT (Bidirectional Encoder Representations from Transformers) to outperform others on eleven NLP tasks with large margin.”
BackgroundInfluentialGraph Representation Learning and Its Applications: A Survey
Hoang, V. T., Jeon, H.-J., You, E.-S., Yoon, Y., Jung, S., & Lee, O. (2023) · Sensors · doi:10.3390/s23084168 · Cited by 81
Evaluating the representational power of pre-trained DNA language models for regulatory genomics
Tang, Z., Somia, N., Yu, Y., & Koo, P. K. (2025) · Genome biology · doi:10.1186/s13059-025-03674-8 · Cited by 85
On Learning Universal Representations Across LanguagesPreprint
Wei, X., Weng, R., Hu, Y., Xing, L., Yu, H., & Luo, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15960 · Cited by 79
“However, this simple binary classification task is not enough to model sentence-level representations (Joshi et al., 2020; Yang et al., 2019; Liu et al., 2019; Lan et al., 2020; Conneau et al., 2020).”
BackgroundQuestion Answering with Long Multiple-Span Answers
Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., & Reddy, C. K. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.342 · Cited by 57
“We also use the QA versions of BERT , SpanBERT (Joshi et al., 2020), and XL-Net , as the baselines.”
MethodSPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 121
Generalizing Natural Language Analysis through Span-relation Representations
Jiang, Z., Xu, W. H., Araki, J., & Neubig, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.192 · Cited by 68
TWilBert: Pre-trained deep bidirectional transformers for Spanish Twitter
González, J. Á., Hurtado, L.-F., & Plà, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.09.078 · Cited by 69
“However, the benefits of the NSP signal have been a controversial topic in the literature [19, 17].”
BackgroundMethodInfluentialPilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 95
“We used the same modified MRQA dataset and preprocessing steps that were used in Joshi et al. (2019).”
MethodCoreference Resolution without Span Representations
Kirstain, Y., Ram, O., & Levy, O. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.3 · Cited by 60
“…of the English OntoNotes 5.0 dataset. c2f refers to the course-to-fine approach of Lee et al. (2017, 2018), as ported to pretrained transformers by Joshi et al. (2019). model over three pretrained models: Longformer-Base, Longformer-Large, and SpanBERT-Large (Beltagy et al., 2020; Joshi et al.,…”
BackgroundMethodInfluentialSelf-citationNamed Entity Recognition in Indian court judgments
Kalamkar, P., Agarwal, A., Tiwari, A. K., Gupta, S., Karn, S., & Raghavan, V. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.15 · Cited by 108
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?
Pruthi, D., Bansal, R., Dhingra, B., Soares, L. B., Collins, M., Lipton, Z. C., Neubig, G., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00465 · Cited by 55
“(2021), and automatically generated explanations from a SpanBERT model (Joshi et al., 2020).”
MethodRelation Classification with Entity Type Restriction
Lyu, S., & Chen, H. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.34 · Cited by 59
PRADA: Practical Black-box Adversarial Attacks against Neural Ranking Models
Wu, C. H., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2022) · ACM Transactions on Information Systems · doi:10.1145/3576923 · Cited by 101
“Specifically, in this work, we choose the fine-tuned BERT model on downstream search tasks for adversarial ranking attack, due to the following: (1) the pre-trained language model BERT has shown good superiority on many text ranking problems [25, 38, 50, 54] in both academia and industry in recent years; and (2) previ…”
BackgroundMethodArabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 52
Enhancing supply chain visibility with knowledge graphs and large language models
AlMahri, S., Xu, L., & Brintrup, A. (2026) · International Journal of Production Research · doi:10.1080/00207543.2025.2575841 · Cited by 38
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation
Guo, J., Xu, L., & Chen, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.36 · Cited by 63
“The masked language model proposed by BERT (Devlin et al., 2018) has become the essential component of the state-of-the-art pre-training methods (Song et al., 2019; Dong et al., 2019; Liu et al., 2019; Joshi et al., 2019; Lample and Conneau, 2019) in natural language understanding tasks.”
MethodTan, X., Zhuang, M., Lu, X. B., & Mao, T. (2021) · IEEE Access · doi:10.1109/access.2021.3052566 · Cited by 62
Train No Evil: Selective Masking for Task-Guided Pre-Training
Gu, Y., Zhang, Z., Wang, X., Liu, Z., & Sun, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.566 · Cited by 77
A survey on semantic processing techniques
Mao, R., He, K., Zhang, X., Chen, G., Ni, J., Yang, Z., & Cambria, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101988 · Cited by 72
CokeBERT: Contextual knowledge selection and embedding towards enhanced pre-trained language models
Su, Y., Han, X., Zhang, Z., Lin, Y., Li, P., Liu, Z., Zhou, J., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.004 · Cited by 72
“SpanBERT (Mandar et al., 2019) and ERNIE 1.”
Decontextualization: Making Sentences Stand-Alone
Choi, E., Palomaki, J., Lamm, M., Kwiatkowski, T., Das, D., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00377 · Cited by 54
Few-shot Intent Classification and Slot Filling with Retrieved Examples
Yu, D., He, L., Zhang, Y., Du, X., Pasupat, P., & Li, Q. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.59 · Cited by 76
Coreference-Aware Dialogue Summarization
Liu, Z., Shi, K., & Chen, N. F. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.53 · Cited by 82
An Enhanced Span-based Decomposition Method for Few-Shot Sequence Labeling
Wang, P., Xu, R., Liu, T., Zhou, Q. S., Cao, Y., Chang, B., & Sui, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.369 · Cited by 76
Make-A-Story: Visual Memory Conditioned Consistent Story Generation
Rahman, T., Lee, H. Y., Ren, J., Tulyakov, S., Mahajan, S., & Sigal, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00246 · Cited by 130
“More recent approaches [13, 24, 28] leverage neural network architectures to obtain improved performance with Transformers [9].”
MethodMachine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 79
“BERT (Joshi et al. 2020) proposed a random span masking strategy based on geometric distribution, indicating that the proposed masking sometimes works even better than masking linguistically-coherent spans.”
BackgroundMethodInfluentialImproving End-to-End Task-Oriented Dialog System with A Simple Auxiliary Task
Lee, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.112 · Cited by 55
Jin, W., Zhao, B., Zhang, Y., Huang, J., & Yu, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122289 · Cited by 64
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Zhang, Z., Chen, S., Zhou, L., Wu, Y., Ren, S., Liu, S., Yao, Z., Gong, X., Dai, L.-R., Li, J., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3379877 · Cited by 72
“Extending SLAM to the multilingual scenario, mSLAM [7] introduces CTC losses and uses SpanBERT [9] to replace the BERT objective for pre-training on character-level text.”
MethodTextHide: Tackling Data Privacy in Language Understanding Tasks
Huang, Y., Song, Z., Chen, D., Li, K., & Arora, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.123 · Cited by 77
Self-citationAMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
Zhang, X., Li, P., & Li, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.37 · Cited by 68
BERT & Family Eat Word Salad: Experiments with Text Understanding
Gupta, A., Kvernadze, G., & Srikumar, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17531 · Cited by 76
HerBERT: Efficiently Pretrained Transformer-based Language Model for PolishPreprint
Mroczkowski, R., Rybak, P., Wróblewska, A., & Gawlik, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01735 · Cited by 146
“Training Objectives We trained all models with an updated version of the MLM objective (Joshi et al., 2019; Martin et al., 2020), masking ranges of subsequent tokens belonging to single words instead of individual (possibly subword) tokens.”
MethodUnsupervised Text Style Transfer with Padded Masked Language Models
Malmi, E., Severyn, A., & Rothe, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.699 · Cited by 54
Incomplete Utterance Rewriting as Semantic Segmentation
Liu, Q., Chen, B., Lou, J., Zhou, B., & Zhang, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.227 · Cited by 72
Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction
Hu, X., Zhang, C., Yang, Y., Li, X., Li, L., Wen, L., & Yu, P. S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.216 · Cited by 73
“Inspired by Joshi et al. (2020), we sample a span length from a geometric distribution (cid:96) ∼ Geo( p ) where (cid:96) ∈ [1 , 10] . p will affect the probability of selecting different span lengths.”
BackgroundMethodInfluentialLi, J., Wei, Q., Ghiasvand, O., Chen, M., Lobanov, V. S., Weng, C., & Xu, H. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01967-7 · Cited by 63
“…embedding models (named BioBERT [7], BlueBERT [8], PubMedBERT [9], and SciBERT [10]) and two general-domain models (named BERT and SpanBERT [11]), for extracting diverse types of clinically relevant entities from three annotated clinical trials corpora: (1) 470 in-house drug development…”
BackgroundHe, X., Zannettou, S., Shen, Y., & Zhang, Y. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00061 · Cited by 35
“Most of previous work [12], [22], [18] frame this task as a sequence labeling task.”
MethodInfluentialUnsupervised Multi-hop Question Answering by Question Generation
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.469 · Cited by 76
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
Zellers, R., Holtzman, A., Peters, M., Mottaghi, R., Kembhavi, A., Farhadi, A., & Choi, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.159 · Cited by 48
“We masked inputs in the bidirectional prefix with Span-BERT masking (Joshi et al., 2020).”
MethodAccelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
Zhang, M., & He, Y. (2020) · arXiv · doi:10.48550/arxiv.2010.13369 · Cited by 50
“Finally, there are multiple research proposing different enhanced versions of Transformer-based networks, such as GPT-2/3 [9, 10], XLNet [1], SpanBERT [11], BioBERT [12], UniLM [13], Turing-NLG [14], and T5 [15].”
BackgroundLess is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T.-Y., & Overwijk, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.220 · Cited by 49
“It first randomly masks some tokens in a sequence and then pre-trains a Transformer to recover them (Joshi et al., 2020; Liu et al., 2019; Clark et al., 2020).”
Multi-task Pre-training Language Model for Semantic Network Completion
Li, D., Zhu, B., Yang, S., Xu, K., Ming, Y., He, Y., & Wang, H. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3627704 · Cited by 73
“SpanBERT [39] extended the BERT by masking contiguous random spans rather than random tokens, and training the span boundary representations to predict the entire content of the masked span without relying on the individual token representations within it.”
BackgroundGeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
Li, Z., Zhou, W., Chiang, Y., & Chen, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.317 · Cited by 104
Krasadakis, P., Sakkopoulos, E., & Verykios, V. S. (2024) · Electronics · doi:10.3390/electronics13030648 · Cited by 77
“Then, they advanced their work with SpanBERT [50].”
BackgroundMixKD: Towards Efficient Distillation of Large-scale Language ModelsPreprint
Liang, K. J., Hao, W., Shen, D., Zhou, Y., Chen, W., Chen, C., & Carin, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00593 · Cited by 87
“…in a self-supervised manner have significantly advanced the state of the art across a wide variety of natural language processing (NLP) tasks (Devlin et al., 2018; Liu et al., 2019c; Yang et al., 2019; Joshi et al., 2020; Sun et al., 2019b; Clark et al., 2020; Lewis et al., 2019; Bao et al., 2020).”
BackgroundSmadi, B. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.107921 · Cited by 51
Shen, T., Quach, V., Barzilay, R., & Jaakkola, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.420 · Cited by 44
Knowledge distillation for BERT unsupervised domain adaptation
Ryu, M., Lee, G., & Lee, K. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01736-y · Cited by 51
“Moreover, to see the baseline model effect, we also evaluate the model performance with the baseline models by DistilBERT (Sanh et al., 2019) and RoBERT (Liu et al., 2019) models respectively.”
MethodInfluentialBootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
Orr, L., Leszczynski, M., Arora, S., Wu, S., Guha, N., Xiao, L., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10363 · Cited by 40
“Given an example, we run inference with the Bootleg model to disambiguate named entities and generate the contextual Bootleg entity embedding matrix, which we feed to a simple Transformer architecture that uses SpanBERT [27] (details in Appendix C).”
Sun, Y., Zheng, Y., Hao, C., & Qiu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03564 · Cited by 83
MeshMAE: Masked Autoencoders for 3D Mesh Data Analysis
Liang, Y., Zhao, S., Yu, B., Zhang, J., & He, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20062-5_3 · Cited by 90
SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
Li, Z., Kim, J., Chiang, Y., & Chen, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.200 · Cited by 90
A Survey for Efficient Open Domain Question Answering
Zhang, Q., Chen, S., Xu, D., Cao, Q., Chen, X., Cohn, T., & Fang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.808 · Cited by 89
Fine-Grained Propaganda Detection with Fine-Tuned BERT
Yoosuf, S., & Yang, Y. (2019) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/d19-5011 · Cited by 68
“One related direction of research is spanBERT (Joshi et al., 2019), which includes a pretraining phase consisting of predicting spans instead of tokens which is inherently more suited for the propaganda dataset.”
QED: A Framework and Dataset for Explanations in Question Answering
Lamm, M., Palomaki, J., Alberti, C., Andor, D., Choi, E., Soares, L. B., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00398 · Cited by 43
“Figure 5 7For formal discussion, see (Carlson, 1977; Krifka, 2003; Abbott, 2004; Mikkelsen, 2011) among others. shows an analysis of 100 referential equality annotations from QED, with a breakdown by type of referring expression in the question and passage.”
BackgroundSelf-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering
You, C., Chen, N., & Zou, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.3 · Cited by 61
PMI-Masking: Principled masking of correlated spansPreprint
Levine, Y., Lenz, B., Lieber, O., Abend, O., Leyton‐Brown, K., Tennenholtz, M., & Shoham, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01825 · Cited by 59
“Random-Span Masking (Joshi et al., 2020) Contiguous random spans are selected iteratively until the 15% masking budget is spent.”
InfluentialSurvey of Neural Text Representation Models
Babić, K., Martinčić-Ipšić, S., & Meštrović, A. (2020) · Information · doi:10.3390/info11110511 · Cited by 54
DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding
Zhang, T., Wang, C., Hu, N., Qiu, M., Tang, C., He, X., & Huang, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21425 · Cited by 45
A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural NetworksPreprint
Mao, H. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00800 · Cited by 73
“Several extensions of BERT have been proposed, such as SpanBERT (Joshi et al., 2020), a training procedure that masks out contiguous spans instead of individual tokens, and ERNIE (Sun et al., 2019b), which masks out full entities or phrase-level units.”
MethodChen, M., Ge, T., Zhang, X., Wei, F., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.581 · Cited by 46
Zhang, T., Cai, Z., Wang, C., Qiu, M., Yang, B., & He, X. (2021) · doi:10.18653/v1/2021.acl-long.457 · Cited by 61
Enhanced-RCNN: An Efficient Method for Learning Sentence Similarity
Peng, S., Cui, H., Xie, N., Li, S., Zhang, J., & Li, X. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3379998 · Cited by 45
“The last framework is called “BERT-based" methods which fine-tune the pre-trained deep language models directly with minimal modifications to perform downstream tasks, including learning sentence similarity [9, 12, 31].”
MethodMining Root Cause Knowledge from Cloud Service Incident Investigations for AIOps
Saha, A., & Hoi, S. C. H. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3510457.3513030 · Cited by 61
“Specifically we apply an ensemble of BERT models (DistilBERT, BERT-base and BERT-large)[7], RoBERTa [19] and SpanBERT [14], each fine-tuned on the standard open-domain extractive-QA dataset SQuAD.”
Unsupervised Pre-training for Biomedical Question AnsweringPreprint
Kommaraju, V., Gunasekaran, K. P., Li, K., Bansal, T., McCallum, A., Williams, I., & Istrate, A.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12952 · Cited by 51
“Recently, selfsupervised pre-training of transformer models [6,17,24] with language modeling objectives has been shown to learn good feature representations and improve performance on many question-answering tasks [6,9,10].”
BackgroundMethodGAP: A novel Generative context-Aware Prompt-tuning method for relation extraction
Chen, Z., Li, Z., Zeng, Y., Zhang, C., & Ma, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123478 · Cited by 51
Pre-training Entity Relation Encoder with Intra-span and Inter-span Information
Wang, Y., Sun, C., Wu, Y., Yan, J., Gao, P., & Xie, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.132 · Cited by 53
“We generate spans similar to (Joshi et al., 2020).”
BackgroundMethodInfluentialUsing Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks
Xia, T., Wang, Y., Tian, Y., & Chang, Y. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449988 · Cited by 64
“These include other unsupervised pre-training tasks such as span prediction [21] and domain-specific pre-training [14], as well as knowledge-based tasks such as semantic role prediction [48], entity recognition [47], and relation prediction [26].”
BackgroundLattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
Lai, Y., Liu, Y., Feng, Y., Huang, S., & Zhao, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.137 · Cited by 58
Self-citationUnified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation
He, W., Dai, Y., Yang, M., Sun, J. Q., Huang, F., Si, L., & Li, Y. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532069 · Cited by 69
“Span Masking [41] improves the original MLM by masking random contiguous spans rather than random individual tokens.”
BackgroundMUDES: Multilingual Detection of Offensive Spans
Ranasinghe, T., & Zampieri, M. (2021) · doi:10.18653/v1/2021.naacl-demos.17 · Cited by 63
Metacognitive Retrieval-Augmented Large Language Models
Zhou, Y., Liu, Z., Jin, J., Nie, J.-Y., & Dou, Z. (2024) · Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589334.3645481 · Cited by 38
“The fine-tuned QA models SpanBERT-large [13] and T5-large are smaller but have been specifically trained on particular datasets.”
BackgroundSLM: Learning a Discourse Language Representation with Sentence Unshuffling
Lee, H., Hudson, D. A., Lee, K., & Manning, C. D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.120 · Cited by 41
Pre-training for Abstractive Document Summarization by Reinstating Source Text
Zou, Y., Zhang, X., Lu, W., Wei, F., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.297 · Cited by 51
“SpanBERT (Joshi et al., 2020) applied only the masked language modeling objective that masks contiguous random spans, rather than random tokens.”
MethodLearning to Ignore: Long Document Coreference with Bounded Memory Neural Networks
Toshniwal, S., Wiseman, S., Ettinger, A., Livescu, K., & Gimpel, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.685 · Cited by 76
Summarization as Indirect Supervision for Relation Extraction
Lu, K., Hsu, I.-H., Zhou, W., Ma, M. D., & Chen, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.490 · Cited by 70
A neural Entity Coreference Resolution review
Stylianou, N., & Vlahavas, I. P. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114466 · Cited by 57
Xiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.136 · Cited by 57
Zero-Shot Information Extraction as a Unified Text-to-Triple Translation
Wang, C., Liu, X., Chen, Z., Hong, H., Tang, J., & Song, D. X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.94 · Cited by 67
“…e.g., BERT (Devlin et al., 2019), GPT (Radford et al., 2018, 2019; Brown et al., 2020), RoBERTa (Liu et al., 2019), transfer well to relation classification datasets in fine-tuning (Joshi et al., 2020; Gao et al., 2019) or few-shot regime (Soares et al., 2019) with architecture modifications.”
BackgroundZhou, Y., Shen, T., Geng, X., Long, G., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.183 · Cited by 113
A Study of Text Vectorization Method Combining Topic Model and Transfer Learning
Yang, X. C., Yang, K., Cui, T., Chen, M., & He, L. (2022) · Processes · doi:10.3390/pr10020350 · Cited by 43
Bridging Anaphora Resolution as Question Answering
Hou, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.132 · Cited by 62
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
Ghaddar, A., Langlais, P., Rashid, A., & Rezagholizadeh, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00386 · Cited by 57
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.