SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 101–200 of 1,012
SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge
Ke, P., Ji, H., Liu, S., Zhu, X., & Huang, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.567 · Cited by 143
“4) Pre-training tasks: SpanBERT (Joshi et al., 2020) masks consecutive spans randomly instead of individual tokens, while XLNet (Yang et al., 2019) designs a training objective combining both reconstruction and autoregressive language modeling.”
MethodA comprehensive survey on relation extraction: recent advances and new frontiers
Zhao, X., Deng, Y., Yang, M., Wang, L., Zhang, R., Cheng, H., Lam, W., Shen, Y., & Xu, R. (2024) · ACM Computing Surveys · doi:10.1145/3674501 · Cited by 168
The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
Singh, S., & Mahmood, A. (2021) · IEEE Access · doi:10.1109/access.2021.3077350 · Cited by 158
“This might be acceptable for co-reference resolution tasks that benefit from smaller input lengths [43], however, is inadequate for Question Answering (QA) tasks where reasoning is required across multiple lengthy documents e.”
BackgroundMERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01589 · Cited by 160
“We use past work in language modeling as a guide [63, 92] and wanted an average span length of around 5 tokens.”
BackgroundMethodInfluentialShould You Mask 15% in Masked Language Modeling?
Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.217 · Cited by 145
Self-citationHong, T., Kim, D., Ji, M., Hwang, W., Nam, D., & Park, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21322 · Cited by 243
“The area-masked LM, inspired by SpanBERT (Joshi et al. 2020), captures consecutive text blocks based on a 2D area in a document.”
MethodInfluentialMinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems
Lin, Z., Madotto, A., Winata, G. I., & Fung, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.273 · Cited by 224
A Long-Text Classification Method of Chinese News Based on BERT and CNN
Chen, X., Cong, P., & Lv, S. (2022) · IEEE Access · doi:10.1109/access.2022.3162614 · Cited by 190
Masked Discrimination for Self-supervised Learning on Point Clouds
Liu, H., Cai, M., & Lee, Y. J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20086-1_38 · Cited by 280
An extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153
Tracing the evolution of AI in the past decade and forecasting the emerging trends
Shao, Z., Zhao, R., Yuan, S., Ding, M., & Wang, Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118221 · Cited by 153
CorefQA: Coreference Resolution as Query-based Span Prediction
Wu, W., Wang, F., Yuan, A., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.622 · Cited by 152
(2021) · doi:10.1145/3462757.3466088 · Cited by 151
Xu, H., Zhou, P., Tan, R., Li, M., & Shen, G. (2021) · Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3485730.3485937 · Cited by 150
SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics
Yin, D., Meng, T., & Chang, K.-W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.341 · Cited by 184
ConvBERT: Improving BERT with Span-based Dynamic Convolution
Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02496 · Cited by 126
“…models, more effective pre-training objectives have been developed, including Masked Language Modeling and Next Sentence Prediction from BERT [34], Generalized Autoregressive Pretraining from XLNet [69], Span Boundary Objective from SpanBERT [26], and Replaced Token Detection from ELECTRA [8].”
MethodDeep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053176 · Cited by 145
“A similar effect was found in SpanBERT [29], another deep contextual word representation which found that masking a mean span of 3.”
ResultInfluentialSelf-citationTACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
Alt, C., Gabryszak, A., & Hennig, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.142 · Cited by 132
Packed Levitated Marker for Entity and Relation Extraction
Ye, D., Lin, Y., Li, P., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.337 · Cited by 144
“Joshi et al. (2020) further learns to store the span information in its boundary tokens for downstream tasks.”
CrossNER: Evaluating Cross-Domain Named Entity Recognition
Liu, Z., Xu, Y., Yu, T., Dai, W., Ji, Z., Cahyawijaya, S., Madotto, A., & Fung, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17587 · Cited by 260
Structured Prediction as Translation between Augmented Natural LanguagesPreprint
Paolini, G., Athiwaratkun, B., Krone, J., Ma, J., Achille, A., Anubhai, R., dos Santos, C. N., Xiang, B., & Soatto, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05779 · Cited by 430
Mole-BERT: Rethinking Pre-training Graph Neural Networks for MoleculesPreprint
Xia, J., Zhao, C., Hu, B., Gao, Z., Tan, C., Liu, Y., Li, S., & Li, S. Z. (2023) · ChemRxiv · doi:10.26434/chemrxiv-2023-dngg4 · Cited by 258
Self-citationEnabling Language Models to Fill in the Blanks
Donahue, C., Lee, M., & Liang, P. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.225 · Cited by 125
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 191
Entities as Experts: Sparse Memory Access with Entity Supervision
Févry, T., Baldini Soares, L., FitzGerald, N., Choi, E., & Kwiatkowski, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.400 · Cited by 130
Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning
Wang, N., Bian, J., Li, Y., Li, X., Mumtaz, S., Kong, L., & Xiong, H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00836-4 · Cited by 132
“In the TBTH architecture, the RNAErnie approach directly combines the RNAErnie basic block and the task-specific head to construct an end-to-end neural network for downstream tasks and then trains the neural network using the labelled datasets in a supervised learning manner.”
MethodTraining Question Answering Models From Synthetic Data
Puri, R., Spring, R., Shoeybi, M., Patwary, M. A., & Catanzaro, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.468 · Cited by 217
Recognizing Emotion Cause in Conversations
Poria, S., Majumder, N., Hazarika, D., Ghosal, D., Bhardwaj, R., Yu, S., Hong, P., Ghosh, R., Roy, A., Chhaya, N., et al. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09925-7 · Cited by 204
“SpanBERT Fine-tuned on SQuAD We use SpanBERT (Joshi et al., 2020) as the second baseline model.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 175
Cross-Lingual Natural Language Generation via Pre-Training
Chi, Z., Dong, L., Wei, F., Wang, W., Mao, X.-L., & Huang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6256 · Cited by 131
BERT based clinical knowledge extraction for biomedical knowledge graph construction and analysis
Harnoune, A., Rhanoui, M., Mikram, M., Yousfi, S., Elkaimbillah, Z., & Asri, B. E. (2021) · Computer Methods and Programs in Biomedicine Update · doi:10.1016/j.cmpbup.2021.100042 · Cited by 163
“Joshi et al. [47] presented SpanBERT which is a pre-training method designed to better represent and predict text spans.”
MethodCTRLsum: Towards Generic Controllable Text Summarization
He, J., Kryscinski, W., McCann, B., Rajani, N., & Xiong, C. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.396 · Cited by 100
Towards Long-Form Video Understanding
Wu, C.-Y., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00192 · Cited by 222
“, [34]), while additionally using ‘next-sentence-prediction’ helps on others (e.”
MethodSamant, R. M., Bachute, M. R., Gite, S., & Kotecha, K. (2022) · IEEE Access · doi:10.1109/access.2022.3149798 · Cited by 123
“SpanBERT [68] is a BERT extension that improves the representation and prediction of text spans.”
BackgroundHubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414460 · Cited by 119
“Regarding the first decision, we adopt the same strategies used in SpanBERT [21] and wav2vec 2.0 [11] for mask generation, where p % of the frames are randomly selected as start indices, and spans of l frames are masked.”
MethodInfluentialXiao, D., Zhang, H., Li, Y., Sun, Y., Tian, H., Wu, H., & Wang, H. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/553 · Cited by 121
MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
Zeng, M., Tan, X., Wang, R., Ju, Z., Qin, T., & Liu, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.70 · Cited by 171
Re2G: Retrieve, Rerank, Generate
Glass, M., Rossiello, G., Chowdhury, M. F. M., Naik, A., Cai, P., & Gliozzo, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.194 · Cited by 79
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120
RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
Xiao, S., Liu, Z., Shao, Y., & Cao, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.35 · Cited by 283
“For example, tasks like entity masking, phrase masking and span masking (Sun et al., 2019; Joshi et al., 2020) may help the pre-trained models to better support the sequence labeling applications, such as entity resolution and question answering.”
BackgroundMedChatZH: A tuning LLM for traditional Chinese medicine consultations
Tan, Y., Zhang, Z., Li, M., Pan, F., Duan, H., Huang, Z., Deng, H., Yu, Z., Yang, C., Shen, G., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108290 · Cited by 141
Self-citationA Mutual Information Maximization Perspective of Language Representation Learning
Kong, L., d’Autume, C. D. M., Wang, L., Yu, L., Dai, Z., & Yogatama, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08350 · Cited by 101
“Previous work suggests that the next sentence prediction objective is not necessary to train a high quality BERT encoder and the masked language modeling appears to be the key to learn good representations (Liu et al., 2019; Joshi et al., 2019; Lample & Conneau, 2019), so we focus on masked language modeling here.”
BackgroundMethodInfluentialDialogue-Based Relation Extraction
Yu, D., Sun, K., Cardie, C., & Yu, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.444 · Cited by 160
Qin, Y., Lin, Y., Takanobu, R., Liu, Z., Li, P., Ji, H., Huang, M., Sun, M., & Zhou, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.260 · Cited by 92
Chuang, Y.-S., Liu, C.-L., Lee, H.-Y., & Lee, L.-S. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1570 · Cited by 105
“Next sentence prediction (NSP) usually performed in BERT training is not used here because some recent studies indicated it may not be helpful [27, 28, 29, 30].”
MethodAttention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
DeRose, J. F., Wang, J., & Berger, M. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028976 · Cited by 109
“predicting text spans [22], entities [54], and word senses [27].”
BackgroundLabel Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction
Sainz, O., de Lacalle, O. L., Labaka, G., Barrena, A., & Agirre, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.92 · Cited by 179
“Joshi et al. (2020) pretrains a masked language model on random contiguous spans to learn span-boundaries and predict the entire masked span.”
BackgroundMethodResultInfluentialSurvey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 177
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
Wang, D., Ding, N., Li, P., & Zheng, H.-T. (2021) · doi:10.18653/v1/2021.acl-long.181 · Cited by 131
“In the training of CLINE, we follow the setting of RoBERTa (Liu et al., 2019) to omit the next sentence prediction (NSP) objective since previous works have shown that NSP objective can hurt the performance on the downstream tasks (Liu et al., 2019; Joshi et al., 2020).”
Companion Proceedings of the Web Conference 2022
Guo, K., Defretiere, C., Diefenbach, D., Gravier, C., & Gourru, A. (2022) · doi:10.1145/3487553 · Cited by 107
HTR-VT: Handwritten text recognition with vision transformer
Li, Y., Chen, D., Tang, T., & Shen, X. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110967 · Cited by 124
“Our work draws inspiration from BERT [40], SpanBERT [53] and MASS [52], which leverage the prediction of randomly masked words or tokens to learn expressive language representations.”
MethodSelf-citationIt’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations
Tan, S., Joty, S., Kan, M.-Y., & Socher, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.263 · Cited by 84
“Following Joshi et al. (2019), we fine-tune SpanBERTSQuAD 2 for another 4 epochs on our adversarial training set.”
BackgroundMethodWord-Level Coreference Resolution
Dobrovolskii, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.605 · Cited by 161
“We also replicate the span-level model by Joshi et al. (2020) to use it as a baseline for comparison.”
MethodInfluentialSCARF: Self-Supervised Contrastive Learning using Random Feature CorruptionPreprint
Bahri, D., Jiang, H., Tay, Y., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15147 · Cited by 205
“…f . include randomly replacing words and distinguishing between real and fake phrases (Collobert et al., 2011; Mnih & Kavukcuoglu, 2013), random token replacement (Mikolov et al., 2013; Clark et al., 2020), and adjacent sentences (Joshi et al., 2020; Lan et al., 2019; de Vries et al., 2019).”
InfluentialFeng, Z., Zhang, Z., Yu, X., Fang, Y., Li, L., Chen, X., Lu, Y., Liu, J., Yin, W., Feng, S., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00977 · Cited by 192
“For language-based tasks, knowledge-enhanced models used knowledge masking strategy [11, 34] or knowledge-aware pre-training tasks [33, 35] to understand the language data distribution.”
MethodDOBF: A Deobfuscation Pre-Training Objective for Programming LanguagesPreprint
Rozière, B., Lachaux, M.-A., Szafraniec, M., & Lample, G. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2102.07492 · Cited by 179
“Since then, a myriad of studies have proposed to modify the MLM objective, either by masking contiguous spans of text (Song et al., 2019; Joshi et al., 2020), masking named entities”
BackgroundLi, B., Fei, H., Liao, L., Zhao, Y., Teng, C., Chua, T., Ji, D., & Li, F. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612053 · Cited by 158
Exploring Software Naturalness through Neural Language Models
Buratti, L., Pujar, S., Bornea, M. A., McCarley, S., Zheng, Y., Rossiello, G., Morari, A., Laredo, J., Thost, V., Zhuang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12641 · Cited by 51
“conjecture that makingthe pre-trainingtask moredifficult by maskinglongerspans of sourcecodecould result instrongermodelsondownstreamtasks. Indeed,thereisevidencethatsuchtechniquesarebeneficial in NLP. [9, 16] As an alternative, we choose types of strings (for example, text strings that are legal variable or functionnam…”
BackgroundExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 173
“Efforts have also been made to tailor pre-training objectives to specific tasks, e.g., question answering (Ram et al., 2021; Jia et al., 2021), dialogue (Li et al., 2020), and span selection tasks (Joshi et al., 2020).”
BackgroundSelf-Supervised Time Series Representation Learning via Cross Reconstruction Transformer
Zhang, W., Yang, L. Q., Geng, S., & Hong, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3292066 · Cited by 152
“For example, SpanBERT proposed to mask random contiguous spans, rather than random individual tokens [33].”
RevCore: Review-Augmented Conversational Recommendation
Lu, Y., Bao, J., Song, Y., Ma, Z., Cui, S., Wu, Y., & He, X. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.99 · Cited by 73
Trex: Learning Execution Semantics from Micro-Traces for Binary SimilarityPreprint
Pei, K.-X., Xuan, Z., Yang, J.-F., Jana, S., & Ray, B. (2020) · arXiv.org · doi:10.48550/arxiv.2012.08680 · Cited by 141
“For each chosen token to mask, we randomly choose from the masking window size from {1, 3, 5}, which determines how many consecutive neighboring tokens of the chosen tokens are also masked [44].”
MethodKotei, E., & Thirunavukarasu, R. (2023) · Information · doi:10.3390/info14030187 · Cited by 89
“For example, the SpanBERT [64] is a variant of BERT proposed for content-masked prediction without using masked token representations, as performed in [69].”
BackgroundMethodKnowledge Enhanced Masked Language Model for Stance Detection
Kawintiranon, K., & Singh, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.376 · Cited by 139
“SpanBERT (Joshi et al., 2019) masks tokens within a given span range.”
BackgroundMed-BERT: A Pretraining Framework for Medical Records Named Entity Recognition
Liu, N., Hu, Q., Xu, H., Xu, X., & Chen, M. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3131180 · Cited by 122
“Other methods such as Spanbert [17] and Xlnet [18] also improve BERT in terms of changing pretraining tasks and training methods.”
MethodGenerate & Rank: A Multi-task Framework for Math Word Problems
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., & Liu, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.195 · Cited by 88
WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
Huang, J., Tang, D., Zhong, W., Lu, S., Shou, L., Gong, M., Jiang, D., & Duan, N. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.23 · Cited by 159
Toward Green AI: A Methodological Survey of the Scientific Literature
Barbierato, E., & Gatti, A. (2024) · IEEE Access · doi:10.1109/access.2024.3360705 · Cited by 116
“SpanBERT [103] is a language representation model that was pre-trained on large amounts of text data.”
BackgroundMulti-task learning for natural language processing in the 2020s: Where are we going?
Worsham, J., & Kalita, J. K. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.05.031 · Cited by 106
“Since the release of BERT, there have been a number of modifications which have surpassed the base-line score on GLUE (Phang et al., 2018; Joshi et al., 2019; Lan et al., 2019).”
ResultA Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 156
“SpanBERT (Joshi et al., 2020), ERNIE (Sun et al., 2019b), and BERT-WWM (Cui et al., 2019) proposed to mask adjacent tokens.”
MethodHow to Train BERT with an Academic Budget
Izsak, P., Berchansky, M., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.831 · Cited by 76
Self-citationRevealing the Myth of Higher-Order Inference in Coreference Resolution
Xu, L., & Choi, J. D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.686 · Cited by 127
Learning Dense Representations of Phrases at Scale
Lee, J., Sung, M., Kang, J., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.518 · Cited by 68
“In our pilot experiments, we found that Span-BERT (Joshi et al., 2020) leads to superior performance compared to BERT (Devlin et al., 2019).”
ResultSelf-citationNeural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81
“tions by fusing both the left and the right context which are useful in downstream applications, a fact which is in line with many other works that leverage MLM’s (Roberts et al., 2020; Devlin et al., 2018; Joshi et al., 2020; Lan et al., 2019).”
ResultGDPNet: Refining Latent Multi-View Graph for Relation Extraction
Xue, F., Sun, A., Zhang, H., & Chng, E. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17670 · Cited by 106
“Joshi et al. (2020) propose SpanBERT to learn better representations, and achieve SoTA performance on TACRED (Zhang et al. 2017), a sentence-level RE dataset.”
BackgroundMethodResultInfluentialFew-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 98
“The standard approach to question answering is to pretrain a masked language model on raw text, and then fine-tune it with a span selection layer on top (Devlin et al., 2019; Joshi et al., 2020; Liu et al., 2019).”
BackgroundMethodInfluentialSelf-citationLLM-Prop: predicting the properties of crystalline materials using large language models
Rubungo, A. N., Arnold, C. B., Rand, B. P., & Dieng, A. B. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01536-2 · Cited by 144
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Chi, Z., Huang, S., Dong, L., Ma, S., Zheng, B., Singhal, S., Bajaj, P., Song, X., Mao, X.-L., Huang, H., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.427 · Cited by 71
Electric Power Audit Text Classification With Multi-Grained Pre-Trained Language Model
Meng, Q., Song, Y., Mu, J., Lv, Y., Yang, J., Xu, L., Zhao, J., Ma, J., Yao, W., Wang, R., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3240162 · Cited by 83
Document AI: Benchmarks, Models and ApplicationsPreprint
Cui, L., Xu, Y., Lv, T., & Wei, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08609 · Cited by 155
“It can be interpreted as extending the interval mask operation for one-dimensional text in SpanBERT (Joshi et al., 2020) to an interval mask for text blocks in a twodimensional space.”
BackgroundImproving Crisis Events Detection Using DistilBERT with Hunger Games Search Algorithm
Adel, H., Dahou, A., Mabrouk, A., Elaziz, M. A., Kayed, M., Elhenawy, I., AlShathri, S. I., & Ali, A. A. (2022) · Mathematics · doi:10.3390/math10030447 · Cited by 87
“SpanBERT [37] is another BERT for action recognition that could be helpful to put to the librarian of BERT.”
BackgroundFairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
Cheng, P., Hao, W., Yuan, S., Si, S., & Carin, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06413 · Cited by 50
Revisiting Large Language Models as Zero-shot Relation Extractors
Li, G., Wang, P., & Ke, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.459 · Cited by 164
When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain
Shah, R., Chawla, K., Eidnani, D., Shah, A., Du, W., Chava, S., Raman, N., Smiley, C., Chen, J., & Yang, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.148 · Cited by 74
The Effect of Natural Distribution Shift on Question Answering Models
Miller, J., Krauth, K., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2004.14444 · Cited by 74
What do you mean, BERT? Assessing BERT as a Distributional Semantics ModelPreprint
Mickus, T., Paperno, D., Constant, M., & van Deemter, K. (2019) · arXiv (Cornell University) · doi:10.7275/t778-ja71 · Cited by 88
“other works (Lample and Conneau, 2019; Yang et al., 2019; Joshi et al., 2019; Liu et al., 2019) stress that the usefulness and pertinence of the NSP task were not obvious.”
BackgroundSkillSpan: Hard and Soft Skill Extraction from English Job Postings
Zhang, M., Jensen, K. N., Sonniks, S. D., & Plank, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.366 · Cited by 140
Symbolic Music Generation with Transformer-GANs
Muhamed, A., Li, L., Shi, X., Yaddanapudi, S., Chi, W., Jackson, D., Suresh, R., Lipton, Z. C., & Smola, A. J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16117 · Cited by 102
“…• We use the Transformer-XL as our generator and pre-trained BERT as the discriminator; • We pretrain the BERT discriminator in the SpanBERT style (Joshi et al. 2020); • We design an algorithm using the Gumbel-Softmax trick and a variant of the Truncated Backpropagation Through Time (TBPTT)…”
MethodA Primer on Pretrained Multilingual Language Models
Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A., & Kumar, P. (2025) · ACM Computing Surveys · doi:10.1145/3727339 · Cited by 108
Entity Linking Meets Deep Learning: Techniques and Solutions
Shen, W., Li, Y., Liu, Y., Han, J., Wang, J., & Yuan, X. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3117715 · Cited by 64
“SpanBERT [124] extends BERT to better represent and predict text spans.”
BackgroundSelf-citationLearning Rich Representation of Keyphrases from Text
Kulkarni, M., Mahata, D., Arora, R., & Bhowmik, R. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.67 · Cited by 55
Transformer: A General Framework from Machine Translation to Others
Zhao, Y., Zhang, J., & Zong, C. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1393-5 · Cited by 93
“Furthermore, many BERT-like mod- els have been proposed on the basis of BERT, such as RoBERTa[136] and SpanBERT[137].”
MethodGreco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 104
KLEJ: Comprehensive Benchmark for Polish Language Understanding
Rybak, P., Mroczkowski, R., Tracz, J., & Gawlik, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.111 · Cited by 110
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Span Selection Pre-training for Question Answering
Glass, M., Gliozzo, A., Chakravarti, R., Ferritto, A., Pan, L., Bhargav, G. P. S., Garg, D., & Sil, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.247 · Cited by 62
Event detection in online social network: Methodologies, state-of-art, and evolution
Hu, X., Ma, W., Chen, C., Wen, S., Zhang, J., Xiang, Y., & Fei, G. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100500 · Cited by 73
InterBERT: Vision-and-Language Interaction for Multi-modal PretrainingPreprint
Lin, J., An, Y., Zhang, Y., Liu, J., Zhou, J., & Yang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13198 · Cited by 102
“Inspired by [18], we randomly mask continuous text spans by 10% with an average length of 3 words.”
BackgroundMethodMengzi: Towards Lightweight yet Ingenious Pre-trained Models for ChinesePreprint
Zhang, Z., Zhang, H., Chen, K., Guo, Y., Hua, J., Wang, Y., & Zhou, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06696 · Cited by 124
“Recent studies have investigated denoising strategies (Raffel et al., 2020; Lewis et al., 2020), model architecture (Yang et al., 2019), and auxiliary objectives (Lan et al., 2020; Joshi et al., 2020) to enhance the model capacity in pre-training.”
Background
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.