SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 601–700 of 1,012
DeB3RTa: A Transformer-Based Model for the Portuguese Financial Domain
Pires, H. F., Paucar, V. L., & Carvalho, J. P. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030051 · Cited by 16
Distilling Wisdom: A Review on Optimizing Learning From Massive Language Models
Zhang, D., Listiyani, D., Singh, P., & Mohanty, M. (2025) · IEEE Access · doi:10.1109/access.2025.3554586 · Cited by 20
“SpanBERT [65] employs span-based pre-training, using span boundary objectives to enhance NLP task performance and entity recognition.”
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
Kang, M., Han, M., & Hwang, S. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.493 · Cited by 20
A Graph Representation of Semi-structured Data for Web Question Answering
Zhang, X., Shou, L., Pei, J., Gong, M., Wen, L., & Jiang, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.5 · Cited by 19
WLV-RIT at SemEval-2021 Task 5: A Neural Transformer Framework for Detecting Toxic Spans
Ranasinghe, T., Sarkar, D., Zampieri, M., & Ororbia, A. G. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.111 · Cited by 18
A New Approach to Overgenerating and Scoring Abstractive Summaries
Song, K., Wang, B., Feng, Z., & Liu, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.110 · Cited by 16
OKGIT: Open Knowledge Graph Link Prediction with Implicit Types
Chandrahas, & Talukdar, P. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.225 · Cited by 16
End-to-End AMR Corefencence Resolution
Fu, Q., Song, L., Du, W., & Zhang, Y. (2021) · doi:10.18653/v1/2021.acl-long.324 · Cited by 25
“• Pipeline (Anikina et al., 2020): it uses an off-the-shelf coreference system (Lee et al., 2018) with SpanBERT (Joshi et al., 2020) embeddings, and an AMR-to-text aligner (Flanigan et al., 2014).”
MethodDIRECT : A Transformer-based Model for Decompiled Identifier Renaming
Nitin, V., Saieva, A., Ray, B., & Kaiser, G. E. (2021) · doi:10.18653/v1/2021.nlp4prog-1.6 · Cited by 17
“SpanBERT (Joshi et al., 2020) focuses on contiguous spans of masked tokens with a modi-fied pre-training objective.”
BackgroundFrame-based Multi-level Semantics Representation for text matching
Guo, S., Guan, Y., Li, R., Li, X., & Tan, H. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107454 · Cited by 14
Zeng, C., & Li, S. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/5375334 · Cited by 15
“To a certain extent, the span masking, entity-level masking, and phrase-level masking can be regarded as special cases of the N -gram masking scheme [5, 9].”
BackgroundMethodResultInfluentialRetractedFine-Grained Entity Typing with a Type Taxonomy: a Systematic Review
Wang, R., Hou, F., Cahan, S. F., Chen, L., Jia, X., & Ji, W. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3148980 · Cited by 15
“Notably, ELMo [138], [127], BERT [140], [141], [142], [143], [144], SpanBERT [145], [146], RoBERTa [147], [142], XLNet [148], [142] have been employed to encode contexts for FGET.”
MethodResearch on Question Answering System Based on BERT Model
Yin, J. (2022) · 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA) · doi:10.1109/cvidliccea56201.2022.9824408 · Cited by 15
MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
Xie, P., Li, Z., Zhao, Z., Liu, J., & Hu, X. (2022) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3221043 · Cited by 15
Pre-training Language Models with Deterministic Factual Knowledge
Li, S., Li, X., Shang, L., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.764 · Cited by 20
Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twitter
Sundriyal, M., Kulkarni, A., Pulastya, V., Akhtar, M. S., & Chakraborty, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.525 · Cited by 17
Unifying Structure Reasoning and Language Pre-Training for Complex Reasoning Tasks
Wang, S., Wei, Z., Xu, J., Li, T., & Fan, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3325973 · Cited by 19
“Although some attempts can be viewed as structure modeling in text, they only consider reasoning over simple triplets [46, 47, 48, 49].”
BackgroundKPQA: A Metric for Generative Question Answering Using Keyphrase Weights
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.170 · Cited by 25
“For these cases, we use SpanBERT (Joshi et al., 2020) to resolve the coreferences in the para-graphs and extract all of the supporting sentences to compose answer sentences.”
MethodDefining and Learning Refined Temporal Relations in the Clinical Narrative
Wright-Bettner, K., Lin, C., Miller, T. A., Bethard, S. J., Dligach, D., Palmer, M. S., Martin, J., & Savova, G. (2020) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2020.louhi-1.12 · Cited by 14
“…(Lee et al., 2020), Xlnet (Yang et al., 2019), ALBERT (Lan et al., 2020), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2019), and SpanBERT (Joshi et al., 2020) were used to encode each input sequence with a relational candidate by the [CLS] token, which was fed to the classification layer…”
MethodFanfictionNLP: A Text Processing Pipeline for Fanfiction
Yoder, M. M., Khosla, S., Shen, Q., Naik, A., Jin, H., Muralidharan, H., & Rosé, C. P. (2021) · NUSE · doi:10.18653/v1/2021.nuse-1.2 · Cited by 24
VD-PCR: Improving visual dialog with pronoun coreference resolution
Yu, X., Zhang, H., Hong, R., Song, Y., & Zhang, C. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108540 · Cited by 21
“For the PCR task on VisPro, we use the end-to-end coreference resolution model based on ELMo [19], BERT [13], and SpanBERT [14] as baselines 2 .”
MethodNeural Coreference Resolution for ArabicPreprint
Aloraini, A., Yu, J., & Poesio, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00286 · Cited by 14
“Later, Joshi et al. (2019a) introduces S PAN BERT which is trained for tasks that involve spans.”
BackgroundMethodResultInfluentialLIME: Learning Inductive Bias for Primitives of Mathematical ReasoningPreprint
Wu, Y., Rabe, M. N., Li, W., Ba, J. L., Grosse, R., & Szegedy, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06223 · Cited by 52
“Where BERT (Devlin et al., 2019) masks out a fraction of the input tokens, later works demonstrated the advantages of masking out subsequences (Song et al., 2019; Dong et al., 2019; Joshi et al., 2020; Raffel et al., 2020; Conneau & Lample, 2019) and whole sentences (Zhang et al., 2020).”
BackgroundCLINE: Contrastive Learning with Semantic Negative Examples for Natural Language UnderstandingPreprint
Wang, D., Ding, N., Li, P., & Zheng, H.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00440 · Cited by 32
“In the training of CLINE, we follow the setting of RoBERTa (Liu et al., 2019) to omit the next sentence prediction (NSP) objective since previous works have shown that NSP objective can hurt the performance on the downstream tasks (Liu et al., 2019; Joshi et al., 2020).”
Pack Together: Entity and Relation Extraction with Levitated Marker.Preprint
Ye, D., Lin, Y., & Sun, M. (2021) · arXiv (Cornell University) · Cited by 14
SkillSpan: Hard and Soft Skill Extraction from English Job PostingsPreprint
Zhang, M., Jensen, K. N., Sonniks, S. D., & Plank, B. (2022) · IT University Of Copenhagen (IT University of Copenhagen) · doi:10.48550/arxiv.2204.12811 · Cited by 30
“1 2 We present strong baselines for the task including a new SpanBERT (Joshi et al., 2020) trained from scratch, and domain-adapted variants (Gururangan et al.”
BackgroundMethodInfluentialInvestigating Educational and Noneducational Answer Selection for Educational Question Generation
Steuer, T., Filighera, A., & Tregel, T. (2022) · IEEE Access · doi:10.1109/access.2022.3180838 · Cited by 13
“Its parameter size is comparable to BERT, but it is explicitly designed toward text span prediction [18].”
MethodLECM: A model leveraging emotion cause to improve real-time emotion recognition in conversations
Lu, W., Hu, Z., Lin, J., & Wang, L. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112900 · Cited by 14
Neural semi-Markov CRF for Monolingual Word Alignment
Lan, W., Jiang, C., & Xu, W. H. (2021) · doi:10.18653/v1/2021.acl-long.531 · Cited by 13
“First, source and target sentences are concatenated together and encoded by the pre-trained SpanBERT (Joshi et al., 2020) model.”
MethodInfluentialPQSCT: Pseudo-Siamese BERT for Concept Tagging With Both Questions and Solutions
Huang, T., Hu, S., Yang, H., Geng, J., Liu, S., Zhang, H., & Yang, Z. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3275707 · Cited by 20
MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation
Wang, G., Li, Y., Liu, Y., Deng, G., Li, T., Xu, G., Liu, Y., Wang, H., & Wang, K. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652297 · Cited by 24
Self-citationGeneral Point Model Pretraining with Autoencoding and Autoregressive
Li, Z., Gao, Z., Tan, C., Ren, B., Yang, L. T., & Li, S. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01980 · Cited by 22
“In recent years, the natural language processing (NLP) [4, 19, 20, 29, 40] and computer vision (CV) [3, 7, 9, 49, 57, 66] realms have witnessed a proliferation of transformer-based pretrained models.”
BackgroundZhang, Y., Liu, J., Zhong, X., & Wu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126651 · Cited by 16
SimiT: A Text Similarity Method Using Lexicon and Dependency Representations
İnan, E. (2020) · New Generation Computing · doi:10.1007/s00354-020-00099-8 · Cited by 13
“SpanBERT [26] extends BERT for better representation and prediction of text spans using an extremely large number of parameters.”
BackgroundConundrums in Entity Coreference Resolution: Making Sense of the State of the Art
Lu, J. J., & Ng, V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.536 · Cited by 24
“…4https://github.com/kentonl/e2e-coref 5https://github.com/mandarjoshi90/coref 6This is the independent version in Joshi et al. (2019). et al. (2019) employ embeddings pretrained using a new method called SpanBERT (Joshi et al., 2020), which is designed to better represent text spans than BERT.”
MethodScaling Within Document Coreference to Long Texts
Thirukovalluru, R., Monath, N., Shridhar, K., Zaheer, M., Sachan, M., & McCallum, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.343 · Cited by 27
“The computation bottle-neck mostly depends on the high dimensionality of the encoded token representations, which are from transformer-based language models (Joshi et al., 2020).”
BackgroundMethodInfluentialEnd-To-End Neural Coreference Resolution Revisited: A Simple Yet Effective Baseline
Lai, T., Bui, T., & Kim, D. S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746254 · Cited by 23
Modular Self-Supervision for Document-Level Relation Extraction
Zhang, S., Wong, C., Usuyama, N., Jain, S., Naumann, T., & Poon, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.429 · Cited by 14
Automated Paraphrase Quality Assessment Using Language Models and Transfer Learning
Nicula, B., Dascalu, M., Newton, N., Orcutt, E., & McNamara, D. (2021) · De Computis · doi:10.3390/computers10120166 · Cited by 12
“These models can be introduced alongside innovative masking techniques [17], or leverage smaller representations such as TinyBERT [18].”
BackgroundMention detection in coreference resolution: survey
Lata, K., Singh, P., & Dutta, K. (2022) · Applied Intelligence · doi:10.1007/s10489-021-02878-2 · Cited by 20
“The authors obtained the input representation with the SpanBERT model [163].”
MethodModeling Multi-Granularity Hierarchical Features for Relation Extraction
Liang, X., Wu, S., Li, M., & Li, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.375 · Cited by 15
Micallef, K., Gatt, A., Tanti, M., & van der Plas, L. (2022) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/2022.deeplo-1.10 · Cited by 47
“In both cases, pre-training is performed using the Masked Language Modelling Objective (MLM) only, since the Next Sentence Prediction (NSP) objective was found to be detrimental to downstream performance (Joshi et al., 2020; Liu et al., 2019).”
MethodMinutolo, A., Guarasci, R., Damiano, E., Pietro, G. D., Fujita, H., & Esposito, M. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07641-3 · Cited by 15
Certified Robustness to Word Substitution Ranking Attack for Neural Ranking Models
Wu, C. H., Zhang, R., Guo, J., Chen, W., Fan, Y., de Rijke, M., & Cheng, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557256 · Cited by 25
“Neural ranking models (NRMs) [6, 30, 36], especially pre-trained ranking models [22, 27, 34], have led to substantial performance improvements in a wide range of search tasks [14, 20, 27].”
BackgroundA Survey of Extractive Question Answering
Wang, L., Zheng, K., Qian, L., & Li, S. (2022) · International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS) · doi:10.1109/hdis56859.2022.9991478 · Cited by 13
“In the following study, numerous methods add Pre-trained Language Models(PLMs), such as BERT [8], RoBERTa [9], and SpanBERT [10] to the pointer header to get the scope of the answer span, the wealth of information contained in PLMs can be effective assistance for reasoning.”
MethodDeCoDE: Detection of Cognitive Distortion and Emotion Cause Extraction in Clinical Conversations
Singh, G. V., Ghosh, S. K., Ekbal, A., & Bhattacharyya, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_11 · Cited by 13
“Low performances by RoBERTa [23] and SpanBERT [19] shows the difficulty of powerful language models in perceiving critical tasks as emotion cause extraction, more so, especially in clinical situations where training data may be insufficient.”
BackgroundMethodInfluentialAhmed, M., Khan, H. U., Iqbal, T., Alarfaj, F. K., Alomair, A., & Almusallam, N. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1422 · Cited by 13
InforMask: Unsupervised Informative Masking for Language Model Pretraining
Sadeq, N., Xu, C., & McAuley, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.395 · Cited by 16
“…results on GLUE (Wang et al., 2018) benchmark in Appendix C. Baselines We compare InforMask in two settings: (a) We use the same tokenizer and hyperparameters to pretrain BERT random masking (Devlin et al., 2019), SpanBERT (Joshi et al., 2020) and PMI-Masking (Levine et al., 2021) for 3 epochs.”
BackgroundMethodInfluentialZheng, A., Zhao, H., Luo, Z., Feng, C., Liu, X., & Ye, Y. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102638 · Cited by 16
“Importance of pre-training selection 15 In the token representation component, we replace SciBERT with SpanBERT (Joshi et al., 2020) or BERT (Devlin et al.”
Improving Adverse Drug Event Extraction with SpanBERT on Different Text Typologies
Portelli, B., Passabì, D., Lenzi, E., Serra, G., Santus, E., & Chersoni, E. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-93080-6_8 · Cited by 13
Out of Context: A New Clue for Context Modeling of Aspect-based Sentiment Analysis
Xing, B., & Tsang, I. W.-H. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13410 · Cited by 15
Modeling Task Interactions in Document-Level Joint Entity and Relation Extraction
Xu, L., & Choi, J. D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.395 · Cited by 14
Self-Supervised Learning With Segmental Masking for Speech Representation
Yue, X., Lin, J., Ritter-Gutierrez, F., & Li, H. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3191845 · Cited by 13
“Furthermore, motivated by spanBERT [28], we would study the effect of the segmental masking unit size.”
MethodImproving Machine Reading Comprehension via Adversarial TrainingPreprint
Yang, Z., Cui, Y., Che, W., Liu, T., Wang, S., & Hu, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03614 · Cited by 15
MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text TranslationPreprint
Chen, J., Ma, M., Zheng, R., & Huang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11445 · Cited by 21
“Span Masking Similar with SpanBERT (Joshi et al., 2020), we first sample a serial of span widths and then apply those spans randomly to different positions of the input signal.”
KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative LearningPreprint
He, B., Jiang, X., Xiao, J., & Liu, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03551 · Cited by 17
“…LIMIT-BERT (Zhou, Zhang, and Zhao 2019) semantic/syntactic phrase masking - KEPLER (Wang et al. 2019) - knowledge representation learning SpanBERT (Joshi et al. 2020a) span masking - WKLM (Xiong et al. 2020) - entity replacement checking K-Adapter (Wang et al. 2020) - relation classification,…”
BackgroundMethodResultInfluentialStudying Strategically: Learning to Mask for Closed-book QAPreprint
Ye, Q., Li, B. Z., Wang, S., Bolte, B., Ma, H., Yih, W.-T., Ren, X., & Khabsa, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15856 · Cited by 17
“Current self-supervised objectives are mostly heuristic , including masked language modeling (Devlin et al., 2019), span boundary representation learning (Joshi et al., 2020), corrupted sentence reconstruction (Lewis et al., 2020), gap sentence generation (Zhang et al., 2020), etc.”
BackgroundLess is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak DecoderPreprint
Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T., & Overwijk, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09206 · Cited by 21
“Motivated from MLM, many token-level tasks have been proposed, such as span masking (Joshi et al., 2020), whole-word masking (Liu et al., 2019) and replacement detection (Clark et al., 2020).”
A Review on BERT: Language Understanding for Different Types of NLP TaskPreprint
Islam, M. S., & Zhang, L. (2024) · Preprints.org · doi:10.20944/preprints202401.1857.v1 · Cited by 15
CREAD: Combined Resolution of Ellipses and Anaphora in Dialogues
Tseng, B.-H., Bhargava, S., Lu, J., Moniz, J. R. A., Piraviperumal, D., Li, L., & Yu, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.265 · Cited by 27
Multilingual Coreference Resolution with Harmonized Annotations
Pražák, O., Konopík, M., & Sido, J. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_125 · Cited by 35
Can Pre-trained Language Models Interpret Similes as Smart as Human?
He, Q., Cheng, S., Li, Z., Xie, R., & Xiao, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.543 · Cited by 24
TAPHSIR: towards AnaPHoric ambiguity detection and ReSolution in requirements
Ezzini, S., Abualhaija, S., Arora, C., & Sabetzadeh, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558928 · Cited by 16
“Based on the empirical findings in our earlier work [3], we know that for the task of anaphora resolution, the SpanBERT language model [8] outperforms alternatives.”
BackgroundSchema matching based on energy domain pre-trained language model
Pan, Z., Yang, M., & Monti, A. (2023) · Energy Informatics · doi:10.1186/s42162-023-00277-0 · Cited by 13
“The same conclusion is also mentioned in the article of Spanbert (Joshi et al. 2020).”
DLRGeoTweet: A comprehensive social media geocoding corpus featuring fine-grained places
Hu, X., Elßner, T., Zheng, S., Serere, H. N., Kersten, J., Klan, F., & Qiu, Q. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103742 · Cited by 16
Training Experimental Language Models with Low Resources, for the Hungarian Language
Yang, Z. G., & Váradi, T. (2023) · Acta Polytechnica Hungarica · doi:10.12700/aph.20.5.2023.5.11 · Cited by 18
Research on Multifeature Fusion False Review Detection Based on DynDistilBERT-BiLSTM-CNN
Zhang, J., Huang, K., Lang, D., Xu, Y., Li, H.-A., & Li, X. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3410015 · Cited by 12
“SpanBERT enhances BERT’s pretraining mechanism by predicting text spans within the context, thus capturing long-distance dependencies more effectively [23].”
BackgroundDetecting Personally Identifiable Information Through Natural Language Processing: A Step Forward
Mainetti, L., & Elia, A. (2025) · Applied System Innovation · doi:10.3390/asi8020055 · Cited by 17
“Driven by the increasing demands of NLP, several pre-trained transformer models have been proposed, including BERT (Bidirectional Encoder Representations From Trans-formers), RoBERTa, DeBERTa, GPT, and ELECTRA, which are thoroughly described by Joshi et al. in [32] and by Clark et al. in [33].”
Video Question Answering with Phrases via Semantic Roles
Sadhu, A., Chen, K., & Nevatia, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.196 · Cited by 19
KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
Lee, H., Yoon, J., Hwang, B., Joe, S., Min, S.-J., & Gwon, Y. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412023 · Cited by 14
“RoBERTa [12], ALBERT [11] and other variants [13],”
BackgroundVerification mechanism to obtain an elaborate answer span in machine reading comprehension
Yu, P., Li, X., Song, J., Luo, Y., Hu, S., & Qian, W. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.084 · Cited by 12
Attend, Memorize and Generate: Towards Faithful Table-to-Text Generation in Few Shots
Zhao, W., Ye, L., Wan, Y., & Yu, P. S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.347 · Cited by 12
“The widely used pre-trained models (PTMs) for table-to-text generation can be categorized into two classes: text-to-text PTMs (Radford et al., 2018; Devlin et al., 2019; Dong et al., 2019; Lewis et al., 2020; Joshi et al., 2020) and structured data-to-text PTMs (Chen et al., 2020b; Herzig et al., 2020; Xing and Wan, 2…”
BackgroundelBERto: Self-supervised commonsense learning for question answering
Zhan, X., Li, Y., Dong, X., Liang, X., Hu, Z., & Carin, L. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109964 · Cited by 12
Injecting User Identity Into Pretrained Language Models for Document-Level Sentiment Classification
Cao, X., Yu, J., & Zhuang, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3158975 · Cited by 15
EntSUM: A Data Set for Entity-Centric Extractive Summarization
Maddela, M., Kulkarni, M., & Preoţiuc-Pietro, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.237 · Cited by 36
From spoken dialogue to formal summary: An utterance rewriting for dialogue summarization
Fang, Y., Zhang, H., Chen, H., Ding, Z., Long, B., Lan, Y., & Zhou, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.283 · Cited by 25
“To obtain the edit matrix, we conduct three neural networks, as shown in For co-reference data augmentation, we replace the co-reference word with its specific name entity through a co-reference resolution model (Joshi et al., 2020) and then augment the dataset with these replacement data.”
MethodSliding space-disparity transformer for stereo matching
Rao, Z., He, M., Dai, Y., & Shen, Z. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07621-7 · Cited by 13
“, text translation [5, 10, 39], question answering [9, 21, 32], and reading comprehension [1, 52].”
BackgroundA Brief Survey of Text Classification Methods
Jiao, Q. (2023) · IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) · doi:10.1109/iciba56860.2023.10165621 · Cited by 14
“Inrecentprogresses, MLM is also extended to different definitions for better language understanding, such as Seq2Seq-MLM [48], Enhanced-MLM [13], [32], and Random Contiguous Words Masking [21].”
MethodPTCSpell: Pre-trained Corrector Based on Character Shape and Pinyin for Chinese Spelling Correction
Wei, X., Huang, J., Yu, H., & Liu, Q. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.394 · Cited by 24
“Large pre-trained language models (PTMs) based on Transformer (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), XLNET (Yang et al., 2019) and SpanBERT (Joshi et al., 2020) have been proposed and are being increasingly used for CSC tasks.”
On the Role of Bidirectionality in Language Model Pre-Training
Artetxe, M., Du, J., Goyal, N., Zettlemoyer, L., & Stoyanov, V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.293 · Cited by 24
Self-citationICDAR 2023 Competition on Document UnderstanDing of Everything (DUDE)
Landeghem, J. V., Tito, R., Borchmann, Ł., Pietruszka, M., Jurkiewicz, D., Powalski, R., Józiak, P., Biswas, S., Coustaty, M., & Stanisławek, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41679-8_24 · Cited by 21
Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction
Wang, H., Zhang, H., Deng, Y., Gardner, J. R., Roth, D., & Chen, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.39 · Cited by 29
Sent2Span: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations
Liu, S., Sun, Y., Li, B., Wang, W., Bourgeois, F. T., & Dunn, A. G. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.147 · Cited by 12
Enhance Text-to-Text Transfer Transformer with Generated Questions for Thai Question Answering
Phakmongkol, P., & Vateekul, P. (2021) · Applied Sciences · doi:10.3390/app112110267 · Cited by 14
“The last step, Question filtration, is performed by a BERT model trained with question answering objectives with human-labeled data.”
MethodA Distributed Data Fabric Architecture based on Metadate Knowledge Graph
Li, X., Yang, D., Xia, X., Zhang, K., & Liu, K. (2022) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit55514.2022.9943831 · Cited by 14
“The knowledge graph layer is divided into entity recognition, relationship extraction, entity disambiguation [12], anaphora resolution [13], weight analysis, ontology reasoning, rule reasoning and other modules.”
MethodOpen-Vocabulary Argument Role Prediction For Event Extraction
Jiao, Y., Li, S., Xie, Y., Zhong, M., Ji, H., & Han, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.395 · Cited by 16
Entity and Evidence Guided Relation Extraction for DocREDPreprint
Huang, K., Wang, G., Ma, T., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12283 · Cited by 14
“These datasets have facilitated much research progress in this area such as (Wang et al., 2016; Alt et al., 2019; Zhao et al., 2019) on SemEval2010 Task 8 and (Zhang et al., 2017; Guo et al., 2019; Baldini Soares et al., 2019; Joshi et al., 2019) on TACRED.”
BackgroundMethodHarperValleyBank: A Domain-Specific Spoken Dialog CorpusPreprint
Wu, M., Nafziger, J., Scodary, A., & Maas, A. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13929 · Cited by 18
“Ou baselines use recent ideas from contrastive learning [27, 28, 29, 30, 31, 32, 33, 34] where representations are learned by discriminating between specific instances in a dataset.”
BackgroundBhattacharjee, A., Hasan, T., Samin, K., Islam, S., Rahman, M. S., Iqbal, A., & Shahriyar, R. (2021) · arXiv (Cornell University) · Cited by 12
Enabling Language Representation with Knowledge Graph and Structured Semantic Information
Xu, W., Fang, M., Li, Y., Jiang, H., Geng, L., & Zuo, C. (2021) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai50917.2021.9447453 · Cited by 11
“Since then, researchers have made many efforts for further optimization, mainly focusing on the encoder and pre-training process [10] [11] [12] [13].”
BackgroundWhat does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text
Fang, B., Baldwin, T., & Verspoor, K. M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.275 · Cited by 27
“Directions for future work include: (1) joint learning with COREFERENCE and TRANSFORMED relations, which differ only in whether there is a state change or not, such that considering them together may be effective; (2) incorporation of external knowledge, including knowledge about ingredient entities, which may further…”
BackgroundRoBERTa-CoA: RoBERTa-Based Effective Finetuning Method Using Co-Attention
Kim, J.-H., Park, S.-W., Kim, J.-Y., Park, J., Jung, S.-H., & Sim, C.-B. (2023) · IEEE Access · doi:10.1109/access.2023.3328352 · Cited by 12
“In 2019, bidirectional encoder representations from trans-formers (BERT) [4], [5], [6], [7] emerged as a breakthrough model in the MRC field, optimized for pretraining and fine-tuning.”
MethodHu, S., Zhou, H., Hergul, M., Gritta, M., Zhang, G., Iacobacci, I., Vulić, I., & Korhonen, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00609 · Cited by 14
“…the cards; 2) annotating character-level span indices for each slot value v ARA ; 3) indicating with a binary flag for each domain-intent pair d - i whether this dialog act retains coherence of the full dialog, this way also signaling and capturing errors still present in the English MultiWOZ v2.”
BackgroundNeighBERT: Medical Entity Linking Using Relation-Induced Dense Retrieval
Singh, A., Krishnamoorthy, S., & Ortega, J. E. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00136-3 · Cited by 15
Bakker, R. M., Schoevers, A. J., van Drie, R. A. N., Schraagen, M., & de Boer, M. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09437-x · Cited by 20
Sequence-based generative AI design of versatile tryptophan synthases
Lambert, T., Tavakoli, A., Dharuman, G., Yang, J., Bhethanabotla, V. C., Kaur, S., Hill, M., Ramanathan, A., Anandkumar, A., & Arnold, F. H. (2026) · Nature Communications · doi:10.1038/s41467-026-68384-6 · Cited by 12
“Detectable Activity is de fi ned as when the product is measurable by mass spectrometry above background level (three standard deviations above pUC19-containing E. coli ) but below UV detection on the 277nm HPLC UV channel (see Methods for details).”
Pre-training Mention Representations in Coreference Models
Varkel, Y., & Globerson, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.687 · Cited by 15
“SpanBERTCoref is Joshi et al. (2019b) using SpanBERT, trained on GAP with the filtered loss defined in Sec.”
BackgroundMethodInfluentialContext Analysis for Pre-trained Masked Language Models
Lai, Y.-A., Lalwani, G., & Zhang, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.338 · Cited by 14
“Masked language modeling (MLM) introduced in BERT (Devlin et al., 2019) has been widely used as the pre-training task in works including RoBERTa (Liu et al., 2019b), SpanBERT (Joshi et al., 2020), and ALBERT (Lan et al., 2019).”
BackgroundMethodBridging Resolution: A Survey of the State of the Art
Kobayashi, H., & Ng, V. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.331 · Cited by 29
“”, (2) create candidate answers from the candidate antecedents (i.e., the preceding mentions that appear in a fixed sentence window), and (3) use a BERT-based QA system pretrained on the SQuAD corpus (Joshi et al., 2020) to choose the most probable answer (i.e., the antecedent).”
BackgroundMethodInfluentialDetecting Scenes in Fiction: A new Segmentation Task
Zehe, A., Konle, L., Dümpelmann, L. K., Gius, E., Hotho, A., Jannidis, F., Kaufmann, L., Krug, M., Puppe, F., Reiter, N., et al. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.276 · Cited by 38
“As an example, the memory requirements of state of the art coreference resolution models scale with O ( n 4 ) (input length n ) (Lee et al., 2017; Joshi et al., 2020) and their performance deteriorates on longer texts.”
Background
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.