SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 801–900 of 1,012
Predicting Coreference in Abstract Meaning Representations
Anikina, T., Koller, A., & Roth, M. (2020) · Publications of the UdS (Saarland University) · doi:10.22028/d291-36109 · Cited by 15
WebRED: Effective Pretraining And Finetuning For Relation Extraction On The WebPreprint
Ormándi, R., Saleh, M., Winter, E., & Rao, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09681 · Cited by 12
“Recently, pre-training (Bengio et al., 2007; Erhan et al., 2010) has been used effectively to train higher capacity neural networks for language modeling (Devlin et al., 2018; Radford et al., 2018) and relation extraction (Joshi et al., 2019; Shi and Lin, 2019).”
BackgroundImproving Sentence-Level Relation Extraction through Curriculum LearningPreprint
Park, S., & Kim, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09332 · Cited by 12
iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document ExplorationPreprint
Hirsch, E., Eirew, A., Shapira, O., Caciularu, A., Cattan, A., Ernst, O., Pasunuru, R., Ronen, H., Bansal, M., & Dagan, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11621 · Cited by 9
“(1) We use SpanBERT 6 (Joshi et al., 2020), a state-of-the-art transformer-based LM for WD entity coreference resolution, to detect and cluster coreferring entity mentions within each separate document.”
BackgroundMethodPoNet: Pooling Network for Efficient Token Mixing in Long SequencesPreprint
Tan, C.-H., Chen, Q., Wang, W., Zhang, Q., Zheng, S., & Ling, Z.-H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02442 · Cited by 10
“SpanBERT randomly samples contiguous spans inside of individual tokens and augments MLM with a new task to predict the entire masked span.”
BackgroundLanguage models for document understanding
Douzon, T. (2023) · theses.fr (ABES) · doi:10.70675/66ce60f8zcccaz476eza4d6zf801a2256b86 · Cited by 23
Improving Representation With Hierarchical Contrastive Learning for Emotion-Cause Pair Extraction
Hu, G., Zhao, Y., & Lu, G. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3391854 · Cited by 14
“(cid:2) SpanBERT [67] is a pre-trained model following a training objective, i.e., predicting masked contiguous spans instead of tokens.”
Tian, Z., Teng, Y., Ke, X., Chen, Y., & Chen, L. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107766 · Cited by 9
Liu, Q., & Ma, W. (2025) · Future Internet · doi:10.3390/fi17060241 · Cited by 15
Scaling SMILES-Based Chemical Language Models for Therapeutic Peptide Engineering
Feller, A. L., Secor, M., Swanson, S., Wilke, C. O., & Deibler, K. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c00652 · Cited by 12
Overview of BioASQ 8a and 8b: Results of the Eighth Edition of the BioASQ Tasks a and b.
Nentidis, A., Krithara, A., Bougiatiotis, K., & Παλιούρας, Γ. (2020) · CLEF (Working Notes) · Cited by 8
Fast End-to-end Coreference Resolution for Korean
Park, C., Shin, J., Park, S., Lim, J., & Lee, C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.237 · Cited by 9
“The lowest performance among the 10-models is 70.04% F1, and the mean F1 score is 70.37%, both of which still outperforms the 68.62 F1 of the Korean BERT-coref from Joshi et al. (2019b).”
BackgroundMethodResultInfluentialA Novel Named Entity Recognition Scheme for Steel E-Commerce Platforms Using a Lite BERT
Chen, M., Luo, X., Shen, H., Huang, Z., & Peng, Q. (2021) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2021.017491 · Cited by 8
Token2vec: A Joint Self-Supervised Pre-Training Framework Using Unpaired Speech and Text
Yue, X., Ao, J., Gao, X., & Li, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096923 · Cited by 8
“In NLP, GPT [7], RoBERTa [8] and other variants [9, 10] have enabled many language processing tasks to achieve the state-of-the-art performance (e.g., GLUE [11], SuperGLUE [12], and XTREME [13]).”
BackgroundPLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English
Chi, J., Ahmad, W. U., Tian, Y., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.31 · Cited by 11
Gopalakrishnan, S., Chen, V. Z., Dou, W., & Zadrozny, W. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100055 · Cited by 12
InA: Inhibition Adaption on pre-trained language models
Kang, C., Prokop, J., Tong, L., Zhou, H., Hu, Y. S., & Novák, D. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106410 · Cited by 10
RTRL: Relation-aware Transformer with Reinforcement Learning for Deep Question Generation
Zeng, H., Wei, B., & Liu, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112120 · Cited by 11
TCohPrompt: task-coherent prompt-oriented fine-tuning for relation extraction
Long, J., Yin, Z., Liu, C., & Huang, W. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01563-4 · Cited by 8
“SpanBERT [13], an extension of BERT, introduces a random span masking approach during pre-training that allows the model to better represent and predict spans of text.”
A Review on Transformer Models: Applications, Taxonomies, Open Issues and Challenges
Passi, N., Raj, M., & Shelke, N. A. (2024) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon62057.2024.10838047 · Cited by 14
“…ALBERT (A Lite BERT) [5],DistilBERT (Distilled BERT) [6], MobileBERT [7], TinyBERT [8], DeBERTa (Decoding-enhanced BERT with disentangled attention)[9], SpanBERT (Span-based Bidirectional Encoder Representations for Transformers) [10], CamemBERT (Camembert: a Tasty French Language Model) [11].”
Somun: entity-centric summarization incorporating pre-trained language models
İnan, E. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05319-2 · Cited by 8
“SpanBERT [17] extends BERT for better representation and prediction of text spans using an extremely large number of parameters.”
BackgroundMethodPretraining Financial Text Encoder Enhanced by Lifelong Learning
Liu, Z., Huang, D., & Huang, K. (2020) · IEEE Access · doi:10.1109/access.2020.3027622 · Cited by 8
Enhancing Online Knowledge Graph Population with Semantic Knowledge
Fernàndez-Cañellas, D., Rimmek, J. M., Espadaler, J., Garolera, B., Barja, A., Codina, M., Sastre, M., Giró-i-Nieto, X., Riveiro, J. C., & Bou‐Balust, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62419-4_11 · Cited by 11
Knowledge-Enabled Diagnosis Assistant Based on Obstetric EMRs and Knowledge Graph
Zhang, K., Zhao, X., Zhuang, L., Xie, Q., & Zan, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63031-7_32 · Cited by 8
A BERT-Based Two-Stage Model for Chinese Chengyu Recommendation
Tan, M., Jiang, J., & Dai, B. T. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3453185 · Cited by 9
“With further ablation study, NSP is either removed [Joshi et al. 2020; Yang et al. 2019] due to inconsistent improvement or restricted to use sentences from a single document [Liu et al.”
MethodInfluentialCan Transformer Language Models Predict Psychometric Properties?
Laverghetta, A., Nighojkar, A., Mirzakhalov, J., & Licato, J. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.2 · Cited by 15
“Besides the GLUE diagnostic, other taxonomies have been proposed, such as TaxiNLI (Joshi et al., 2020b).”
BackgroundMethodInfluentialAutomated Paraphrase Quality Assessment Using Recurrent Neural Networks and Language Models
Nicula, B., Dascalu, M., Newton, N., Orcutt, E., & McNamara, D. (2021) · International Conference on Intelligent Tutoring Systems · doi:10.1007/978-3-030-80421-3_36 · Cited by 8
Sboev, A. G., Selivanov, A., Александрович, М. И., Борисович, Р. Р., Gryaznov, A. V., Sboeva, S., & Rylkov, G. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6010010 · Cited by 8
Knowledge Enhanced Coreference Resolution via Gated Attention
He, K., Mao, B., Zhou, X., Li, Y., Gong, T., Li, C., & Wu, J. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995551 · Cited by 9
“Our model is based on the higher-order coreference model [9] and SpanBERT pre-training model [10].”
BackgroundMethodSolving Geometry Problems via Feature Learning and Contrastive Learning of Multimodal Data
Jian, P., Guo, F., Wang, Y., & Li, Y. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.023243 · Cited by 8
Park, J., & Cho, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121021 · Cited by 8
Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords
Golchin, S., Surdeanu, M., Tavabi, N., & Kiapour, A. M. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.2 · Cited by 13
MediBioDeBERTa: Biomedical Language Model With Continuous Learning and Intermediate Fine-Tuning
Kim, E., Jeong, Y., & Choi, M.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3341612 · Cited by 13
“…(LMs) have been pre-trained to acquire general knowledge and contextual characteristics from large The associate editor coordinating the review of this manuscript and approving it for publication was Agostino Forestiero . corpora, such as Wikipedia, news articles, and books [2], [4], [5], [6], [7].”
BackgroundWho are you referring to? Coreference resolution in image narrations
Goel, A., Fernando, B., Keller, F., & Bilen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01399 · Cited by 18
“SpanBERT [21] uses a span-based masked prediction objective for pre-training and shows improvements on the downstream task of CR. Stolfo et al .”
BackgroundMethodInfluentialImproving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Zhou, W., Ge, T., Xu, C., Xu, K., & Wei, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.45 · Cited by 16
“Different from conventional KD (Hin-ton et al., 2015) or sequence-level KD (Kim and Rush, 2016), SSR enables the student model to exploit both teacher outputs and the ground truth at the same time.”
BackgroundNormal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction
Li, L., Chen, X., Bi, Z., Xie, X. Y., Deng, S., Zhang, N., Tan, C., Chen, M., & Chen, H. (2021) · International Joint Conference on Knowledge Graphs (IJCKG) · doi:10.1145/3502223.3502237 · Cited by 8
GENE: Global Event Network Embedding
Zeng, Q., Li, M., Lai, T., Ji, H., Bansal, M., & Tong, H. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.5 · Cited by 8
Make aspect-based sentiment classification go further: step into the long-document-level
Wu, Z., Gao, J., Li, Q., Guan, Z., & Chen, Z. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02836-y · Cited by 9
“Transformer-like models have an upper limit of input length except for XLNet [34].”
BackgroundRethinking Denoised Auto-Encoding in Language Pre-Training
Luo, F., Yang, P., Li, S., Ren, X., Sun, X., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.232 · Cited by 9
“Noise types BERT (Devlin et al., 2019) Mask tokens SpanBERT (Joshi et al., 2019) Mask spans RoBERTa (Liu et al., 2019) Mask token XLNet (Yang et al., 2019) Shuffle token ELECTRA (Clark et al., 2019) Replace tokens StructBERT (Wang et al., 2019b) Mask + Shuffle tokens BART (Lewis et al., 2019) Mask +…”
BackgroundIITK-RSA at SemEval-2020 Task 5: Detecting Counterfactuals
Ojha, A. A., Garg, R., Gupta, S., & Modi, A. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.56 · Cited by 8
EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering SystemPreprint
Chaybouti, S., Saghe, A., & Shabou, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02157 · Cited by 9
“In SpanBERT (Joshi et al., 2020), the pre-training task of the language model is masked span prediction instead of masked word prediction to be better adapted to the downstream task of QA, which consists of span extraction.”
Learning Robot Structure and Motion Embeddings using Graph Neural NetworksPreprint
Kim, J. T., Park, J., Choi, S., & Ha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07543 · Cited by 20
An analysis of document graph construction methods for AMR summarizationPreprint
Lee, F.-T., Kedzie, C., Verma, N., & McKeown, K. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13993 · Cited by 11
“For text coreference we use SpanBERT-large (Joshi et al., 2020).”
Machine-in-the-Loop Rewriting for Creative Image Captioning
Padmakumar, V., & He, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.42 · Cited by 7
Zhao, W., Zhang, Y., Wu, D., Wu, F., & Jain, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102769 · Cited by 8
BERT Mutation: Deep Transformer Model for Masked Uniform Mutation in Genetic Programming
Tov, E. S., Sipper, M., & Elyasaf, A. (2025) · Mathematics · doi:10.3390/math13050779 · Cited by 7
Ye, J., Li, J., Zhao, C., Zhou, C., Zhou, X., & Huang, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107588 · Cited by 7
A Survey on Retrieval-Augmented Generation Technology in Large Language Models
Guo, J., Yang, H., & Liu, Q. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-4788-3_23 · Cited by 8
Heterogeneity in entity matching: A survey and experimental analysis
Moslemi, M. H., Mousavi, A., Behkamal, B., & Milani, M. (2026) · Data & Knowledge Engineering · doi:10.1016/j.datak.2026.102575 · Cited by 10
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
Jiao, F., Guo, Y., Niu, Y., Ji, F., Li, F.-L., & Nie, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.13 · Cited by 10
HORNET: Enriching Pre-trained Language Representations with Heterogeneous Knowledge Sources
Zhang, T., Cai, Z., Wang, C., Li, P., Li, Y., Qiu, M., Tang, C., He, X., & Huang, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482436 · Cited by 10
Cooperative Self-training of Machine Reading Comprehension
Luo, H., Li, S.-W. D., Gao, M., Yu, S., & Glass, J. R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.18 · Cited by 15
“Joshi et al. (2020) leveraged a masked span generation task instead of word prediction.”
BackgroundAutomation of Vulnerability Information Extraction Using Transformer-Based Language Models
Chaleshtori, F. H., & Ray, I. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25460-4_37 · Cited by 10
Genealogical Relationship Extraction from Unstructured Text Using Fine-Tuned Transformer Models
Parrolivelli, C., & Stanchev, L. (2023) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc56153.2023.00035 · Cited by 7
“Paper [16] looks to optimize BERT as well, though specifically for Natural Language Processing (NLP) tasks where reasoning between spans of text, rather than just words, is more important.”
BackgroundSeq2seq is All You Need for Coreference Resolution
Zhang, W., Wiseman, S., & Stratos, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.704 · Cited by 37
Open Knowledge Graph Link Prediction with Semantic-Aware Embedding
Wang, J., Huang, H., Wu, Y., Zhang, F., Zhang, S., & Guo, K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123542 · Cited by 9
ArQuAD: An Expert-Annotated Arabic Machine Reading Comprehension Dataset
Obeidat, R., Al-Harbi, M., Al‐Ayyoub, M., & Alawneh, L. M. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10248-6 · Cited by 13
PURE: a Prompt-based framework with dynamic Update mechanism for educational Relation Extraction
Cui, X., Yang, Y., Li, D., Cui, J., Qu, X., Song, C. L., Liu, H., & Ke, S. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01692-w · Cited by 9
“• SpanBERT [14] differs from BERT in both the masking strategy and the training objectives.”
Elbiach, O., Grissette, H., & Nfaoui, E. H. (2025) · AI · doi:10.3390/ai6020031 · Cited by 11
Direct speech-to-speech neural machine translation: A survey
Gupta, M., Dutta, M., & Maurya, C. K. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103317 · Cited by 18
Adapting Coreference Resolution to Twitter Conversations
Aktaṣ, B., Solopova, V., Kohnert, A., & Stede, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.222 · Cited by 9
“Due to its cost efficiency, speed and flexibility, it was later used as basis for several recent state-of-art models, including SpanBERT (Joshi et al., 2020).”
MethodTraining ELECTRA Augmented with Multi-word Selection
Shen, J., Liu, J., Liu, T., Yu, C., & Han, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.219 · Cited by 9
MVP-BERT: Multi-Vocab Pre-training for Chinese BERT
Zhu, W. (2021) · doi:10.18653/v1/2021.acl-srw.27 · Cited by 7
“In Span-BERT (Joshi et al., 2019), text spans are masked .”
BackgroundSemantic key generation based on natural language
Wu, Z., Kang, J., & Jiang, Q. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22711 · Cited by 7
“Joshi et al. 17 proposed the SpanBERT model, which examined a paragraph and used the remaining fragments to predict a word in the paragraph, to enable the model to learn longer information.”
MethodLi, J., Wei, Q., Ghiasvand, O., Chen, M., Lobanov, V. S., Weng, C., & Xu, H. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00095 · Cited by 9
Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension
Zhu, H., Wang, Z., Zhang, H., Liu, M., Zhao, S., & Qin, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.95 · Cited by 8
“State-of-the-art systems based on pre-trained language models (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Dong et al., 2019; Joshi et al., 2020) have achieved remarkable performance on the task.”
BackgroundExploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution
Min, B. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.10 · Cited by 7
“BERT-based models (Joshi et al., 2019a, 2020) further improve E2E-COREF by replacing the Bi-LSTM text encoder with BERT (Devlin et al., 2019) or SPAN BERT (Joshi et al., 2020).”
MethodNeural coreference resolution for Slovene language
Klemen, M., & Žitnik, S. (2021) · Computer Science and Information Systems · doi:10.2298/csis201120060k · Cited by 7
“Subsequent work includes modifications such as the use of more sophisticated contextual embeddings [11] or more specialized ones [12] inside the end-to-end system.”
MethodSame Sentence Prediction: A new Pre-training Task for BERT
Sonmezoz, K., & Amasyalı, M. F. (2021) · Innovations in Intelligent Systems and Applications Conference (ASYU) · doi:10.1109/asyu52992.2021.9598954 · Cited by 7
An Extraction and Representation Pipeline for Literary Characters
Yang, F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21709 · Cited by 8
Unsupervised Slot Schema Induction for Task-oriented Dialog
Yu, D., Wang, M., Cao, Y., Izhak Shafran, J., Shafey, L. E., & Soltau, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.86 · Cited by 20
A Cue Prompt Adapting Model for Relation Extraction
Wang, K., Chen, Y., Wen, K., Wei, C. Y., Dong, B., Zheng, Q., & Qin, Y. (2022) · Connection Science · doi:10.1080/09540091.2022.2161478 · Cited by 8
“…have been developed to learn contextual features and semantic dependencies relevant to the considered entities, such as position embedding (Zeng et al., 2015), multi-channel (Chen et al., 2020), neuralized feature engineering(Chenetal.,2021),andentityindicators(Qinetal.,2021;W.Zhou&Chen,2021).”
MethodA Benchmark for Dutch End-to-End Cross-Document Event Coreference Resolution
Langhe, L. D., Desot, T., Clercq, O. D., & Hoste, V. (2023) · Electronics · doi:10.3390/electronics12040850 · Cited by 7
“Overall, joint methods, with joint inference methods in particular, obtain the best results within the field [6], especially when combined with well-performing entity coreference resolution systems [7] and transformer-based architectures [28].”
BackgroundMethodBERT‐TriF: An inductive short text classification model for power equipment defect records
Ye, Z., Guo, B., & Qi, D. (2023) · Engineering Reports · doi:10.1002/eng2.12671 · Cited by 8
“30 In addition, Some recent studies have combined Graph Convolutional Networks (GCN) to exiting baseline models like attention mechanism and BERT in the task of text classification.”
BackgroundSelf-Attention Over Tree for Relation Extraction With Data-Efficiency and Computational Efficiency
Lyu, S., Zhou, X., Wu, X., Chen, Q., & Chen, H. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3286268 · Cited by 7
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
Hu, D., Hou, X., Du, X., Zhou, M., Jiang, L., Mo, Y., & Shi, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.468 · Cited by 21
Self-Evolution Learning for Discriminative Language Model Pretraining
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.254 · Cited by 14
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution
Gupta, A., Karpinska, M., Zhao, W., Krishna, K., Merullo, J., Yeh, L., Iyyer, M., & O’Connor, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.24 · Cited by 9
Wei, 龚. G., Wenhua, 赵. Z., Xintian, 王. W., Zhenze, 李. L., Yi, 王. W., Xinjing, 赵. Z., Qing, 王. W., Yanhui, 王. W., Lei, 王. W., & Qidai, 陈. C. (2023) · Chinese Journal of Lasers · doi:10.3788/cjl230827 · Cited by 7
Schema-adaptable Knowledge Graph Construction
Ye, H., Gui, H., Xu, X., Chen, X., Chen, H., & Zhang, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.425 · Cited by 19
Han, Q., & Ma, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-56166-3 · Cited by 7
“Notably, inter-layer self-attention in SPAN-BERT 12 and bidirectional transformers in OntoNLP 13 have shown significant improvements in feature extraction capabilities.”
BackgroundFree the Plural: Unrestricted Split-Antecedent Anaphora Resolution
Yu, J., Moosavi, N. S., Paun, S., & Poesio, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.538 · Cited by 16
Ghosh, S., & Kumar, S. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.29 · Cited by 10
Multi-Granularity Contrasting for Cross-Lingual Pre-Training
Li, S., Yang, P., Luo, F., & Xie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.149 · Cited by 7
“Previous work (Joshi et al., 2019) has demonstrated the superiority of span-level representations over word-level (Devlin et al.”
BackgroundElazar, Y., Basmov, V., Goldberg, Y., & Tsarfaty, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00488 · Cited by 20
“Next, we experiment with three neural models based on a pre-trained masked language model (MLM), specifically, SpanBERT (Joshi et al., 2020).”
MethodInfluentialA Semantic Annotation Pipeline towards the Generation of Knowledge Graphs in Tribology
Kügler, P., Marian, M., Dorsch, R., Schleich, B., & Wartzack, S. J. (2022) · Lubricants · doi:10.3390/lubricants10020018 · Cited by 9
“Furthermore, SpanBERT [62] is a pre-training approach, which is focused on a representation of text spans instead of single tokens.”
BackgroundInfluentialSimulating Bandit Learning from User Feedback for Extractive Question Answering
Gao, G., Choi, E., & Artzi, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.355 · Cited by 15
Interpretable Proof Generation via Iterative Backward Reasoning
Qu, H., Cao, Y. H., Gao, J., Ding, L., & Xu, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.216 · Cited by 20
Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis
Ein‐Dor, L., Shnayderman, I., Spector, A., Dankin, L., Aharonov, R., & Slonim, N. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21305 · Cited by 16
Hard-style Selective Context Utilization for dialogue generation based on what user just said
Ling, Y., Zheng, L., Wang, T., Cai, F., & Chen, H. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109873 · Cited by 8
Paragraph-based Transformer Pre-training for Multi-Sentence Inference
Liello, L. D., Garg, S., Soldaini, L., & Moschitti, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.181 · Cited by 9
Improving Robustness and Generality of NLP Models Using Disentangled RepresentationsPreprint
Wu, J., Li, X., Ao, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09587 · Cited by 14
“…have achieved remarkable success in a wide range of NLP tasks, such as language modeling (Xie et al., 2017; Devlin et al., 2018a; Liu et al., 2019; Joshi et al., 2020; Meng et al., 2019b), machine reading comprehension (Seo et al., 2016; Yu et al., 2018), and machine translation (Sutskever et…”
BackgroundSpeaker-Oriented Latent Structures for Dialogue-Based Relation ExtractionPreprint
Nan, G., Luo, G., Leng, S., Xiao, Y., & Lu, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05182 · Cited by 7
“Although pre-trained models have demonstrated effectiveness on RE tasks (Joshi et al., 2020; Wang et al., 2020), we will empirically show that these BERT-based models may still have limitations in capturing speaker-related contexts or addressing the two issues of DiaRE mentioned above.”
MethodResultTransfer Learning for Multi-lingual Tasks -- a SurveyPreprint
Jafari, A. R., Heidary, B., Farahbakhsh, R., Salehi, M., & Jalili, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02052 · Cited by 11
Deep Learning-Based Operators for Evolutionary Algorithms
Tov, E. S., Sipper, M., & Elyasaf, A. (2025) · Genetic and evolutionary computation · doi:10.1007/978-981-96-0077-9_3 · Cited by 6
Chen, B., Zhao, E., BianBaDroMa, ‘., & Ngodrup (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111365 · Cited by 6
Yang, X., Li, B., Jia, H., Ma, Z., Sun, H., Xue, C., & Zhang, Z. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108551 · Cited by 7
Tang, Z. S., & Surdeanu, M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00463 · Cited by 12
“22 Table 1 shows the hyperparameter details for training the neural models for relation classification (SpanBERT) and both relation and explainability classification (Unsupervised Rationale and our approach).”
MethodInfluentialBebensee, B., & Lee, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.6 · Cited by 12
Exploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.441 · Cited by 39
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.