SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 501–600 of 1,012
A Graph Reasoning Network for Multi-turn Response Selection via Customized Pre-training
Liu, Y., Feng, S., Wang, D., Song, K., Ren, F., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17585 · Cited by 21
Self-citationShow Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet
He, X., & Li, V. O. K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17536 · Cited by 19
“…by large-scale pre-trained language models such as GPT (Radford 2018), GPT-2 (Radford et al. 2019), BERT (Devlin et al. 2019), ROBERTA (Liu et al. 2019c), ELECTRA (Clark et al. 2020), SpanBERT (Joshi et al. 2020), XLNet (Yang et al. 2019), MASS (Song et al. 2019) and BART (Lewis et al. 2020).”
MethodImproving question answering performance using knowledge distillation and active learning
Boreshban, Y., Mirbostani, S. M., Ghassem-Sani, G., Mirroshandel, S. A., & Amiriparian, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106137 · Cited by 23
MusicBERT: A Self-supervised Learning of Music Representation
Zhu, H., Niu, Y., Fu, D., & Wang, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475576 · Cited by 22
“Recently,pre-trainingmodels[11,22,42]dominateNLParea,through learninguniversalrepresentationstohelpdownstreamtasks.”
BackgroundT3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP
Li, R., Xiao, W., Wang, L., Jang, H., & Carenini, G. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.26 · Cited by 32
“, 2019) along with other Transformer-based models pretrained on large corpora (Liu et al., 2019; Lewis et al., 2020; Joshi et al., 2020; Lee et al., 2020), the most successful strategy on many NLP leaderboards has been to directly fine-tune such models on the downstream tasks (e.”
BackgroundMethodImproving negation detection with negation-focused pre-training
Truong, T. H., Baldwin, T., Cohn, T., & Verspoor, K. M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.309 · Cited by 31
Empathetic Response Generation via Emotion Cause Transition Graph
Qian, Y., Wang, B., Lin, T.-E., Zheng, Y., Zhu, Y., Zhao, D., Hou, Y., Wu, Y., & Li, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095652 · Cited by 27
“To identify emotion cause spans, we exploit pre-trained span-level SpanBERT [22] to encode the dialog context and corresponding emotion label.”
MethodPaired Representation Learning for Event and Entity Coreference.Preprint
Yu, X., Yin, W., & Roth, D. (2020) · arXiv (Cornell University) · Cited by 21
Building Chinese Biomedical Language Models via Multi-Level Text DiscriminationPreprint
Wang, Q., Dai, S., Xu, B., Lyu, Y., Zhu, Y. T., Wu, H. R., & Wang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07244 · Cited by 27
“, 2019) and then widely adopted by follow-up studies (Liu et al., 2019; Lan et al., 2020; Joshi et al., 2020; Sun et al., 2020).”
BackgroundExamining risks of racial biases in NLP tools for child protective services
Field, A., Coston, A., Gandhi, N., Chouldechova, A., Putnam-Hornstein, E., Steier, D., & Tsvetkov, Y. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594094 · Cited by 18
“For both models we use a SpanBERT encoder [29] and train on OntoNotes [23].”
MethodA Survey of Semantic Parsing Techniques
Jiang, P., & Cai, X. (2024) · Symmetry · doi:10.3390/sym16091201 · Cited by 20
Aircraft assembly process planning based on knowledge graph constructed by integrating LLMs and SLMs
Ma, Y., Zheng, S., Yang, Z., Zheng, P., Leng, J., & Hong, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.11.016 · Cited by 20
ENPAR:Enhancing Entity and Entity Pair Representations for Joint Entity Relation Extraction
Wang, Y., Sun, C., Wu, Y., Zhou, H., Li, L., & Yan, J. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.251 · Cited by 20
“Pre-trained models (Yang et al., 2019; Dong et al., 2019; Joshi et al., 2020) have made many amazing breakthroughs on various NLP down-stream tasks.”
BackgroundData Efficient Masked Language Modeling for Vision and Language
Bitton, Y., Elhadad, M., Stanovsky, G., & Schwartz, R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.259 · Cited by 20
SOSum: a dataset of stack overflow post summaries
Kou, B., Di, Y., Chen, M., & Zhang, T. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528487 · Cited by 28
“Some advanced NLP models SE researchers can fine-tune include BERT, Sentence-bert, and SpanBERT [19, 27, 37].”
BackgroundPhilippy, F., Guo, S., & Haddadan, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.323 · Cited by 96
Learning Better Masking for Better Language Model Pre-training
Yang, D., Zhang, Z., & Zhao, H. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.400 · Cited by 30
Loukachevitch, N., Artemova, E., Batura, T. V., Braslavski, P., Ivanov, V. V., Manandhar, S., Pugachev, A., Рожков, І. М., Shelmanov, A., Tutubalina, E. V., et al. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09674-z · Cited by 24
“Such annotations pose challenges to current state-of-the-art relation extraction 1 3 NEREL: a Russian information extraction dataset with rich… models that lack support for relations between nested and overlapping entities (Alt et al., 2019; Joshi et al., 2020).”
How to Detect AI-Generated Texts?
Nguyen, T. T., Hatua, A., & Sung, A. H. (2023) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon59035.2023.10316132 · Cited by 33
“5 [8], Llama [9], etc. perform (NLP-based tasks) better than their predecessor transformer-based models such as BERT [10], RoBERTa [11], DistilBERT [12], SPANBERT [13], etc.”
MethodA Causal View of Entity Bias in (Large) Language Models
Wang, F., Mo, W., Wang, Y., Zhou, W., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1013 · Cited by 77
Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation
Cui, Y., Xie, L., Zhang, Y., Zhang, M., Yan, Y., & Yin, E. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01106 · Cited by 50
“Furthermore, through three pre-training objectives — the MLM, Span Boundary Ob-jective (SBO) [24], and Perturbation Masking Objective (PMO) — Wu et al. [62] improved the overall performance of a dialogue understanding model.”
BackgroundAli, S. J., Naganathan, V., & Bork, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75872-0_16 · Cited by 39
Span-Based Event Coreference Resolution
Lu, J. J., & Ng, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17591 · Cited by 24
“We then pass the sequence into a pretrained transformer encoder in SpanBERTlarge (Joshi et al. 2020) to encode tokens and their contexts.”
MethodInfluentialLee, D., Ahn, J., Park, H., & Jo, J. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.2 · Cited by 31
“In addition, motivated by SpanBERT (Joshi et al., 2020), we always mask complete words instead of sub-word tokens since translation suggestion requires predictions of complete words.”
Child-directed speech is optimized for syntax-free semantic inference
You, G., Bickel, B., Daum, M. M., & Stoll, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-95392-x · Cited by 27
Sboev, A. G., Sboeva, S., Александрович, М. И., Gryaznov, A. V., Борисович, Р. Р., Naumov, A., Selivanov, A., Rylkov, G., & Ilyin, V. (2022) · Applied Sciences · doi:10.3390/app12010491 · Cited by 24
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation
Ye, J., Ding, Y., Bapna, A., Cherry, C., Zhang, Y., Conneau, A., & Morioka, N. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10938 · Cited by 31
From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in AlibabaPreprint
Qi, X., Hou, K., Liu, T., Yu, Z., Hu, S., & Ou, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08381 · Cited by 28
“That is, in the training stage, we introduce span masking(Joshi et al. 2020) in addition to the naive strategy.”
MethodarXivEdits: Understanding the Human Revision Process in Scientific Writing
Jiang, C., Xu, W., & Stevens, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.641 · Cited by 18
Collecting a large-scale gender bias dataset for coreference resolution and machine translation
Levy, S., Lazar, K., & Stanovsky, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.211 · Cited by 41
GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding
Yan, Y., & Lee, J. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679934 · Cited by 48
“In toponym recognition, we compare GeoRea-soner with fine-tuned models such as BERT [4], SimCSE-BERT [6], SpanBERT [15], SapBERT [19] and GeoLM [17].”
Towards Realistic Few-Shot Relation Extraction
Brody, S., Wu, S., & Benton, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.433 · Cited by 27
“These included RoBERTa (Liu et al., 2019), Span-Bert (Joshi et al., 2020), and LUKE (Peters et al., 2018).”
MethodRST Discourse Parsing with Second-Stage EDU-Level Pre-training
Yu, N., Zhang, M., Fu, G., & Zhang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.294 · Cited by 19
“, 2020), SpanBERT (Joshi et al., 2020), and DeBERTa (He et al.”
BackgroundMethodInfluentialDetection of dangerous events on social media: a critical review
Jamil, M. L., Pais, S., & Cordeiro, J. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00980-y · Cited by 18
“RoBERTa has been proven effective on a variety of tasks such as detection of mental illnesses (Murarka et al. 2020), offensive language detection (Tanase et al. 2020), protest event detection (Re et al. 2021) etc. XLNet (Yang et al. 2019) model is an extension of the Transformer-XL model (Dai et al. 2019).”
MethodInfluentialAttSum: A Deep Attention-Based Summarization Model for Bug Report Title Generation
Ma, X., Keung, J., Yu, X., Zou, H., Zhang, J., & Li, Y. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3236404 · Cited by 18
“Nevertheless, some recent research works [13], [14] have doubts about the necessity of adopting next sentence prediction (NSP) used by BERT since the experimental results could not show consistent improvement in their studies.”
MethodResultPQuAD: A Persian question answering dataset
Darvishi, K., Shahbodaghkhan, N., Abbasiantaeb, Z., & Momtazi, S. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101486 · Cited by 22
“Among the available studies on NewsQA dataset (Joshi et al., 2020; Tay et al., 2018; Kundu and Ng, 2018), the SpanBERT model (Joshi et al.”
BackgroundMethodResultInfluentialPre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Liello, L. D., Garg, S., Soldaini, L., & Moschitti, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.810 · Cited by 22
Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future Directions
Jia, Q., Liu, Y., Ren, S., & Zhu, K. Q. (2023) · ACM Computing Surveys · doi:10.1145/3622933 · Cited by 30
“[42] augmented and refined the original training pairs with an utterance rewriter model [103] and a coreference resolution model [69], leading to understandable dialogues.”
BackgroundMethodSelf-citationPredictive Chemistry Augmented with Text Retrieval
Qian, Y., Li, Z., Tu, Z., Coley, C. W., & Barzilay, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.784 · Cited by 35
“Inspired by previous research on language model pre-training (Devlin et al., 2019; Lewis et al., 2020a; Joshi et al., 2020; Du et al., 2022), the masking is performed by repeatedly sampling spans of length drawn from a Poisson distribution with λ = 3 , until the total number of masked tokens…”
MethodRethinking Point Cloud Registration as Masking and Reconstruction
Chen, G., Wang, M., Yuan, L., Yang, Y. M., & Yue, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01624 · Cited by 25
“This observation leads us to exploit the adaptation of the masked data modeling (MDM), which has exhibited remarkable potential in natural language processing (NLP) [11, 24] and computer vision [20, 48].”
BackgroundMethodAggarwal, S., & Vishwakarma, D. K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124278 · Cited by 19
A Survey of Text-Matching Techniques
Jiang, P., & Cai, X. (2024) · Information · doi:10.3390/info15060332 · Cited by 25
TUNeS: A Temporal U-Net With Self-Attention for Video-Based Surgical Phase Recognition
Funke, I., Rivoir, D., Krell, S., & Speidel, S. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2025.3535228 · Cited by 19
“In addition, we mask spans in the token sequences that are input to the Transformer blocks, inspired by the pre-training procedures for masked language models [52], [53].”
CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
Su, Y., Han, X., Lin, Y., Zhang, Z., Liu, Z., Li, P., Zhou, J., & Sun, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3105013 · Cited by 19
“…designed specific self-supervised learning objectives to help PLMs learn specific abilities in the pre-training phase, such as representing token spans [6] and entities [15], [16], [17], [17], [18], conferential reasoning [19], multi-lingualism [20], multi-modality [21], [22], [23], [24], [25], etc.”
BackgroundBidirectional Language Modeling: A Systematic Literature Review
Jahan, M. S., Khan, H. U., Akbar, S., Farooq, M. U., Gul, S., & Amjad, A. (2021) · Scientific Programming · doi:10.1155/2021/6641832 · Cited by 24
“Research studies GLUE [10, 11, 14, 15, 20, 27, 29–33, 36–48, 50–52] BLEU [28, 29, 36, 41, 50, 52, 54] SQuAD [10, 11, 20, 27–31, 33–36, 40, 41, 49, 50, 52] RACE [27, 28, 33, 34, 38, 43] 10 Scientific Programming large-scale textual. model uses lexical, syntactic, and knowledge information with MLM…”
MethodGeneralizing Cross-Document Event Coreference Resolution Across Multiple Corpora
Bugert, M., Reimers, N., & Gurevych, I. (2021) · Computational Linguistics · doi:10.1162/coli_a_00407 · Cited by 36
“…(event mention detection and event coreference resolution), several authors have decided to omit the first step and work on gold mentions alone (Cybulska and Vossen 2015; Kenyon-Dean, Cheung, and Precup 2018; Barhom et al. 2019; Meged et al. 2020) which simplifies the task and system development.”
BackgroundInfluentialA span-based joint model for extracting entities and relations of bacteria biotopes
Zuo, M., & Zhang, Y. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab593 · Cited by 17
Raval, S., Sedghamiz, H., Santus, E., Alhanai, T., Ghassemi, M. M., & Chersoni, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.300 · Cited by 25
Coreference Resolution for the Biomedical Domain: A Survey
Lu, P., & Poesio, M. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.2 · Cited by 22
Rethinking data-driven networking with foundation models: challenges and opportunities
Le, F., Srivatsa, M., Ganti, R., & Sekar, V. (2022) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3563766.3564109 · Cited by 27
“html [25] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy.”
BackgroundCAPT: Contrastive Pre-Training for Learning Denoised Sequence RepresentationsPreprint
Luo, F., Yang, P., Li, S., Ren, X., & Sun, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06351 · Cited by 21
“4 SpanBERT [13] 64.”
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2021) · arXiv (Cornell University) · Cited by 19
AutoRC: Improving BERT Based Relation Classification Models via Architecture Search
Zhu, W. (2021) · doi:10.18653/v1/2021.acl-srw.4 · Cited by 26
“The metric is MUC F1 and we choose the state-of-theart (SOTA) SpanBERT (Joshi et al., 2019) as base-”
MethodAn ELECTRA-Based Model for Neural Coreference Resolution
Gargiulo, F., Minutolo, A., Guarasci, R., Damiano, E., Pietro, G. D., Fujita, H., & Esposito, M. (2022) · IEEE Access · doi:10.1109/access.2022.3189956 · Cited by 18
A Survey of Text Representation Methods and Their Genealogy
Siebers, P., Janiesch, C., & Zschech, P. (2022) · IEEE Access · doi:10.1109/access.2022.3205719 · Cited by 25
“SpanBERT.”
BackgroundInfluentialEnhancing text classification with attention matrices based on BERT
Yu, Z., Li, H., & Feng, J. (2023) · Expert Systems · doi:10.1111/exsy.13512 · Cited by 19
“And SpanBERT (Joshi et al., 2020) introduces the concept of phrase masking to encourage the model to learn more comprehensive information.”
BackgroundNegative Review or Complaint? Exploring Interpretability in Financial Complaints
Das, S., Singh, A., Saha, S., & Maurya, A. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3338357 · Cited by 21
“SpanBERT [32] < USER > Please help me to settle my loan request its already 1 month passed.”
BackgroundPTCAS: Prompt tuning with continuous answer search for relation extraction
Chen, Y., Shi, B., & Xu, K. (2024) · Information Sciences · doi:10.1016/j.ins.2023.120060 · Cited by 16
Zou, L., He, Z., Zhou, C., & Zhu, W. (2024) · International Journal of Digital Earth · doi:10.1080/17538947.2024.2348668 · Cited by 18
“SpanBERT extends BERT to better represent and infer text spans (Joshi et al. 2020).”
CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios
Zeng, Z., Wang, Y., Xie, R., Ye, W., & Zhang, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652115 · Cited by 33
“Thus, we believe that a possible reason is the high similarities between pre-training tasks, functional code, and test generation tasks, so that base LLMs can accomplish these tasks without aligning with upstream and downstream tasks [20, 42, 47].”
BackgroundEnhancing deep neural networks with morphological information
Klemen, M., Krsnik, L., & Robnik‐Šikonja, M. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000080 · Cited by 27
“apture the language morphology. Since the release of BERT, several new pre-training objectives have been proposed, such as syntactic and semantic phrase masking [Zhou et al., 2020b] and span masking [Joshi et al., 2020]. In further work, it makes sense to apply these models to the DP task in order to test how well the…”
MethodTwo-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding
Kim, S., Kim, G., Shin, S., & Lee, S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414619 · Cited by 23
“Following SpanBERT [19], we randomly choose pd of all tokens as a starting index and mask Md consecutive tokens from every selected index with allowing overlap.”
MethodTarget-oriented Fine-tuning for Zero-Resource Named Entity Recognition
Zhang, Y., Meng, F., Chen, Y., Xu, J., & Zhou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.140 · Cited by 16
Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval
Xiao, S., Liu, Z., Han, W., Zhang, J., Shao, Y., Lian, D., Li, C., Sun, H., Deng, D., Zhang, L., et al. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511957 · Cited by 20
“Such models leverage large-scale transformers as backbones and get fully pretrained with massive corpus, which achieves strong expressiveness, and contributes to numerous downstream tasks, such as document retrieval [4, 20, 37] and question answering [14, 19, 22].”
BackgroundUsing Type Information to Improve Entity Coreference Resolution
Khosla, S., & Rosé, C. P. (2020) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2020.codi-1.3 · Cited by 24
Sagawa, T., & Kojima, R. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-01075-4 · Cited by 15
“Span-MLM presents a more challenging scenario requiring the model to interpret broader contextual information [17], thereby enhancing its ability to understand molecular structures from SMILES representations.”
Better Fine-Tuning with Extracted Important Sentences for Abstractive Summarization
Sahu, A., & Sanjeevi, S. G. (2021) · International Conference on Communication, Control and Information Sciences (ICCISc) · doi:10.1109/iccisc52257.2021.9484944 · Cited by 18
Yan, S., Wang, J., & Song, Z. (2022) · Future Internet · doi:10.3390/fi14080234 · Cited by 15
Supplementing domain knowledge to BERT with semi-structured information of documents
Chen, J., Wei, Z., Wang, J., Wang, R., Gong, C., Zhang, H., & Miao, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121054 · Cited by 18
Jiang, Y. E., Liu, T., Ma, S., Zhang, D., Sachan, M., & Cotterell, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.435 · Cited by 37
“We modify the model by replacing the span representations with SpanBERT embeddings (Joshi et al., 2020).”
MethodChang, H., Xu, H., van Genabith, J., Xiong, D., & Zan, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3310879 · Cited by 16
Temporal and spatial self supervised learning methods for electrocardiograms
Chen, W., Wang, H., Zhang, L., & Zhang, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-90084-2 · Cited by 22
Chakhtouna, A., Sekkate, S., & Adib, A. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01088-y · Cited by 15
“These embeddings capture semantic meanings and syntactic structures in textual data, making RoBERTa suitable for various NLP tasks such as sentiment analysis, text classification, and question answering [43, 44].”
Effective Use of Transformer Networks for Entity Tracking
Gupta, A., & Durrett, G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1070 · Cited by 24
“Transformer based pre-trained language models (Devlin et al., 2019; Radford et al., 2018, 2019; Joshi et al., 2019; Yang et al., 2019) have been shown to perform remarkably well on a range of tasks, including entity-related tasks like coreference resolution (Kantor and Globerson, 2019) and named entity recognition (De…”
BackgroundEntity Recognition and Relation Extraction from Scientific and Technical Texts in Russian
Bruches, E. P., Pauls, A., Batura, T. V., & Isachenko, V. (2020) · Science and Artificial Intelligence Conference (S.A.I.ence) · doi:10.1109/s.a.i.ence50533.2020.9303196 · Cited by 16
“85 for NER and the SpanBERT [9] model gives the best results of 0.”
MethodInfluentialArabizi Language Models for Sentiment Analysis
Baert, G., Gahbiche, S., Gadek, G., & Pauchet, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.51 · Cited by 17
“We thus decide to exclude NSP from our pre-training step and to only focus on MLM. 11 https://huggingface.co/transformers/index.html 12 https://github.com/google/sentencepiece MLM usually obtains good performance as pre-training step for Transformer model, in order to solve NLP tasks, in particular with long sequences…”
ResultInfluentialChhablani, G., Sharma, A., Pandey, H., Bhartia, Y., & Suthaharan, S. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.27 · Cited by 27
“The work presented in this paper aims to provide a comprehensive analysis of simple Token Classifica-tion (TC) and Span Prediction (SP) methods across multiple BERT-based models - BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and SpanBERT (Joshi et al., 2020).”
MethodInfluentialYang, Y., Cao, Z., Zhao, P., Zeng, D. D., Zhang, Q., & Luo, Y. (2021) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2021.08.002 · Cited by 16
“The pre-trained language models adapted in this paper is SpanBERT [30] .”
MethodZaporojets, K., Deleu, J., Jiang, Y., Demeester, T., & Develder, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.88 · Cited by 24
Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables
Wang, W., Zhang, Z., Du, Y., Chen, B., Xie, J., & Luo, W. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.366 · Cited by 26
Sentiment Analysis of News Headlines Using Simple Transformers
Singh, A. K., & Jain, G. (2021) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon51346.2021.9544806 · Cited by 17
NLP-Fast: A Fast, Scalable, and Flexible System to Accelerate Large-Scale Heterogeneous NLP Models
Kim, J., Hur, S., Lee, E., Lee, S., & Kim, J. (2021) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact52795.2021.00013 · Cited by 20
MetaQA: Combining Expert Agents for Multi-Skill Question Answering
Puerto, H., Şahin, G. G., & Gurevych, I. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.259 · Cited by 23
A novel chinese relation extraction method using polysemy rethinking mechanism
Zhao, Q., Gao, T., & Guo, N. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03817-5 · Cited by 15
“Sun et al. [5–8] However, most of the current work is based on English datasets, and only a minimal amount is based on datasets from the Chinese corpus.”
BackgroundSemi‐automatic coding of open‐ended text responses in large‐scale assessments
Andersen, N., Zehner, F., & Goldhammer, F. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12717 · Cited by 17
“An additional focus on corefer-ences could mitigate this information loss by referring pronouns to the relevant entity, meaning that additional rules (e.g. Hobbs, 1978) or trans-formers (e.g. Joshi et al., 2020) could be used to reconstruct this coreference automatically.”
Computational Models of Anaphora
Poesio, M., Yu, J., Paun, S., Aloraini, A., Lu, P., Haber, J., & Çokal, D. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-111653 · Cited by 42
“…Kuhn 2014, Durrett & Klein 2014, Fernandes et al. 2014, Martschat & Strube 2015) and then to neural models that appear to have addressed many of the problems of previous models, resulting in an impressive performance (Wiseman et al. 2015; Lee et al. 2017, 2018; Joshi et al. 2020; Yu et al. 2020c).”
BackgroundMethodNeural machine translation for Hungarian
Laki, L. J., & Yang, Z. G. (2022) · Acta Linguistica Academica · doi:10.1556/2062.2022.00576 · Cited by 24
Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
Shi, P., Hu, M. Q., Shi, X., & Ren, F. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634706 · Cited by 17
“Based on stacking Transformer and large training corpus, BERT [22], XLNet [59], RoBERTa [25], ELECTRA [8], SpanBERT [21], and others are springing up.”
MethodZhang, X., Zeng, Y., Zhang, J., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.40 · Cited by 21
An Empirical Study of Efficient ASR Rescoring with TransformersPreprint
Huang, H., & Peng, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11450 · Cited by 23
“With the usage of self-attention mechanism and residual connections, it allows for successful training of very deep and high capacity networks, resulting in SOTA performance in many NLP tasks [1, 12, 13, 14].”
BackgroundA Cluster Ranking Model for Full Anaphora ResolutionPreprint
Yu, J., Uma, A., & Poesio, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09532 · Cited by 15
“, 2016b) so many systems attempted to explore features beyond those of mention pairs (Björkelund and Kuhn, 2014; Clark and Manning, 2015; Clark and Manning, 2016b; Lee et al., 2018; Kantor and Globerson, 2019; Joshi et al., 2019b; Joshi et al., 2019a).”
BackgroundLearning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14
Whang, T., Lee, D., Oh, D., Lee, C., Han, K., Lee, D., & Lee, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17653 · Cited by 41
The NLP Task Effectiveness of Long-Range Transformers
Qin, G., Feng, Y., & Van Durme, B. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.273 · Cited by 14
“10, we try to replace P layers of SpanBERT in a top-down manner, where P = 1 , 2 , . . . , 12 .”
MethodInfluentialA multi-modal transformer for cell type-agnostic regulatory predictions
Javed, N. M., Weingarten, T., Sehanobish, A., Roberts, A., Dubey, A., Choromański, K., & Bernstein, B. (2025) · Cell Genomics · doi:10.1016/j.xgen.2025.100762 · Cited by 30
“47 The network is then tasked with predicting the normalized ATAC-seq counts within each maskedbin.”
Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
Mehta, V., Sharma, C., & Thiyagarajan, K. (2025) · Sensors · doi:10.3390/s25206394 · Cited by 21
“[6], ALBERT [7], DistilBERT [8], RoBERTa [9], SpanBERT [10], GPT [11], GPT-2 [12], GPT-3 [13], GPT-4 [1], Llama 3.1 [14], InstructGPT [15], BART [16], DeepSeek R1 [5]), along with their parameter count and key feature of improvement.”
InfluentialSSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-Supervised Learning
Lv, P., Li, P., Zhu, S., Zhang, S., Chen, K., Liang, R., Yue, C., Xiang, F., Cai, Y., Ma, H., et al. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24374 · Cited by 27
Graph Receptive Transformer Encoder for Text Classification
Aras, A., Alikaşifoğlu, T., & Koç, A. (2024) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2024.3380362 · Cited by 15
Listwise Generative Retrieval Models via a Sequential Learning Process
Tang, Y., Zhang, R., Guo, J., de Rijke, M., Chen, W., & Cheng, X. (2024) · ACM Transactions on Information Systems · doi:10.1145/3653712 · Cited by 37
“Examples of discriminative models include BERT [23], RoBERTa [61], and SpanBERT [40].”
Method
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.