SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 301–400 of 1,012
Multi-Granularity Structural Knowledge Distillation for Language Model Compression
Liu, C., Tao, C., Feng, J., & Zhao, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.71 · Cited by 78
“Different from SpanBERT (Joshi et al., 2020) that randomly selects token spans whose start positions and lengths are sampled from some distributions for masked language modeling, we pro-pose to extract spans that have complete meanings.”
BackgroundTRANS-BLSTM: Transformer with Bidirectional LSTM for Language UnderstandingPreprint
Huang, Z., Xu, P., Liang, D., Mishra, A., & Xiang, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07000 · Cited by 54
“We note that using n-gram masking (for example, with n = 3) (Joshi et al., 2019; Lan et al., 2019) with the length of each n-gram mask selected randomly can further improve the downstream task accuracy (for example, 2% F1 score increase was observed on SQuAD 1.1 data set with n-gram masking and…”
BackgroundMethodInfluentialInterpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 80
“Deep learning based models have achieved state-of-the-art results on a variety of natural language processing (NLP) tasks such as tagging [127, 165, 217, 120], text classification [100, 221, 144, 121, 32, 199], machine translation [194, 59, 222, 187], natural language understanding [210, 110, 181, 200], dialog [4, 207…”
BackgroundMethodDialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation
Chen, W., Gong, Y., Wang, S., Yao, B., Qi, W., Wei, Z., Hu, X., Zhou, B., Mao, Y., Chen, W., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.333 · Cited by 64
“Recent research (Joshi et al., 2020; Lewis et al., 2020) on masked language models show the advantages of masking spans over masking individual words or subword units.”
BackgroundGraph Machine Learning in the Era of Large Language Models (LLMs)
Wang, S., Huang, J., Chen, Z., Song, Y., Tang, W., Mao, H., Fan, W., Liu, H., Liu, X., Yin, D., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3732786 · Cited by 103
“Three major lines of work in this direction include entity-centric tasks [106, 177, 228, 231, 256], relation-centric tasks [188], and beyond.”
New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles
Nguyen, K. V., Huynh, T. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3527631 · Cited by 55
“Four popular neural-network-based and transfer-learning-based MRC methods are chosen for the framework: DrQA Reader [2], QANet [52], BERT [7], and ALBERT [23], because these methods achieved state-of-the-art performances in many MRC tasks on SQuAD [7], NewsQA [19], and CMRC [4].”
MethodNentidis, A., Katsimpras, G., Vandorou, E., Krithara, A., Gascó, L., Krallinger, M., & Παλιούρας, Γ. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85251-1_18 · Cited by 49
BERT-like Pre-training for Symbolic Piano Music Classification TasksPreprint
Chou, Y.-H., Chen, I., Chang, C.-J., Ching, J., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05223 · Cited by 64
“We do not use NSP for our model since it was later shown to be not that useful (Conneau & Lample, 2019; Joshi et al. , 2020; Z. Yang et al. , 2019); moreover, NSP requires a clear definition of “sentences”, which is not well-defined for our MIDI sequences.”
MethodDesigning Precise and Robust Dialogue Response Evaluators
Zhao, T., Lala, D., & Kawahara, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.4 · Cited by 42
Nentidis, A., Krithara, A., Bougiatiotis, K., Krallinger, M., Rodriguez-Penagos, C. G., Villegas, M., & Παλιούρας, Γ. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58219-7_16 · Cited by 45
“They treat answer generation as an extractive machine comprehension task and explore several different pretrained language models, including BERT, BioBERT, XLNet [47] and SpanBERT [18].”
MethodA Sentiment-Aware Contextual Model for Real-Time Disaster Prediction Using Twitter Data
Song, G., & Huang, D.-G. (2021) · Future Internet · doi:10.3390/fi13070163 · Cited by 32
Chada, R., & Natarajan, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.491 · Cited by 68
Phrase Retrieval Learns Passage Retrieval, Too
Lee, J., Wettig, A., & Chen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.297 · Cited by 58
“(1) and (2), and they are initialized with BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2020), respectively.”
MethodSelf-citationMISS: Multi-Interest Self-Supervised Learning Framework for Click-Through Rate Prediction
Guo, W., Zhang, C., He, Z., Qin, J., Guo, H., Chen, B., Tang, R., He, X., & Zhang, R. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00059 · Cited by 54
Word Graph Guided Summarization for Radiology Findings
Hu, J., Li, J., Chen, Z., Shen, Y., Song, Y., Wan, X., & Chang, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.441 · Cited by 47
Liu, Z., Li, H., Wang, H., Liao, Y., Liu, X., & Wu, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120435 · Cited by 39
Downstream Model Design of Pre-trained Language Model for Relation Extraction TaskPreprint
Li, C., & Tian, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03786 · Cited by 50
“These methods [32, 25, 10, 27] have made some success, but we believe that they have not yet fully utilized the language model.”
MethodAll NLP Tasks Are Generation Tasks: A General Pretraining FrameworkPreprint
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · Cited by 56
Natural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 85
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.61 · Cited by 37
Re-TACRED: Addressing Shortcomings of the TACRED Dataset
Stoica, G., Platanios, E. A., & Poczos, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17631 · Cited by 31
AP-BERT: enhanced pre-trained model through average pooling
Zhao, S., Zhang, T., Hu, M., Chang, W., & You, F. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03190-3 · Cited by 42
“The SpanBERT [25] model masks a phrase and forces the model to learn more comprehensive information.”
BackgroundKnowledge Transfer via Pre-training for Recommendation: A Review and Prospect
Zeng, Z., Xiao, C., Yao, Y., Xie, R., Liu, Z., Lin, F., Lin, L., & Sun, M. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.602071 · Cited by 55
“The fine-tuning paradigm has shown the effectiveness in other areas, such as natural language processing (Devlin et al., 2019; Joshi et al., 2020).”
BackgroundMethodCapturing Global Informativeness in Open Domain Keyphrase Extraction
Sun, S. Y., Liu, Z., Xiong, C., Liu, Z., & Bao, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_21 · Cited by 57
“Experiments on two large-scale KPE datasets, OpenKP [39] and KP20k [23] with web pages and scientific papers, demonstrate JointKPE’s robust effectiveness with the widely used pre-trained model, BERT [7], and its two pre-training variants, SpanBERT [15] and RoBERTa [19].”
MethodInfluentialPanDa: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3376453 · Cited by 53
“Fine-tuning the pretrained language models (PLMs) has become the de facto standard for natural language processing (NLP), and has achieved remarkable success in numerous downstream tasks [6, 32, 16, 36, 1, 25, 4, 8, 23].”
BackgroundSelf-Explaining Structures Improve NLP ModelsPreprint
Sun, Z., Fan, C. C., Han, Q., Sun, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01786 · Cited by 62
“The strategy of using the starting and ending representations to represent a text span has been used in many recent works on span-level features in texts (Li et al., 2019; Joshi et al., 2020; Wu et al., 2019).”
MethodZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text EncodersPreprint
Song, Y., Zhang, T., Wang, Y., & Lee, K.-F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01279 · Cited by 55
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · arXiv · doi:10.48550/arxiv.2203.00343 · Cited by 37
Reasoning Like Program Executors
Pi, X., Liu, Q., Chen, B., Ziyadi, M., Lin, Z., Fu, Q., Gao, Y., Lou, J., & Chen, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.48 · Cited by 73
Interpreting Black Box Models via Hypothesis Testing
Burns, C., Thomason, J., & Tansey, W. (2020) · ACM-IMS Foundations of Data Science Conference (FODS) · doi:10.1145/3412815.3416889 · Cited by 51
Attending to Entities for Better Text Understanding
Cheng, P., & Erk, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6254 · Cited by 52
Context-Aware Answer Extraction in Question Answering
Seonwoo, Y., Kim, J., Ha, J.-W., & Oh, A. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.189 · Cited by 42
Pretraining model for biological sequence data
Song, B., Li, Z., Lin, X., Wang, J., Wang, T., & Fu, X. (2021) · Briefings in Functional Genomics · doi:10.1093/bfgp/elab025 · Cited by 39
“Some NLP pretraining models [68, 70, 89–91] are also transferredtobiologicalsequences.”
BackgroundOAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services
Liu, X., Yin, D., Zheng, J., Zhang, X., Zhang, P., Yang, H., Dong, Y., & Tang, J. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539210 · Cited by 58
“But for heterogeneous academic entities, we expect OAG-BERT to memorize them well and thus develop a span-aware entity masking strategy combining the advantages of both ERNIE [55] and SpanBERT [17].”
BackgroundMethodInfluentialLarge language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 48
“Pre ‐ trained models such as BioBERT [174] and Span-BERT [175] have achieved notable success in this domain.”
Self-citationSyntactic Structure Distillation Pretraining for Bidirectional Encoders
Kuncoro, A., Kong, L., Fried, D., Yogatama, D., Rimell, L., Dyer, C., & Blunsom, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00345 · Cited by 49
ExpBERT: Representation Engineering with Natural Language Explanations
Murty, S., Koh, P. W., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.190 · Cited by 40
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Li, Z., Wang, W., Dong, L., Wei, F., & Xu, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.600 · Cited by 45
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability
Chen, J., Shen, D., Chen, W., & Yang, D. (2021) · doi:10.18653/v1/2021.acl-long.338 · Cited by 54
ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 90
Coreference Resolution through a seq2seq Transition-Based System
Bohnet, B., Alberti, C., & Collins, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00543 · Cited by 90
E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commercePreprint
Zhang, D., Yuan, Z., Liu, Y., Zhuang, F., Chen, H., & Xiong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02835 · Cited by 55
“epresent words based on the entire input context to deal with word polysemy, thus captures semantics more accurately. Following PLMs, many endeavors have been made for further optimization. SpanBERT (Joshi et al. 2020) proposes to reconstruct randomly masked spans instead of single words. However, the span consists of…”
BackgroundSelf-citationWang, Y., Chen, M., Zhou, W., Cai, Y., Liang, Y., Liu, D., Yang, B., Liu, J., & Hooi, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.224 · Cited by 63
Spike Transformer: Monocular Depth Estimation for Spiking Camera
Zhang, J., Tang, L., Yu, Z., Lu, J., & Huang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20071-7_3 · Cited by 66
“Following the practice of Transformers in NLP and image-based tasks, we term the feature of those tokens as spike embeddings, which thus can be received as inputs to Transformer.”
BackgroundMethodInfluentialChai, Y., Xie, H., & Qin, J. S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11405-5 · Cited by 67
GT-Finder: Classify the family of glucose transporters with pre-trained BERT language models
Shah, S. M. A., Taju, S. W., Ho, Q., Nguyen, T., & Ou, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104259 · Cited by 37
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
Li, S., Li, X., Shang, L., Dong, Z., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.136 · Cited by 30
EventBERT: A Pre-Trained Model for Event Correlation Reasoning
Zhou, Y., Geng, X., Shen, T., Long, G., & Jiang, D. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511928 · Cited by 108
Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning
Chen, X., Li, L., Zhang, N., Tan, C.-Q., Huang, F., Si, L., & Chen, H.-J. (2022) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531746 · Cited by 52
OCR-IDL: OCR Annotations for Industry Document Library Dataset
Biten, A. F., Tito, R., Gómez, L., Valveny, E., & Karatzas, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25069-9_16 · Cited by 60
Dahou, A., Mabrouk, A., Ewees, A. A., Gaheen, M. A., & Elaziz, M. A. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122546 · Cited by 34
Autoregressive Structured Prediction with Language Models
Liu, T., Jiang, Y. E., Monath, N., Cotterell, R., & Sachan, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.70 · Cited by 87
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Wei, C., Xie, S. M., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2106.09226 · Cited by 32
“Other papers ablate various aspects of pretraining, such as changing the masking scheme [14, 20, 40] or permuting the word order [32].”
Gamage, G., Silva, D. D., Mills, N., Alahakoon, D., & Manic, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00953-2 · Cited by 46
“Module 1 begins with a state-of-the-art language model, such as BERT [65] which has been effectively applied on diverse NLP tasks such as Reading Comprehension [66, 67] and Natural Language Inference [68, 69].”
MethodProbing Linguistic Features of Sentence-Level Representations in Relation Extraction
Alt, C., Gabryszak, A., & Hennig, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.140 · Cited by 46
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation
Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021) · doi:10.18653/v1/2021.acl-long.259 · Cited by 61
“To represent OOV words in multilingual settings, the mixture mapping method (Wang et al., 2019) utilized a mixture of English subwords embedding, but it has been shown useless for domain-specific words by Tai et al. (2020).”
A Literature Review on Bidirectional Encoder Representations from Transformers
Shreyashree, S., Sunagar, P., Rajarajeswari, S., & Kanavalli, A. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-6723-7_23 · Cited by 48
Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding
Wang, J., Huang, W., Qiu, M., Shi, Q., Wang, H., Li, X., & Gao, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.207 · Cited by 52
EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Lin, C., Miller, T., Dligach, D., Bethard, S. J., & Savova, G. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.21 · Cited by 55
“We also combined entity-centric masking with Span-BERT (Joshi et al., 2020) trained the model on MIMIC-SMALL with different random seeds.”
BackgroundMethodInfluentialRecent Advances of Computational Intelligence Techniques for Composing Music
Wen, Y.-W., & Ting, C.-K. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3221126 · Cited by 37
“Muhamed et al. [178] introduced the SpanBERT pretraining technique [224] to strengthen the transformer-based discriminator.”
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models
Zhong, Q., Ding, L., Shen, L., Mi, P., Liu, J., Du, B., & Tao, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.300 · Cited by 63
EEG2Text: Open Vocabulary EEG-to-Text Translation with Multi-View Transformer
Liu, H., Hajialigol, D., Antony, B., Han, A., & Wang, X. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825980 · Cited by 65
Wei, W., Wang, Z. K., Pang, B., Wang, J., & Liu, X. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3525502 · Cited by 41
“Transformer model [5] incorporates self-attention and has proven to be successful in capturing long-term relationships and interactions in sequential data, particularly in natural language [6], [7] and video processing [8], [9].”
A Cross-Task Analysis of Text Span Representations
Toshniwal, S., Shi, H., Shi, B., Gao, L., Livescu, K., & Gimpel, K. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.20 · Cited by 45
Approximate selection with guarantees using proxies
Kang, D. D., Gan, E., Bailis, P. D., Hashimoto, T., & Zaharia, M. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3407790.3407804 · Cited by 67
“odel is human labeling for both datasets. The proxy model for OntoNotes is a baseline provided by with the fine-grained entities [18]. The proxy model for TACRED is the state-of-the-art SpanBERT model [42]. We choose different models to demonstrate that SUPG is agnostic to proxy model choice. 6.2 Baseline Methods Fail…”
MethodInfluentialIncremental Neural Coreference Resolution in Constant Memory
Xia, P., Sedoc, J., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.695 · Cited by 53
“Implementation We use the model dimensions and training hyperparameters from the baseline model, a publicly available coreference resolution model by Joshi et al. (2019, 2020).”
BackgroundMethodInfluentialLeng, X.-L., Miao, X.-A., & Liu, T. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10336-3 · Cited by 31
Euphemistic Phrase Detection by Masked Language Model
Zhu, W., & Bhat, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.16 · Cited by 34
On Generalization in Coreference Resolution
Toshniwal, S., Xia, P., Wiseman, S., Livescu, K., & Gimpel, K. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.12 · Cited by 57
REFinD: Relation Extraction Financial Dataset
Kaur, S., Smiley, C., Gupta, A. K., Sain, J. P., Wang, D., Siddagangappa, S., Aguda, T., & Shah, S. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591911 · Cited by 67
“FLANG-BERT (FB) [15], (iv) SpanBERT-base (SB) [10], (v) SpanBERT-large (SL) [10], (vi) FLANG-SpanBERT (FS) [15], (vii)”
MethodCheng, P., Yang, X., Chen, A., Yu, Z., Smith, K. E., Costa, A. B., Flores, M. G., Bian, J., & Wu, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae078 · Cited by 37
Transformers on Sarcasm Detection with Context
Kumar, A., & Anand, V. (2020) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2020.figlang-1.13 · Cited by 30
“Joshi et al. (2020) introduced pretraining method to represent and predict span instead of words.”
MethodExternal features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 35
Lai, T., Ji, H., Bui, T., Tran, Q. H., Dernoncourt, F., & Chang, W. H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.274 · Cited by 42
ZoKa: a fake news detection method using edge-weighted graph attention network with transfer models
İnan, E. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07057-z · Cited by 34
“Further, neural language models including BERT [4], ALBERT [14] and RoBERTa [17], TinyBERT [9], DistillBERT and SpanBERT [10] are compared.”
MethodStreamlining Cross-Document Coreference Resolution: Evaluation and ModelingPreprint
Cattan, A., Eirew, A., Stanovsky, G., Joshi, M., & Dagan, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11032 · Cited by 40
“In this paper, we observe that research on CD coreference has been lagging behind the impressive strides made in WD coreference (Lee et al., 2017, 2018; Joshi et al., 2019, 2020), achieving 83.5 F1 (Wu et al., 2019).”
InfluentialSelf-citationRelation Classification as Two-way Span-PredictionPreprint
Cohen, A., Rosenman, S., & Goldberg, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04829 · Cited by 44
“, 2019) and (Joshi et al., 2020) that showed inferior results compared to ALBERT The same paper reports an additional results based on external training data, which is not comparable.”
MethodResultInfluentialProceedings of the Third Workshop on Narrative Understanding
(2021) · doi:10.18653/v1/2021.nuse-1 · Cited by 31
Language Modelling with PixelsPreprint
Rust, P., Lotz, J. F., Bugliarello, E., Salesky, E., de Lhoneux, M., & Elliott, D. (2022) · Lirias · doi:10.48550/arxiv.2207.06991 · Cited by 74
“The idea behind the span masking approach, inspired by T5 (Raffel et al., 2020) and SpanBERT (Joshi et al., 2020a), is that it masks more meaningful units of text (full words or phrases) than random masking where the model more often has to fill in (parts of) individual characters, thereby…”
BackgroundCross-document Coreference Resolution over Predicted Mentions
Cattan, A., Eirew, A., Stanovsky, G., Joshi, M., & Dagan, I. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.453 · Cited by 29
Self-citationGenHPF: General Healthcare Predictive Framework for Multi-Task Multi-Source Learning
Hur, K., Oh, J., Kim, J., Kim, J., Lee, M. J., Cho, E., Moon, S.-E., Kim, Y., Atallah, L., & Choi, E. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3327951 · Cited by 66
“MLM and SpanMLM [28], [39]: For MLM pretraining, we randomly mask a fixed ratio of tokens among the whole patient event history, and the pretraining objective is to predict the masked tokens based on bidirectional attention.”
MethodLarge-Language-Model-Enabled Health Management for Internet of Batteries in Electric Vehicles
Peng, H., Liu, C., & Li, H. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3513002 · Cited by 36
“Building upon the success of BERT, several enhanced models have been developed, such as RoBERTa, ALBERT, DistilBERT, and SpanBERT [22], [23].”
On Losses for Modern Language Models
Aroca-Ouellette, S., & Rudzicz, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.403 · Cited by 40
Ma, Y., Hiraoka, T., & Okazaki, N. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.187 · Cited by 30
LOME: Large Ontology Multilingual Extraction
Xia, P., Qin, G., Vashishtha, S., Chen, Y., Chen, T., May, C., Harman, C., Rawlins, K., White, A. S., & Durme, B. V. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.19 · Cited by 65
Rodrigues, F. B., Giozza, W. F., Albuquerque, R. D. O., & Villalba, L. J. G. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3159677 · Cited by 43
“SpanBERT [32] is a pretrained language model based on the transformer.”
BackgroundNatural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 53
“, Phrase-BERT (PBERT) [39], which has been demonstrated to be more effective than other BERT models in phrase-level semantics representation [18, 31].”
MethodKEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 43
Nonparametric masked language modeling
Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-T., Hajishirzi, H., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.132 · Cited by 16
Self-citationGenerative Prompt Tuning for Relation Classification
Han, J., Zhao, S., Cheng, B. H., Ma, S., & Lü, W. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.231 · Cited by 44
CapWAP: Image Captioning with a Purpose
Fisch, A., Lee, K., Chang, M.-W., Clark, J. H., & Barzilay, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.705 · Cited by 39
Mathematical Reasoning via Self-supervised Skip-tree Training
Rabe, M. N., Lee, D. M., Bansal, K., & Szegedy, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04757 · Cited by 24
“SpanBERT additionally considers a span boundary objective, which is to predict the masked out subsequence only from the tokens adjacent to the missing subsequence [Joshi et al., 2020].”
BackgroundTowards employing native information in citation function classification
Zhang, Y., Zhao, R., Wang, Y., Chen, H., Mahmood, A., Zaib, M., Zhang, W. E., & Sheng, Q. Z. (2022) · Scientometrics · doi:10.1007/s11192-021-04242-0 · Cited by 28
“Joshi et al. (2020) proposed SpanBERT, which via changing the mask objective in BERT and obtain a better performance in different NLP tasks.”
BackgroundINSET: Sentence Infilling with INter-SEntential Transformer
Huang, Y., Zhang, Y., Elachqar, O., & Cheng, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.226 · Cited by 38
“Other related works (Liu et al., 2019; Joshi et al., 2020) also require knowledge of the span length as an input to their models, and thus are different from our work.”
BackgroundResultInfluential“Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding
Brahman, F., Huang, M., Tafjord, O., Zhao, C., Sachan, M., & Chaturvedi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.150 · Cited by 82
Entity-aware Transformers for Entity Search
Gerritse, E. J., Hasibi, F., & de Vries, A. P. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531971 · Cited by 48
“[25] Fabio Petroni, Tim Rocktäschel, Patrick S.”
BackgroundInfluentialReinforcement learning-driven deep question generation with rich semantics
Guan, M., Mondal, S. K., Dai, H., & Bao, H. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103232 · Cited by 28
E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3341917 · Cited by 39
“0 [27], XLNet [28], SpanBERT [26], SemBERT [59], etc. Note that the models with larger capabilities could always achieve higher performance, and thus we only compare our E2S2-BART models with the pretrained models of comparable size for a fair comparison.”
MethodResultAutomated Smell Detection and Recommendation in Natural Language Requirements
Veizaga, A., Shin, S. Y., & Briand, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3361033 · Cited by 31
“These solutions incorporate both traditional and state-of-the-art NLP and ML techniques, such as SpanBERT [45].”
MethodA T5-based interpretable reading comprehension model with more accurate evidence training
Guan, B., Zhu, X., & Yuan, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103584 · Cited by 31
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.