SpanBERT: Improving Pre-training by Representing and Predicting Spans
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT large , our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“XLM [46] performs MLM on a concatenation of parallel bilingual sentence pairs, called Translation Language Modeling (TLM).”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredIn the citing paper
“Conneau and Lample [46] tried to initialize the entire encoder and decoder by a multilingual pre-trained BERT model and showed a significant improvement could be achieved on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
In the citing paper
“In the experiment, this approach can surpass the BERT-style pretraining proposed by Conneau and Lample [46] both on unsupervised MT and English-Romanian supervised MT.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 4×
“…Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.”
BackgroundInfluentialXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 4×
“While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a significant impact.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“, 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019).”
Method
Research works printed but not linked: 7
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Andrew M Dai and Quoc V Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) , pages 3079-3087.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP .
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 . · arXiv
- Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, and Luke Zettlemoyer. 2019a. pair2vec: Compositional word-pair embeddings for crosssentence inference. In North American Association for Computational Linguistics (NAACL) , pages 3597-3608.
- Liu Yinhan, 2019, arxiv preprint arXiv:1907.11692 · arXiv
- Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems (NIPS) .
Cited by
Items 701–800 of 1,012
Tackling Zero Pronoun Resolution and Non-Zero Coreference Resolution Jointly
Chen, S., Gu, B., Qu, J., Li, Z., Liu, A., Zhao, L., & Chen, Z. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.40 · Cited by 17
“Instead of using Word2vec and Elmo, Joshi et al. (2019, 2020) apply BERT to get better span representations.”
MethodiFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration
Hirsch, E., Eirew, A., Shapira, O., Caciularu, A., Cattan, A., Ernst, O., Pasunuru, R., Ronen, H., Bansal, M., & Dagan, I. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.33 · Cited by 14
“(1) We use SpanBERT 6 (Joshi et al., 2020), a state-of-the-art transformer-based LM for WD entity coreference resolution, to detect and cluster coreferring entity mentions within each separate document.”
BackgroundMethodHierarchical Context Tagging for Utterance Rewriting
Jin, L. M., Song, L., Jin, L., Yu, D., & Gildea, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21331 · Cited by 19
“Pretraining methods like SpanBERT (Joshi et al. 2020) refine such span representations.”
MethodYe, Q., Yang, R., Cheng, C.-L., Peng, L., & Lan, Y. (2023) · Evidence-Based Complementary and Alternative Medicine · doi:10.1155/2023/2088698 · Cited by 9
“For example, RoBERTa [6], XLNET [7], SpanBERT [8], and ERNIE [9] are trained in the abovementioned way.”
PoetryBERT: Pre-training with Sememe Knowledge for Classical Chinese Poetry
Zhao, J., Bai, T., Wei, Y., & Wu, B. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-8991-9_26 · Cited by 15
Khalid, H., Murtaza, G., & Abbas, Q. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3595861 · Cited by 16
“Additionally, there are a few other variants of the BERT family, such as ALBERT [29], SpanBERT [33], and TinyBERT [34].”
MethodLearning Chemical Rules of Retrosynthesis with Pre-training
Jiang, Y., Wei, Y., Wu, F. L., Huang, Z., Kuang, K., & Wang, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25640 · Cited by 11
“Concretely, we use the span-mask (Joshi et al. 2020) to generate masks in lengths of [1 , 10] for each molecule.”
MethodThe tail wagging the dog: Dataset construction biases of social bias benchmarks
Selvam, N., Dev, S., Khashabi, D., Khot, T., & Chang, K.-W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.118 · Cited by 11
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y. W., & Gui, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.35 · Cited by 21
Li, H., Wu, Y., Schlegel, V., Batista-Navarro, R., Nguyen, T.-T., Kashyap, A. R., Zeng, X., Beck, D., WINKLER, S. S., & Nenadić, G. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.49 · Cited by 26
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking
Lee, M., Park, J.-H., Kim, J., Kim, K.-M., & Lee, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.502 · Cited by 12
Global-View and Speaker-Aware Emotion Cause Extraction in Conversations
An, J., Ding, Z., Li, K., & Xia, R. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3319990 · Cited by 11
Modeling Context in Answer Sentence Selection Systems on a Latency Budget
Han, R., Soldaini, L., & Moschitti, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.261 · Cited by 14
Stay Together: A System for Single and Split-antecedent Anaphora Resolution
Yu, J., Moosavi, N. S., Paun, S., & Poesio, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.329 · Cited by 16
Linguistically Driven Multi-Task Pre-Training for Low-Resource Neural Machine Translation
Mao, Z., Chu, C., & Kurohashi, S. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3491065 · Cited by 12
Predicting Human Psychometric Properties Using Computational Language Models
Laverghetta, A., Nighojkar, A., Mirzakhalov, J., & Licato, J. (2022) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-04572-1_12 · Cited by 16
Salient Span Masking for Temporal Understanding
Cole, J. R., Chaudhary, A., Dhingra, B., & Talukdar, P. P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.222 · Cited by 23
CopyBERT: A Unified Approach to Question Generation with Self-Attention
Varanasi, S., Amin, S., & Neumann, G. (2020) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2020.nlp4convai-1.3 · Cited by 19
“…the self-copy mechanism, we also experimented by initializing with a variant of BERT 4 called SpanBERT (Joshi et al., 2019), which is pre-trained to predict longer masked spans to encourage better entity masking and has already shown to improve QA results when compared to BERT (Joshi et al., 2019).”
MethodInfluentialInterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language InteractionPreprint
Lin, J., An, Y., Zhang, Y., Liu, J., Zhou, J., & Yang, H. (2020) · arXiv (Cornell University) · Cited by 11
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben-Zaken, E., Ravfogel, S., & Goldberg, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.10199 · Cited by 11
Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER
Zeng, Q., Yu, W., Yu, M., Jiang, T., Weninger, T., & Jiang, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.429 · Cited by 10
“Moreover, the pre-training aimed at predicting random masked tokens or tokens in random masked spans (Joshi et al., 2020), which could be too weakly associated with the target task of entity recognition and typing.”
BackgroundMethodInfluentialGOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Zadeh, A. H., Edo, I., Awad, O. M., & Moshovos, A. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00071 · Cited by 10
“Table VIII evaluates GOBO SpanBERT [12] on the SQuADv1.”
MethodRambelli, G., Chersoni, E., Lenci, A., Blache, P., & Huang, C. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.26 · Cited by 11
“We 5 Notice that the usefulness of this secondary objective function was questioned, and it was indeed removed in more recent models (Yang et al., 2019; Liu et al., 2019; Joshi et al., 2020). used the XL version of GPT-2 for our experiments.”
BackgroundMethodJin-fen, L., & Xiao, L. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.237 · Cited by 10
Domain-oriented Language Modeling with Adaptive Hybrid Masking and Optimal Transport Alignment
Zhang, D., Yuan, Z., Liu, Y., Liu, H., Zhuang, F., Xiong, H., & Chen, H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467215 · Cited by 10
“(3) The effects of different masking schemes: BERT-NP and SpanBERT can perform better consistently than BERT-PT, indicating the advantage of phrase/span based masking strategy over word based masking strategy.”
BackgroundInfluentialSelf-citationChen, J., Ma, M., Zheng, R., & Huang, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-733 · Cited by 10
Nandy, A., Sharma, S., Maddhashiya, S., Sachdeva, K., Goyal, P., & Ganguly, N. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.392 · Cited by 17
ANNA”:" Enhanced Language Representation for Question Answering
Jun, C., Jang, H., Sim, M., Kim, H. J., Choi, J., Min, K., & Bae, K. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.13 · Cited by 10
Biographical Semi-Supervised Relation Extraction Dataset
Plum, A., Ranasinghe, T., Jones, S., Orǎsan, C., & Mitkov, R. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531742 · Cited by 14
A Comparison of Transformer-Based Language Models on NLP Benchmarks
Greco, C. M., Tagarelli, A., & Zumpano, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_45 · Cited by 11
Ask2Mask: Guided Data Selection for Masked Speech Modeling
Baskar, M. K., Rosenberg, A., Ramabhadran, B., Zhang, Y., & Moreno, P. J. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3186162 · Cited by 12
“The importance of block masking is motivated by the improvements observed in Span-BERT by [21] and ERNIE [12].”
MethodCOVID-19 Claim Radar: A Structured Claim Extraction and Tracking System
Li, M., Reddy, R. G., Wang, Z., Chiang, Y.-S., Lai, T., Yu, P., Zhang, Z., & Ji, H. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.13 · Cited by 12
Graph Refinement for Coreference Resolution
Miculicich, L., & Henderson, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.215 · Cited by 22
DocTime: A Document-level Temporal Dependency Graph Parser
Mathur, P., Morariu, V. I., Kaynig, V., Gu, J., Dernoncourt, F., Tran, Q., Nenkova, A., Manocha, D., & Jain, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.73 · Cited by 23
Link of Transformers in CV and NLP: A Brief Survey
Jia, J., Chen, X., Yang, A., He, Q., Dai, P., & Liu, M. (2022) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai55851.2022.9904151 · Cited by 13
“Joshi et al.[11] use a different approach from the BERT masking scheme, replacing random tokens with masking contiguous random spans and it trains the span boundary representations to predict entire masked spans, without using the token representations within it.”
MethodGanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Yang, J. Y., Ma, S., Dong, L., Huang, S., Huang, H., Yin, Y., Zhang, D., Yang, L., Wei, F., & Li, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.522 · Cited by 27
Wang, Z., Liu, X., & Zhang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.434 · Cited by 17
Fan, Y., Li, B., Sataer, Y., Gao, M., Shi, C., Cao, S., & Gao, Z. (2023) · Applied Sciences · doi:10.3390/app13169412 · Cited by 12
“…employed the bottom-up and top-down discourse parsers in [18] as our clause-parsing models and conducted experimental trials of the base version of different pretrained language models (PLMs, i.e., BERT [39], RoBERTa [40], SpanBERT [41], XLNet [42], and DeBERTa [43]) to obtain better performances.”
MethodWang, J., Cui, W., Zhu, T., Ning, H., & Liu, Z. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3390755 · Cited by 13
“Building upon this foundation, The work [26] introduces the idea of Span Masking [37], which masks continuous segments of sensor data along the time axis and then reconstructing these masked segments.”
MethodContinual Domain Adaptation for Machine Reading Comprehension
Su, L., Guo, J., Zhang, R., Fan, Y., Lan, Y., & Cheng, X. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412047 · Cited by 13
A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News
Poot, C., & van Cranenburgh, A. (2020) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.48550/arxiv.2011.01615 · Cited by 25
“Joshi et al. (2019) report a similar document length effect for English with their end-to-end model.”
BackgroundMethodPENELOPIE: Enabling Open Information Extraction for the Greek Language through Machine Translation
Papadopoulos, D., Papadakis, N. K., & Matsatsinis, N. F. (2021) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2021.eacl-srw.4 · Cited by 15
Generalizing Discriminative Retrieval Models using Generative Tasks
Liu, B., Zamani, H., Lu, X., & Culpepper, J. S. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449863 · Cited by 12
“[20] proposed spanBERT which extends BERT by masking contiguous random spans.”
BackgroundOn the Influence of Masking Policies in Intermediate Pre-training
Ye, Q., Li, B. Z., Wang, S., Bolte, B., Ma, H., Yih, W.-T., Ren, X., & Khabsa, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.573 · Cited by 15
“, 2019), span boundary representation learning (Joshi et al., 2020), corrupted sentence reconstruction (Lewis et al.”
BackgroundRobust Question Answering Through Sub-part Alignment
Chen, J., & Durrett, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.98 · Cited by 15
TopNet: Learning from Neural Topic Model to Generate Long Stories
Yang, Y., Pan, B., Cai, D., & Sun, H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467410 · Cited by 13
“With little input information, the neural model degenerates to language model [8, 11, 21, 39] that generates a story without taking the input into consideration.”
BackgroundWang, G., Liu, S., & Wei, F. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02596-9 · Cited by 11
Tracing Origins: Coreference-aware Machine Reading Comprehension
Huang, B., Zhang, Z., & Zhao, H. Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.91 · Cited by 13
CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations
Yu, C., Zhang, H., Song, Y., & Ng, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.93 · Cited by 28
Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models
Arora, S., Wu, S., Liu, E., & Ré, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.137 · Cited by 10
“Model modifications include introducing continuous vector representations for entities or auxiliary objectives (Zhang et al., 2019; Peters et al., 2019; Ya-mada et al., 2020; Wang et al., 2020; Xiong et al., 2020; Joshi et al., 2020a; Su et al., 2021).”
BackgroundMicroblog Retrieval Based on Concept-Enhanced Pre-Training Model
Wang, Y., Wang, Z., Zhang, H., & Liu, Z. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3552311 · Cited by 13
DocAMR: Multi-Sentence AMR Representation and Evaluation
Naseem, T., Blodgett, A., Kumaravel, S., O’Gorman, T., Lee, Y., Flanigan, J., Astudillo, R. F., Florian, R., Roukos, S., & Schneider, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.256 · Cited by 12
Efficient long-distance relation extraction with DG-SpanBERTPreprint
Chen, J., Hoehndorf, R., Elhoseiny, M., & Zhang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03636 · Cited by 11
“Our work is inspired by the superior performance of the SpanBERT model [8] and the capability of a dependency tree on grouping long clauses together and extracting the grammar structure of the sentence [5].”
BackgroundMethodInfluentialLess is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder.Preprint
Lu, S., Xiong, C., He, D., Ke, G., Malik, W., Dou, Z., Bennett, P. N., Liu, T., & Overwijk, A. (2021) · arXiv (Cornell University) · Cited by 10
Attacking Open-domain Question Answering by Injecting Misinformation
Pan, L., Chen, W., Kan, M.-Y., & Wang, W. Y. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.35 · Cited by 9
“…we consider five state-of-the-art QA models with public code that achieved strong results on the public leader board of SQuAD: RoBERTa-large (Liu et al., 2019), Span-BERT (Joshi et al., 2020), Longformer (Beltagy et al., 2020), ELEC-TRA (Clark et al., 2020), and DeBERTa-V3 (He et al., 2023).”
BackgroundNishida, N., & Matsumoto, Y. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00451 · Cited by 11
“We used AdamW and Adam optimizers for optimizing the trans-former’s parameters ( θ bert ) and the task-specific parameters ( θ task ), respectively, following Joshi et al. (2019).”
BackgroundMethodInfluentialWeng, K.-H., Liu, C., & Chen, C. (2023) · JMIR Medical Informatics · doi:10.2196/46348 · Cited by 10
“SpanBERT [36] extends BERT with span-based masking and an additional training objective, resulting in a better performance on span-based tasks.”
BackgroundSemantics of Multiword Expressions in Transformer-Based Models: A Survey
Miletić, F., & Walde, S. S. I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00657 · Cited by 31
“SpanBERT is pretrained by masking contiguous spans of text instead of individual tokens, and uses span boundary representations to encode span content ( Joshi et al., 2020).”
MethodEnhanced Large Language Models
Jeny, J. R. V., Babu, G. B., Reddy, S. J., Muthukumaran, N., & Abhiram, K. (2024) · International Conference on Inventive Computation Technologies (ICICT) · doi:10.1109/icict60155.2024.10544411 · Cited by 9
Improving biomedical Named Entity Recognition with additional external contexts
Tho, B. D., Nguyen, M.-T., Le, D. T., Ying, L.-L., Inoue, S., & Nguyen, T.-T. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104674 · Cited by 12
Set-pMAE: spatial-spEctral-temporal based parallel masked autoEncoder for EEG emotion recognition
Pan, C., Lu, H., Lin, C., Zhong, Z., & Liu, B. (2024) · Cognitive Neurodynamics · doi:10.1007/s11571-024-10162-5 · Cited by 10
“SpanBERT (Joshi et al. 2020) proposes to mask successive ranges instead of random individual words.”
Relevance-aware prompt-tuning method for multimodal social entity and relation extraction
Chen, Z., Li, Z., Liu, M., Zhang, C., & Ma, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130316 · Cited by 9
Design Patterns for Large Language Model Based Neuro-Symbolic Systems
de Boer, M., Smit, Q. T. S., van Bekkum, M. A., Meyer-Vitali, A., & Schmid, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251377499 · Cited by 18
Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts
Han, R., Chen, H., Tian, Y., & Peng, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.104 · Cited by 32
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
Huber, P., & Carenini, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.170 · Cited by 24
Transfer Learning for Low-Resource Multilingual Relation Classification
Nag, A., Samanta, B., Mukherjee, A., Ganguly, N., & Chakrabarti, S. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3554734 · Cited by 16
“Existing works Model architecture Dataset language [17, 43, 55, 67, 76, 89] Convolution Neural Network English [42, 50, 80, 92, 93, 97] Recurrent Neural Network English [12, 46, 47, 81–83] Dependency based model English [28, 31, 32, 45, 71, 74, 86, 88, 90] Distance supervision based English [5, 33, 84] Graph Neural Ne…”
MethodZhou, C., Che, C., Zhang, X. S., Zhang, Q., & Zhou, D. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109836 · Cited by 17
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Detecting and Grounding Important Characters in Visual Stories
Liu, D., & Keller, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26551 · Cited by 11
QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search
Xie, J., Liang, Y., Liu, J., Xiao, Y., Wu, B., & Ni, S. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599891 · Cited by 12
“Likewise, SpanBERT [11] no longer masks individual tokens but continuous spans to enhance the model’s text comprehension.”
BackgroundDo Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 33
“…(Chen et al., 2017), QANet (Yu et al., 2018), BiDAF++ (Clark and Gardner, 2018), MnemonicReader (Hu et al., 2017), FusionNet (Huang et al., 2018), BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), RoBERTa (Liu et al., 2019), ELECTRA (Clark et al., 2020), and SpanBERT (Joshi et al., 2020).”
BackgroundMethodLearning Event-aware Measures for Event Coreference Resolution
Yao, Y., Li, Z., & Zhao, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.855 · Cited by 17
“For other event features, the authors trained a simple classification model based on SpanBERT (Joshi et al., 2020).”
MethodExplainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling
Gunaratna, K., Srinivasan, V., Yerukola, A., & Jin, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.245 · Cited by 11
Why Can’t Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity
Liu, Y. J., & Zeldes, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.227 · Cited by 10
Self-citationCrema, C., Buonocore, T. M., Fostinelli, S., Parimbelli, E., Verde, F., Fundarò, C., Manera, M. R., Ramusino, M. C., Capelli, M., Costa, A. C., et al. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104557 · Cited by 18
“Li et al. [21] compared four biomedical BERT models (BioBERT, SciBERT, BlueBERT [22], PubMedBERT [23]) and two open-domain models (BERT and SpanBERT [24]) by fine-tuning them on three clinical datasets, showing that the domain-specific models outperformed the open-domain ones with the best model…”
MethodOptimizing Large Language Models on Multi-Core CPUs: A Case Study of the BERT Model
Zhao, L., Gao, W., & Fang, J. (2024) · Applied Sciences · doi:10.3390/app14062364 · Cited by 13
Semantic Encoding Algorithm for Classification and Retrieval of Aviation Safety Reports
Gao, Y., Zhu, G., Duan, Y., & Mao, J. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3359356 · Cited by 9
“BERT WWM [38] and SpanBert [39] modify the masking scheme of BERT.”
A Detailed Study on Anaphora Resolution System for Asian Languages
Prajapati, P., Goyal, V., & Kaur, K. (2024) · SN Computer Science · doi:10.1007/s42979-024-03191-8 · Cited by 9
“For coreference resolution, SpanBERT, described by [150], represents an LLM demonstrating commendable performance when evaluated on the OneNotes corpus.”
Representation learning in discourse parsing: A survey
Song, W., & Liu, L. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1685-2 · Cited by 11
RELATION EXTRACTION DATASET FOR THE RUSSIAN
RANEPA, M., Gordeev, D., Davletov, A., RANEPA, M., Rey, A., RANEPA, M., Akzhigitova, G., RANEPA, M., Geymbukh, G. A., & RANEPA, M. (2020) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2020-19-348-360 · Cited by 9
“It correlates with annotation procedures by other researchers [6].”
BackgroundMethodTransfer learning for fine-grained entity typing
Hou, F., Wang, R., & Zhou, Y. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01549-5 · Cited by 9
An End-To-End Actor-Critic-Based Neural Coreference Resolution System
Wang, Y., Shen, Y., & Jin, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413579 · Cited by 13
Multilingual Sentiment Analysis and Toxicity Detection for Text Messages in Russian
Bogoradnikova, D., Makhnytkina, O. V., Matveev, A., Zakharova, A., & Akulov, A. (2021) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct52173.2021.9435584 · Cited by 11
“Among modern methods for general span detection, we can point out SpanBERT [19], a pre-training method that is designed to better represent and predict spans of text.”
MethodPre-training Universal Language Representation
Li, Y., & Zhao, H. (2021) · doi:10.18653/v1/2021.acl-long.398 · Cited by 12
“, 2019), StructBERT (Joshi et al., 2020), SpanBERT (Wang et al.”
BackgroundMethodInfluentialA Navigation System for Safe Routing
Kaur, R., Goyal, V., Gunturi, V. M. V., Saini, A., Sanadhya, K., Gupta, R., & Ratra, S. (2021) · IEEE International Conference on Mobile Data Management (MDM) · doi:10.1109/mdm52706.2021.00047 · Cited by 10
“To extract the location(s) from the text, we performed Named Entity Recognition using the SpanBERT model [12] trained on addresses of Delhi.”
Model Selection for Cross-lingual Transfer
Chen, Y., & Ritter, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.459 · Cited by 10
“There is a need for more NLP research on low-resource languages (Joshi et al., 2020b).”
Deep Clustering of Text Representations for Supervision-Free Probing of Syntax
Gupta, V., Shi, H., Gimpel, K., & Sachan, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21317 · Cited by 19
“As with POSI, our results suggest that mBERT outperforms both SpanBERT and E-BERT for the CoLab task as well.”
BackgroundMethodInfluentialNLIRE: A Natural Language Inference method for Relation Extraction
Hu, W., Liu, L., Sun, Y., Wu, Y., Liu, Z., Zhang, R., & Peng, T. (2021) · Journal of Web Semantics · doi:10.1016/j.websem.2021.100686 · Cited by 9
Representation Learning for Resource-Constrained Keyphrase Generation
Wu, D., Ahmad, W., Dev, S., & Chang, K.-W. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.49 · Cited by 23
TRAttack”:" Text Rewriting Attack Against Text Retrieval
Song, J., Zhang, J., Zhu, J., Tang, M., & Yang, Y. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.20 · Cited by 12
“Further, many variants based on BERT are proposed and achieve better performance, such as SpanBERT (Joshi et al., 2020), ERNIE (Sun et al., 2019; Xiao et al., 2020), etc.”
MethodRecognition of They/Them as Singular Personal Pronouns in Coreference Resolution
Baumler, C., & Rudinger, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.250 · Cited by 14
“While Span-BERT would have received similar exposure during pre-training, Span-BERT-base achieved a significantly lower accuracy than BERT-base on most WinoNB datasets.”
BackgroundMethodInfluentialLiu, M., Xiang, J., Xu, X., & Hu, H. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3551637 · Cited by 9
AMOM: Adaptive Masking over Masking for Conditional Masked Language Model
Xiao, Y., Xu, R., Wu, L., Li, J., Qin, T., Liu, T., & Zhang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26615 · Cited by 16
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models
Wijesiriwardene, T., Wickramarachchi, R., Gajera, B., Gowaikar, S., Gupta, C., Chadha, A. R., Reganti, A., Sheth, A., & Das, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.218 · Cited by 45
Integrating Merkle Trees with Transformer Networks for Secure Financial Computation
Wang, X., Lin, W., Zhang, W., Huang, Y., Li, Z., Liu, Q., Yang, X., Yao, Y., & Lv, C. (2024) · Applied Sciences · doi:10.3390/app14041386 · Cited by 9
Emergent Communication Pretraining for Few-Shot Machine Translation
Li, Y., Ponti, E. M., Vulić, I., & Korhonen, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.416 · Cited by 19
“…on masked language modeling or a similar objective, as proposed in the omnipresent BERT model (Devlin et al., 2019) and its extensions (Liu et al., 2019; Conneau and Lample, 2019; Song et al., 2019; Joshi et al., 2020), and then fine-tuning the model further on a downstream task (Wang et al., 2019).”
Facts2Story: Controlling Text Generation by Key Facts
Orbach, E., & Goldberg, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.211 · Cited by 19
Singh, P., Sandhu, S., Kumar, S., & Modi, A. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.231 · Cited by 10
Neural Semi-supervised Learning for Text Classification Under Large-Scale PretrainingPreprint
Sun, Z., Fan, C. C., Sun, X., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.08626 · Cited by 13
“ing (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lewis et al., 2019; Bao et al., 2020; Joshi et al., 2020) can also be viewed as a specific type of semi-supervised learning model, by leveraging the information of data in the general domain.”
BackgroundMethodInfluential
Metadata
- Authors
- Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O.
- Affiliations
- University of Washington (US)
- Princeton University (US)
- Meta (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 64-77
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2962753370
- Cited by
- 1,827
- References
- 51
- Citations by year
- 2026: 84 · 2025: 169 · 2024: 214 · 2023: 373 · 2022: 342 · 2021: 393 · 2020: 224
- FWCI
- 130.82
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- SpanBERT, BERT pre-training, span prediction, question answering, coreference resolution, relation extraction, text representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.