MASS: Masked Sequence to Sequence Pre-training for Language Generation
Open accessThe DOI matches its OpenAlex record; 20 of the 20 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“…language understanding tasks such as sentiment classification (Socher et al., 2013), natural language inference (Bowman et al., 2015), named entity recognition (Tjong Kim Sang & De Meulder, 2003) and SQuAD question answering (Rajpurkar et al., 2016), which usually have limited supervised data.”
BackgroundA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 1×
“…and features to the downstream tasks, which includes word-level representations (Brown et al., 1992; Ando & Zhang, 2005; Blitzer et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al., 2015; Logeswaran & Lee,…”
Background
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
“The largest improvements have come from pre-training on both source and target languages (Song et al., 2019; Lample & Conneau, 2019), but this requires pretraining on all languages of interest.”
BackgroundMethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10529 · Cited by 1,827
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
Multilingual Denoising Pre-training for Neural Machine Translation
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., & Zettlemoyer, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00343 · Cited by 1,527
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
“…groups: encoder-only models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), and ELECTRA (Clark et al., 2020), decoder-only models like GPT (Radford et al., 2019), and encoder-decoder models such as MASS (Song et al., 2019), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020).”
BackgroundMethodInfluentialUnified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206
“Transformer-based sequence models has led to a dramatic improvement in performance when finetuned for both natural language understanding and text generation tasks (Radford et al., 2018a; Devlin et al., 2019; Rothe et al., 2019; Yang et al., 2019; Joshi et al., 2019; Song et al., 2019; Dong et al., 2019; Lewis et al.,…”
BackgroundInfluentialOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
“More advanced pretrained text generators (Radford et al., 2018, 2019; Dong et al., 2019; Song et al., 2019; Khandelwal et al., 2019; Rothe et al., 2020) are even better at capturing world knowledge as they are informed by a vast amount of background text.”
BackgroundOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
“Specifically, the 6 -layer model of 768 hidden dimensions distilled from BERT BASE is 2 .”
BackgroundMethodInfluentialConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P., Ding, N., & Soricut, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00356 · Cited by 612
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
“Finally, there has been extensive research on modifying architectures [22, 59] and pre-training procedures [70, 36, 49, 60, 53, 14] for improving summarization performance.”
BackgroundMPNet: Masked and Permuted Pre-training for Language Understanding
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09297 · Cited by 514
Self-citationMask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · doi:10.18653/v1/w19-5301 · Cited by 461
Document Ranking with a Pretrained Sequence-to-Sequence Model
Nogueira, R., Jiang, Z., Pradeep, R., & Lin, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.63 · Cited by 448
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Rothe, S., Narayan, S., & Severyn, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00313 · Cited by 391
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 385
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 374
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
“This approach leverages recent progress in QA and QG to ask and answer human readable, on-topic questions (Devlin et al., 2019; Song et al., 2019).”
BackgroundNeural Machine Translation for Low-resource Languages: A Survey
Ranathunga, S., Lee, E.-S. A., Prifti Skenduli, M., Shekhar, R., Alam, M., & Kaur, R. (2023) · ACM Computing Surveys · doi:10.1145/3567592 · Cited by 313
A Unified Generative Framework for Various NER Subtasks
Yan, H., Gui, T., Dai, J., Guo, Q., Zhang, Z., & Qiu, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.451 · Cited by 299
“We mainly focus on the newly proposed BART (Lewis et al., 2020) model because it can achieve better performance than MASS (Song et al., 2019).”
BackgroundMethodA Unified Generative Framework for Aspect-based Sentiment Analysis
Yan, H., Dai, J., Ji, T., Qiu, X., & Zhang, Z. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.188 · Cited by 298
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279
“…(Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al., 2017; Artetxe et al., 2017).”
BackgroundA Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wang, W., Bao, H., Huang, S., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.188 · Cited by 257
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12804 · Cited by 254
Incorporating BERT into Neural Machine Translation
Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., & Liu, T.-Y. (2020) · arXiv · doi:10.48550/arxiv.2002.06823 · Cited by 228
Self-citationMachine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196
“such as MASS [104], T5 [57], and ULMFiT [105], can also be used for NLG tasks.”
InfluentialMachine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196
“such as MASS [104], T5 [57], and ULMFiT [105], can also be used for NLG tasks.”
InfluentialNaseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194
“MASS [149] is another recently proposed LM.”
Automatic Detection of Machine Generated Text: A Critical Survey
Jawahar, G., Abdul-Mageed, M., & Lakshmanan, V. S. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.208 · Cited by 156
Tracing the evolution of AI in the past decade and forecasting the emerging trends
Shao, Z., Zhao, R., Yuan, S., Ding, M., & Wang, Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118221 · Cited by 153
Applications of natural language processing and large language models in materials discovery
Jiang, X., Wang, W., Tian, S., Wang, H., Lookman, T., & Su, Y. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01554-0 · Cited by 145
“Forexample,themaskedlanguagemodel(MLM)isa self-supervisedtaskthatattemptstopredictthemaskedwordsinasentence giventherestofthewords 67 .”
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84
“Self-supervised pre-training has achieved great success in a wide range of natural language understanding (Devlin et al., 2018; Liu et al., 2019; Wang et al., 2019; Lan et al., 2019) and generation tasks (Song et al., 2019; Lewis et al., 2019; Bi et al., 2020).”
BackgroundUnsupervised translation of programming languages
Lachaux, M.-A., Roziere, B., Chanussot, L., & Lample, G. (2020) · arXiv · doi:10.48550/arxiv.2006.03511 · Cited by 83
“Subsequent work showed that pretraining the entire model (and not only word representations) in a cross-lingual way could lead to significant improvements in unsupervised machine translation [29, 33, 43].”
BackgroundA Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
OpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2111.01998 · Cited by 60
“The PromptTrainer is a controller that controls the data flow and the training process with some unique attributes, users can also implement the training process in a conventional fashion. include T5 (Raffel et al., 2020), MASS (Song et al., 2019) and BART (Lewis et al., 2020), etc.”
BackgroundEvaluating the Evaluation of Diversity in Natural Language Generation
Tevet, G., & Berant, J. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.25 · Cited by 52
(2020) · doi:10.18653/v1/2020.findings-emnlp.190 · Cited by 51
DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models
Saito, I., Nishida, K., Nishida, K., & Tomita, J. (2020) · arXiv · doi:10.48550/arxiv.2003.13028 · Cited by 30
Automatic Product Copywriting for E-commerce
Zhang, X., Zou, Y., Zhang, H., Zhou, J., Diao, S., Chen, J., Ding, Z., He, Z., He, X., Xiao, Y., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21508 · Cited by 24
Proof Artifact Co-training for Theorem Proving with Language Models
Han, J. M., Rute, J., Wu, Y., Ayers, E. W., & Polu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.06203 · Cited by 21
“Self-supervised learning methods Recently, selfsupervised methods have revolutionized machine learning models across various domains, such as natural language understanding (Logeswaran & Lee, 2018; Radford et al., 2018; Devlin et al., 2019; Song et al., 2019; Dong et al., 2019; Raffel et al., 2020; Conneau & Lample, 2…”
MethodReinforcing Pretrained Models for Generating Attractive Text Advertisements
Wang, X., Gu, X., Cao, J., Zhao, Z., Yan, Y., Middha, B., & Xie, X. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467105 · Cited by 20
“, MASS [31] and T5 [26], to better encode input sequence during sequence generation.”
BackgroundThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
“the masked tokens autoregressively (Raffel et al., 2020; Song et al., 2019).”
MethodPretrained Language Models for Text Generation: A Survey
Li, J., Tang, T., Zhao, W. X., & Wen, J.-R. (2021) · arXiv · doi:10.48550/arxiv.2105.10311 · Cited by 15
CHASE: Commonsense-Enriched Advertising on Search Engine with Explicit Knowledge
Zhang, C., Zhou, J., Zang, X., Xu, Q., Yin, L., He, X., Liu, L., Xiong, H., & Dou, D. (2021) · Proceedings of the 30th ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3481902 · Cited by 7
Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model
Zhang, Y., Jiang, J., Xia, G., & Dixon, S. (2022) · arXiv · doi:10.48550/arxiv.2208.11671 · Cited by 2
Metadata
- Authors
- Song, K., Tan, X., Qin, T., Lu, J., & Liu, T.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2019-05-07
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.02450
- OpenAlex
- W2944815030
- Cited by
- 733
- References
- 20
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.