The DOI matches its DataCite record; 60 of the 60 references it cites were found in a registry or an index.
Unified Language Model Pre-training for Natural Language Understanding and Generation
Open accessAbstract
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“A difference is that UniLM predictions are conditionally independent, whereas BART's are autoregressive.”
From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Dong et al., 2019
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This integration leverages the linguistic capabilities of Llama 3, utilizing both its textual output and intermediate embeddings at the token rate (Devlin et al., 2018; Dong et al., 2019; Raffel et al., 2020; Guo et al., 2023) to enhance the prediction of prosody features, thus reducing the lookahead required by the model.”
From The Llama 3 Herd of Models (2024) · cites this paper as Dong et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al., 2018; Yang et al., 2019; Dong et al., 2019; Liu et al., 2019c; Lan et al., 2019).”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Dong et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2004) · Cited by 9,718
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Research works printed but not linked: 3
- [2][2] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 01 2006. -
- [14][14] Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. Grounded response generation task at dstc7. In AAAI Dialog System Technology Challenges Workshop , 2019. -
- [22][22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , San Diego, CA, 2015. -
Cited by
Sorted by citations
Items 601–700 of 1,008
Diversity Enhanced Narrative Question Generation for Storybooks
Yoon, H., & Bak, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.31 · Cited by 15
Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia
Suhartono, D., Majiid, M. R. N., & Fredyan, R. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12717-9 · Cited by 28
Liu, C., Wang, D., Zhao, Z., Hu, D., Wu, M., Lin, L., Liu, J., Zhang, H., Shen, S., Li, B., et al. (2024) · Journal on Computing and Cultural Heritage · doi:10.1145/3676969 · Cited by 29
Synergy of Sight and Semantics: Visual Intention Understanding with CLIP
Yang, Q., Ye, M., & Tao, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_9 · Cited by 11
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers
Ahuja, K., Balachandran, V., Panwar, M., He, T., Smith, N. A., Goyal, N., & Tsvetkov, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00733 · Cited by 21
Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge
Sun, K., Yu, D., Chen, J., Yu, D., & Cardie, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.598 · Cited by 10
Learning to Generate Questions by Learning to Recover Answer-containing Sentences
Back, S., Kedia, A., Chinthakindi, S. C., Lee, H., & Choo, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.132 · Cited by 11
Summarization of German Court Rulings
Glaser, I., Moser, S., & Matthes, F. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.19 · Cited by 10
Unified Question Generation with Continual Lifelong Learning
Yuan, W., Yin, H., He, T., Chen, T., Wang, Q., & Cui, L. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511930 · Cited by 11
Steuer, T., Filighera, A., Tregel, T., & Miede, A. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.900304 · Cited by 10
He, X., Gong, Y., Jin, A.-L., Qi, W., Zhang, H., Jiao, J., Zhou, B., Cheng, B., Yiu, S., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.53 · Cited by 12
A Survey of Challenges and Methods in the Computational Modeling of Multi-Party Dialog
Ganesh, A., Palmer, M. S., & Kann, K. (2023) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2023.nlp4convai-1.12 · Cited by 10
BinAIV: Semantic-enhanced vulnerability detection for Linux x86 binaries
Gu, Y., Shu, H., & Kang, F. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103508 · Cited by 11
A Quantum Entanglement-Based Approach for Computing Sentence Similarity
Yan, Y., Qiu, D., & Yan, R. (2020) · IEEE Access · doi:10.1109/access.2020.3025958 · Cited by 11
Image Caption Generation for News Articles
Yang, Z., & Okazaki, N. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.176 · Cited by 10
Li, Z., Zhang, J., Fei, Z., Feng, Y., & Zhou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.91 · Cited by 13
Regularizing transformers with deep probabilistic layers
Aguilera, A. C., Olmos, P. M., Artés-Rodríguez, A., & Pérez‐Cruz, F. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.032 · Cited by 13
Cao, J., & Ananiadou, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.182 · Cited by 10
Bachiri, Y.-A., & Mouncif, H. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-06458-6_11 · Cited by 10
DSM: Question Generation over Knowledge Base via Modeling Diverse Subgraphs with Meta-learner
Guo, S., Zhang, J., Wang, Y., Zhang, Q., Li, C., & Chen, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.281 · Cited by 11
Self-citationCommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense ReasoningPreprint
Lin, B. Y., Shen, M., Xing, Y., Zhou, P., & Ren, X. (2019) · arXiv (Cornell University) · Cited by 9
QURIOUS: Question Generation Pretraining for Text GenerationPreprint
Narayan, S., Simões, G., Ma, J., Craighead, H., & McDonald, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11026 · Cited by 14
Dong, Y., Romascanu, A., & Cheung, J. C. K. (2020) · arXiv (Cornell University) · Cited by 8
Graph-based Multi-hop Reasoning for Long Text GenerationPreprint
Zhao, L., Xu, J., Lin, J., Zhang, Y., Yang, H., & Sun, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13282 · Cited by 10
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic ConstraintsPreprint
Lu, X., West, P., Zellers, R., Bras, R. L., Bhagavatula, C., & Choi, Y. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2010.12884 · Cited by 204
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2021) · arXiv (Cornell University) · Cited by 11
What BERT Sees: Cross-Modal Transfer for Visual Question Generation
Scialom, T., Bordes, P., Dray, P.-A., Staiano, J., & Gallinari, P. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.39 · Cited by 9
WYWEB: A NLP Evaluation Benchmark For Classical Chinese
Zhou, B., Chen, Q., Zhong, X., Zhong, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.204 · Cited by 15
Semantic Encoding Algorithm for Classification and Retrieval of Aviation Safety Reports
Gao, Y., Zhu, G., Duan, Y., & Mao, J. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3359356 · Cited by 9
Ingram, W. A., Wu, J. F., Kahu, S. Y., Manzoor, J. A., Banerjee, B., Ahuja, A., Choudhury, M. H., Salsabil, L., Shields, W., & Fox, E. A. (2024) · International Journal on Digital Libraries · doi:10.1007/s00799-024-00395-4 · Cited by 9
Zhuang, Y., Zhang, J., Li, X., Liu, C., Yu, Y., Dong, W., & He, K. (2024) · JMIR Medical Informatics · doi:10.2196/63020 · Cited by 10
Exploring the use of LLMs in the Italian legal domain: A survey on recent applications
Siino, M. (2025) · Computer law & security review · doi:10.1016/j.clsr.2025.106164 · Cited by 12
Bai, T., Zhao, J., Zhu, J., Han, S., Chen, J., Li, B., & Kot, A. C. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3091683 · Cited by 11
Exploring Pre-Trained Language Models to Build Knowledge Graph for Metal-Organic Frameworks (MOFs)
An, Y., Greenberg, J., Hu, X. T., Kalinowski, A., Fang, X., Zhao, X., McClellan, S., Uribe‐Romo, F. J., Langlois, K., Furst, J. D., et al. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020568 · Cited by 12
A novel semantic-enhanced generative adversarial network for abstractive text summarization
Vo, T. (2023) · Soft Computing · doi:10.1007/s00500-023-07890-x · Cited by 9
Benchmarking Diverse-Modal Entity Linking with Generative Models
Wang, S., Li, A. H., Zhu, H., Zhang, S., Perera, P., Hang, C.-W., Ma, J., Wang, W. Y., Wang, Z., Castelli, V., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.497 · Cited by 11
Self-citationFact Check: Assessing the Response of ChatGPT to Alzheimer's Disease Myths
Huang, S. S., Song, Q., Beiting, K. J., Duggan, M. C., Hines, K., Murff, H., Leung, V., Powers, J., Harvey, T. S., Malin, B., et al. (2024) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2024.105178 · Cited by 9
Wu, X., Jiang, N., & Wu, Y. (2020) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/2020.ngt-1.8 · Cited by 10
CERG: Chinese Emotional Response Generator with Retrieval Method
Zhou, Y., & Ren, F. (2020) · Research · doi:10.34133/2020/2616410 · Cited by 9
AUTH @ CLSciSumm 20, LaySumm 20, LongSumm 20
Gidiotis, A., Stefanidis, S. P., & Tsoumakas, G. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.28 · Cited by 10
Frame Semantics guided network for Abstractive Sentence Summarization
Guan, Y., Guo, S., Li, R., Li, X., & Hu, Z. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106973 · Cited by 12
Shinoda, K., Sugawara, S., & Aizawa, A. (2021) · doi:10.18653/v1/2021.acl-srw.21 · Cited by 10
GMN: Generative Multi-modal Network for Practical Document Information Extraction
Cao, H., Ma, J., Guo, A., Hu, Y., Liu, H., Jiang, D., Liu, Y., & Ren, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.276 · Cited by 9
ICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarization
Zhang, Z., Shu, C., Chen, Y., Xiao, J., Zhang, Q., & Zheng, L. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892884 · Cited by 10
AMOM: Adaptive Masking over Masking for Conditional Masked Language Model
Xiao, Y., Xu, R., Wu, L., Li, J., Qin, T., Liu, T., & Zhang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26615 · Cited by 9
FRSUM: Towards Faithful Abstractive Summarization via Enhancing Factual Robustness
Wu, W., Li, W., Liu, J., Xiao, X., Cao, Z., Li, S., & Wu, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.267 · Cited by 9
Contrastive Demonstration Tuning for Pre-trained Language Models
Liang, X., Zhang, N., Cheng, S., Zhang, Z., Tan, C., & Chen, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.56 · Cited by 11
Neural Knowledge Bank for Pretrained Transformers
Dai, D., Jiang, W., Dong, Q., Lyu, Y., & Sui, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_60 · Cited by 16
Attention Head Masking for Inference Time Content Selection in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.397 · Cited by 9
I-MAD: A Novel Interpretable Malware Detector Using Hierarchical Transformer.Preprint
Li, M. Q., Fung, B. C. M., Charland, P., & Ding, S. H. H. (2019) · arXiv (Cornell University) · Cited by 9
InterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language InteractionPreprint
Lin, J., An, Y., Zhang, Y., Liu, J., Zhou, J., & Yang, H. (2020) · arXiv (Cornell University) · Cited by 11
Accelerating Real-Time Question Answering via Question GenerationPreprint
Fang, Y., Wang, S., Gan, Z., Sun, S., Liu, J., & Zhu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05167 · Cited by 17
Mixed-Lingual Pre-training for Cross-lingual SummarizationPreprint
Xu, R., Zhu, C., Shi, Y., Zeng, M., & Huang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08892 · Cited by 9
Wang, J., Wang, H., Deng, J., Wu, W., & Zhang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04699 · Cited by 17
Huang, S. S., Song, Q., Beiting, K. J., Duggan, M. C., Hines, K., Murff, H. J., Leung, V., Powers, J. S., Harvey, T. S., Malin, B., et al. (2023) · medRxiv · doi:10.1101/2023.09.04.23294917 · Cited by 9
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
Qin, Y., Yuan, W., Neubig, G., & Liu, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1014 · Cited by 16
On the Trade-off between Redundancy and Cohesiveness in Extractive Summarization
Cardenas, R., Gallé, M., & Cohen, S. B. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15191 · Cited by 10
A Web Semantic Mining Method for Fake Cybersecurity Threat Intelligence in Open Source Communities
Li, Z., Yu, X., & Zhao, Y. (2024) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.350095 · Cited by 8
Interactive Concept Network Enhanced Transformer for Remote Sensing Image Captioning
Zhang, C., Ren, Z., Hou, B., Meng, J., Li, W., & Jiao, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3523305 · Cited by 8
ReranKGC: A cooperative retrieve-and-rerank framework for multi-modal knowledge graph completion
Gao, M., Xie, Y., Chen, W., Zhang, F., Ding, F., Wang, T., Yao, J., Zheng, J., & Wong, K. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107467 · Cited by 9
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Yang, J. Y., Ma, S., Dong, L., Huang, S., Huang, H., Yin, Y., Zhang, D., Yang, L., Wei, F., & Li, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.522 · Cited by 13
Self-citationPrompt Tuning for Unified Multimodal Pretrained Models
Yang, H., Lin, J., Yang, A., Wang, P., & Zhou, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.27 · Cited by 12
Diversify Question Generation with Retrieval-Augmented Style Transfer
Gou, Q., Xia, Z., Yu, B., Yu, H., Huang, F., Li, Y., & Nguyen, C.-T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.104 · Cited by 12
Learning Cluster Patterns for Abstractive Summarization
Jo, S.-G., Park, S.-H., Kim, J.-J., & On, B.-W. (2023) · IEEE Access · doi:10.1109/access.2023.3346911 · Cited by 9
Software Defect Prediction via Generative Adversarial Networks and Pre-Trained Model
Song, W., Gan, L., & Bao, T. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01503119 · Cited by 8
ReqCompletion: Domain-Enhanced Automatic Completion for Software Requirements
Lian, X., Ma, J., Lv, H., & Zhang, L. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00023 · Cited by 10
Garg, P. K., Chakraborty, R., & Dandapat, S. K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112969 · Cited by 8
A multi-task learning framework with prompt-based hybrid alignment for vision-language model
Han, J., Wei, Z., Ding, C., & Wang, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130121 · Cited by 10
NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
Tao, Y., Wang, Z., Zhang, H., Wang, L., & Gu, J. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0011-6_7 · Cited by 8
Youngmann, B., Yom‐Tov, E., Gilad-Bachrach, R., & Karmon, D. (2021) · Information Retrieval · doi:10.1007/s10791-021-09392-6 · Cited by 10
Pretraining Financial Text Encoder Enhanced by Lifelong Learning
Liu, Z., Huang, D., & Huang, K. (2020) · IEEE Access · doi:10.1109/access.2020.3027622 · Cited by 8
Toward Stance-based Personas for Opinionated Dialogues
Scialom, T., Tekiroğlu, S. S., Staiano, J., & Guerini, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.238 · Cited by 8
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Zhou, W., Ge, T., Xu, C., Xu, K., & Wei, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.45 · Cited by 13
Self-citationKeep Learning: Self-supervised Meta-learning for Learning from Inference
Kedia, A., & Chinthakindi, S. C. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.6 · Cited by 7
Towards Improving Open Student Answer Assessment using Pretrained Transformers
Khayi, N. A., Rus, V., & Tamang, L. J. (2021) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.v34i1.128483 · Cited by 9
FastSeq: Make Sequence Generation Faster
Yan, Y., Hu, F., Chen, J., Bhendawade, N., Ye, T., Gong, Y., Duan, N., Cui, D., Chi, B., & Zhang, R. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.26 · Cited by 8
Learning to Express in Knowledge-Grounded Conversation
Zhao, X., Fu, T., Tao, C., Wu, W., Zhao, D., & Yan, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.164 · Cited by 8
Liu, M., Xiang, J., Xu, X., & Hu, H. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3551637 · Cited by 9
Use and Design of Chatbots for the Circular Economy
Zota, R. D., Cîmpeanu, I.-A., & Dragomir, D.-A. (2023) · Sensors · doi:10.3390/s23114990 · Cited by 8
Recurrent Attention Networks for Long-text Modeling
Li, X., Li, Z., Luo, X., Xie, H., Lee, X. Z., Zhao, Y., Wang, F. L., & Li, Q. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.188 · Cited by 11
The Web Can Be Your Oyster for Improving Language Models
Li, J., Tang, T., Zhao, W. X., Wang, J., Nie, J., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.46 · Cited by 11
Towards Boosting the Open-Domain Chatbot with Human Feedback
Lu, H. C., Bao, S., He, H., Wang, F., Wu, H., & Wang, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.224 · Cited by 15
Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation
Li, X., Guo, D., Gong, Y., Lin, Y., Shen, Y., Qiu, X., Jiang, D., Chen, W., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.9 · Cited by 18
Towards building a Robust Industry-scale Question Answering System
Chakravarti, R., Ferritto, A., Iyer, B., Pan, L., Florian, R., Roukos, S., & Sil, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-industry.9 · Cited by 8
Liao, J., Shi, Y., Gong, M., Shou, L., Eskimez, Ş. E., Lu, L., Qu, H., & Zeng, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414626 · Cited by 8
Addressing Semantic Drift in Question Generation for Semi-Supervised Question AnsweringPreprint
Zhang, S., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06356 · Cited by 19
Working Memory GraphsPreprint
Loynd, R., Fernandez, R., Çelikyılmaz, A., Swaminathan, A., & Hausknecht, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07141 · Cited by 7
Pre-training Text Representations as Meta LearningPreprint
Lv, S., Wang, Y., Guo, D., Tang, D., Duan, N., Zhu, F., Gong, M., Shou, L., Ma, R., Jiang, D., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05568 · Cited by 7
Self-citationComparative Study of Machine Learning Models and BERT on SQuADPreprint
Patel, D., Raval, P., Parikh, R., & Shastri, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11313 · Cited by 13
A Further Study of Unsupervised Pre-training for Transformer Based Speech RecognitionPreprint
Jiang, D., Li, W., Zhang, R., Cao, M. Y., Luo, N., Han, Y., Zou, W., & Li, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09862 · Cited by 8
ColdGANs: Taming Language GANs with Cautious Sampling StrategiesPreprint
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04643 · Cited by 8
Variance-reduced Language Pretraining via a Mask Proposal NetworkPreprint
Chen, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05333 · Cited by 8
End-to-End Neural Event Coreference ResolutionPreprint
Lu, Y., Lin, H., Tang, J., Han, X., & Sun, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08153 · Cited by 8
Dimsum @LaySumm 20: BART-based Approach for Scientific Document SummarizationPreprint
Yu, T., Su, D., Dai, W., & Fung, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09252 · Cited by 8
Xiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12148 · Cited by 8
Kim, S., Eric, M., Hedayatnia, B., Gopalakrishnan, K., Liu, Y., Huang, C.-W., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09276 · Cited by 10
Speech-Language Pre-Training for End-to-End Spoken Language UnderstandingPreprint
Qian, Y., Bianv, X., Shi, Y., Kanda, N., Shen, L., Xiao, Z., & Zeng, M. (2021) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.48550/arxiv.2102.06283 · Cited by 57
Personalized Transformer for Explainable RecommendationPreprint
Li, L., Zhang, Y., & Chen, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11601 · Cited by 8
Global-aware Beam Search for Neural Abstractive SummarizationPreprint
Ma, Y., Lan, Z., Zong, L., & Huang, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06891 · Cited by 8
Zero-shot Fact Verification by Claim GenerationPreprint
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14682 · Cited by 15
Self-citation
Metadata
- Authors
- Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03197
- OpenAlex
- W2945260553
- Cited by
- 1,225
- References
- 48
- Citations by year
- 2026: 1 · 2025: 61 · 2024: 146 · 2023: 253 · 2022: 206 · 2021: 289 · 2020: 240 · 2019: 28
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-training, natural language understanding, natural language generation, sequence-to-sequence prediction, abstractive summarization, question answering, question generation, GLUE benchmark, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
