The DOI matches its DataCite record; 60 of the 60 references it cites were found in a registry or an index.
Unified Language Model Pre-training for Natural Language Understanding and Generation
Open accessAbstract
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“A difference is that UniLM predictions are conditionally independent, whereas BART's are autoregressive.”
From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Dong et al., 2019
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This integration leverages the linguistic capabilities of Llama 3, utilizing both its textual output and intermediate embeddings at the token rate (Devlin et al., 2018; Dong et al., 2019; Raffel et al., 2020; Guo et al., 2023) to enhance the prediction of prosody features, thus reducing the lookahead required by the model.”
From The Llama 3 Herd of Models (2024) · cites this paper as Dong et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al., 2018; Yang et al., 2019; Dong et al., 2019; Liu et al., 2019c; Lan et al., 2019).”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Dong et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2004) · Cited by 9,718
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Research works printed but not linked: 3
- [2][2] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 01 2006. -
- [14][14] Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. Grounded response generation task at dstc7. In AAAI Dialog System Technology Challenges Workshop , 2019. -
- [22][22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , San Diego, CA, 2015. -
Cited by
Sorted by citations
Items 301–400 of 1,008
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
Stickland, A. C., Li, X., & Ghazvininejad, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.301 · Cited by 36
CQG: A Simple and Effective Controlled Generation Framework for Multi-hop Question Generation
Fei, Z., Zhang, Q., Gui, T., Liang, D., Wang, S., Wu, W., & Huang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.475 · Cited by 35
CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent DetectionPreprint
Xia, C., Zhang, C., Nguyen, H., Zhang, J., & Yu, P. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01881 · Cited by 42
PeCo: Perceptual Codebook for BERT Pre-training of Vision TransformersPreprint
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N., & Guo, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12710 · Cited by 32
MixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31
Review on Neural Question Generation for Education Purposes
Faraby, S. A., Adiwijaya, A., & Romadhony, A. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00374-x · Cited by 44
LLMDet: A Third Party Large Language Models Generated Text Detection Tool
Wu, K., Pang, L., Shen, H., Cheng, X., & Chua, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.139 · Cited by 64
Socially adaptive cognitive architecture for human-robot collaboration in industrial settings
Freire, I. T., Guerrero-Rosado, O., Amil, A. F., & Verschure, P. F. M. J. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1248646 · Cited by 34
Knowledge graph construction for heart failure using large language models with prompt engineering
Xu, T., Gu, Y., Xue, M., Gu, R., Li, B., & Gu, X. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1389475 · Cited by 35
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Li, Z., Wang, W., Dong, L., Wei, F., & Xu, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.600 · Cited by 38
Self-citationHow to Ask Good Questions? Try to Leverage Paraphrases
Jia, X., Zhou, W., Sun, X., & Wu, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.545 · Cited by 31
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
Liu, X., Wang, Y., Ji, J., Cheng, H., Zhu, X., Awa, E., He, P., Chen, W., Poon, H., Cao, G., et al. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.16 · Cited by 45
Self-citationA Survey of Text Summarization Approaches Based on Deep Learning
Hou, S., Huang, X., Fei, C., Zhang, S., Li, Y., Sun, Q., & Wang, C. (2021) · Journal of Computer Science and Technology · doi:10.1007/s11390-020-0207-x · Cited by 32
Dialogue Graph Modeling for Conversational Machine Reading
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.279 · Cited by 33
Tsao, L.-W., Wang, Y., Lin, H.-S., Shuai, H.-H., Wong, L.-K., & Cheng, W.-H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_14 · Cited by 31
Self-citationMapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 31
Strategic Deductive Reasoning in Large Language Models: A Dual-Agent Approach
Li, S., Zhou, X., Wu, Z., Long, Y., & Shen, Y. (2024) · IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics62053.2024.10797012 · Cited by 30
A Study of Pre-trained Language Models in Natural Language Processing
Duan, J., Zhao, H. J., Zhou, Q., Qiu, M., & Liu, M. (2020) · IEEE International Conference on Smart Cloud (SmartCloud) · doi:10.1109/smartcloud49737.2020.00030 · Cited by 32
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition
Jiang, D., Li, W., Zhang, R., Cao, M. Y., Luo, N., Yang, H., Zou, W., Han, K., & Li, X. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414539 · Cited by 34
CINS: Comprehensive Instruction for Few-Shot Learning in Task-Oriented Dialog Systems
Mi, F., Wang, Y., & Li, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21356 · Cited by 34
Self-citationIterative GNN-based Decoder for Question Generation
Fei, Z., Zhang, Q., & Zhou, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.201 · Cited by 30
Amendable Generation for Dialogue State Tracking
Tian, X., Huang, L., Lin, Y., Bao, S., Huang, H., Yang, Y., Wu, H. R., Wang, F., & Sun, S. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.8 · Cited by 30
Enhancing Abstractive Summarization with Extracted Knowledge Graphs and Multi-Source Transformers
Chen, T., Wang, X., Yue, T., Bai, X., Le, C. X., & Wang, W. (2023) · Applied Sciences · doi:10.3390/app13137753 · Cited by 37
Self-citation“Mathematical word problem generation from commonsense knowledge graph and equations,”
Liu, T., Fang, Q., Ding, W., Li, H., Wu, Z., & Liu, Z. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.348 · Cited by 29
Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks
Zhao, Y., Xu, C., & Wu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.279 · Cited by 32
A Comprehensive Review on Transformers Models For Text Classification
Kora, R., & Mohammed, A. (2023) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc58832.2023.10278387 · Cited by 37
Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation
Guo, J., Zhang, Z., Xu, L., Chen, B., & Chen, E. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3076863 · Cited by 34
Recall and Learn: A Memory-augmented Solver for Math Word Problems
Huang, S., Wang, J., Xu, J., Cao, D., & Yang, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.68 · Cited by 33
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
Yao, Y., Huang, S., Li, D., Wei, F., Chen, H., & Zhang, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_11 · Cited by 33
Self-citationUNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive LearningPreprint
Li, W., Gao, C., Niu, G., Xiao, X., Líu, H., Liu, J., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15409 · Cited by 29
Exploring Prompt-based Few-shot Learning for Grounded Dialog GenerationPreprint
Zheng, C., & Huang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06513 · Cited by 43
Length Control in Abstractive Summarization by Pretraining Information Selection
Liu, Y., Jia, Q. F., & Zhu, K. Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.474 · Cited by 32
Designing Social Robot for Adults Using Self-Determination Theory and AI Technologies
Lu, Y., Chen, C., Chen, P., & Yu, S. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3250465 · Cited by 33
Leveraging Summary Guidance on Medical Report Summarization
Zhu, Y., Yang, X., Wu, Y., & Zhang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3304376 · Cited by 37
MBCFNet: A Multimodal Brain–Computer Fusion Network for human intention recognition
Li, Z., Zhang, G., Okada, S., Wang, L., Zhao, B., & Dang, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111826 · Cited by 29
Explainable natural language processing for corporate sustainability analysis
Ong, K., Mao, R., Satapathy, R., Filho, R. S., Cambria, E., Sulaeman, J., & Mengaldo, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102726 · Cited by 39
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
Riabi, A., Scialom, T., Keraron, R., Sagot, B., Seddah, D., & Staiano, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.562 · Cited by 44
Multilingual AMR-to-Text Generation
Fan, A., & Gardent, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.231 · Cited by 33
Enriching and Controlling Global Semantics for Text Summarization
Nguyen, T., Luu, A. T., Lu, T., & Quan, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.744 · Cited by 30
Turkish abstractive text summarization using pretrained sequence-to-sequence models
Baykara, B., & Güngör, T. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000195 · Cited by 34
Mukayese: Turkish NLP Strikes Back
Safaya, A., Kurtuluş, E., Goktogan, A., & Yüret, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.69 · Cited by 33
Multimodal Fusion Remote Sensing Image–Audio Retrieval
Yang, R., Wang, S., Sun, Y., Zhang, H., Liao, Y., Gu, Y., Hou, B., & Jiao, L. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3194076 · Cited by 28
Tell Me Where To Go: Voice-Controlled Hands-Free Locomotion for Virtual Reality Systems
Hombeck, J., Voigt, H., Heggemann, T., Datta, R. R., & Lawonn, K. (2023) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr55154.2023.00028 · Cited by 30
Li, H., Peng, Q., Xu, M., Wang, Y., Zeng, Z., & Bashir, M. F. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3304473 · Cited by 28
Self-citationA diversity-enhanced knowledge distillation model for practical math word problem solving
Zhang, Y., Zhou, G., Xie, Z., Ma, J., & Huang, J. X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104059 · Cited by 28
A Review: Abstractive Text Summarization Techniques using NLP
Batra, P., Chaudhary, S., Bhatt, K., Varshney, S., & Verma, S. (2020) · International Conference on Advances in Computing, Communication and Materials (ICACCM) · doi:10.1109/icaccm50413.2020.9213079 · Cited by 27
Don’t Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding
Zhu, Q., Khan, H., Soltan, S., Rawls, S., & Hamza, W. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.40 · Cited by 26
BTS: Back TranScription for Speech-to-Text Post-Processor using Text-to-Speech-to-Text
Park, C., Seo, J., Lee, S., Lee, C., Moon, H., Eo, S., & Lim, H. (2021) · Workshop on Asian Translation (WAT) · doi:10.18653/v1/2021.wat-1.10 · Cited by 26
Towards Automatic Job Description Generation with Capability-Aware Neural Networks
Qin, C., Yao, K., Zhu, H., Xu, T., Shen, D., Chen, E., & Xiong, H. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3145396 · Cited by 31
Improving Biomedical Information Retrieval with Neural Retrievers
Luo, M., Mitra, A., Gokhale, T., & Baral, C. R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21352 · Cited by 29
Code Execution with Pre-trained Language Models
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.308 · Cited by 37
E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3341917 · Cited by 32
Applying the Transformer to Character-level TransductionPreprint
Wu, S., Cotterell, R., & Hulden, M. (2020) · arXiv (Cornell University) · doi:10.3929/ethz-b-000518998 · Cited by 27
A Survey of Knowledge Enhanced Pre-trained ModelsPreprint
Yang, J., Xiao, G., Shen, Y.-L., Jiang, W., Hu, X., Zhang, Y., & Peng, J. (2021) · arXiv.org · doi:10.48550/arxiv.2110.00269 · Cited by 64
Interpreting Deep Learning Models in Natural Language Processing: A ReviewPreprint
Sun, X., Yang, D., Li, X., Zhang, T., Meng, Y., Qiu, H., Wang, G., Hovy, E. H., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10470 · Cited by 64
UFO: A UniFied TransfOrmer for Vision-Language Representation LearningPreprint
Wang, J., Hu, X., Gan, Z., Yang, Z., Dai, X., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10023 · Cited by 50
Mao, Z., Wang, H., Du, Y., & Wong, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.100 · Cited by 37
Transforming Wikipedia Into Augmented Data for Query-Focused Summarization
Zhu, H., Dong, L., Wei, F., Qin, B., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3171963 · Cited by 29
Self-citationProbabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Liao, Y., Jiang, X., & Liu, Q. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.24 · Cited by 39
Composed Variational Natural Language Generation for Few-shot Intents
Xia, C., Xiong, C., Yu, P. S., & Socher, R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.303 · Cited by 26
Conversational Semantic Role Labeling
Xu, K., Wu, H., Song, L., Zhang, H., Song, L., & Yu, D. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3074014 · Cited by 25
Question-aware transformer models for consumer health question summarization
Yadav, S., Gupta, D., Abacha, A. B., & Demner‐Fushman, D. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104040 · Cited by 27
RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
Peng, B., Li, C., Zhang, Z., Zhu, C., Li, J., & Gao, J. (2021) · doi:10.18653/v1/2021.acl-long.341 · Cited by 38
Self-citationRetrieval Enhanced Model for Commonsense Generation
Wang, H., Liu, Y., Zhu, C., Shou, L., Gong, M., Xu, Y., & Zeng, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.269 · Cited by 32
A Feasibility Study of Answer-Unaware Question Generation for Education
Dugan, L. D., Miltsakaki, E., Upadhyay, S., Ginsberg, E., Gonzalez, H., Choi, D., Yuan, C., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.151 · Cited by 27
Distilled Dual-Encoder Model for Vision-Language Understanding
Wang, Z., Wang, W., Zhu, H., Liu, M., Qin, B., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.608 · Cited by 34
Self-citationDenoising based Sequence-to-Sequence Pre-training for Text Generation
Wang, L., Zhao, W., Jia, R., Li, S., & Liu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1412 · Cited by 35
Contrastive Learning with Adversarial Perturbations for Conditional Text GenerationPreprint
Lee, S., Lee, D. B., & Hwang, S. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07280 · Cited by 45
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image SynthesisPreprint
Zhang, Z., Ma, J., Zhou, C., Men, R., Li, Z., Ding, M., Tang, J., Zhou, J., & Yang, H. (2021) · arXiv (Cornell University) · Cited by 26
Qian, Y., Wang, X.-T., Cai, L., Han, J., Huang, Z.-G., Lou, Y., Zhang, B.-Y., Wang, Y., Sun, X.-N., Zhang, Y., et al. (2023) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2023.12.004 · Cited by 28
Self-citationBonfigli, A., Bacco, L., Merone, M., & Dell’Orletta⋄, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103003 · Cited by 31
MLSUM: The Multilingual Summarization Corpus
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.647 · Cited by 64
Semantic Role Labeling Guided Multi-turn Dialogue ReWriter
Xu, K., Tan, H., Song, L., Wu, H., Zhang, H., Song, L., & Yu, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.537 · Cited by 25
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yao, Y., Huang, S., Wang, W., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.40 · Cited by 43
Self-citationIntent Classification and Slot Filling for Privacy Policies
Ahmad, W. U., Chi, J., Le, T., Norton, T. B., Tian, Y., & Chang, K.-W. (2021) · doi:10.18653/v1/2021.acl-long.340 · Cited by 25
Benchmarking Robustness of Machine Reading Comprehension Models
Si, C., Yang, Z., Cui, Y., Ma, W., Liu, T., & Wang, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.56 · Cited by 30
Decoding Methods in Neural Language Generation: A Survey
Zarrieß, S., Voigt, H., & Schüz, S. (2021) · Information · doi:10.3390/info12090355 · Cited by 29
Towards information-rich, logical dialogue systems with knowledge-enhanced neural models
Wang, H., Guo, B., Wu, W., Liu, S., & Yu, Z. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.131 · Cited by 31
Automatic Text Evaluation through the Lens of Wasserstein Barycenters
Colombo, P., Staerman, G., Clavel, C., & Piantanida, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.817 · Cited by 35
A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation
Liu, S., Zhao, X., Li, B., Ren, F., Zhang, L., & Yin, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.173 · Cited by 25
Fast and Constrained Absent Keyphrase Generation by Prompt-Based Learning
Wu, H., Ma, B., Liu, W., Chen, T., & Nie, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21402 · Cited by 25
Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask
Ghanem, B., Coleman, L. L., Dexter, J. R., von der Ohe, S., & Fyshe, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.168 · Cited by 27
Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System
Bang, N., Lee, J., & Koo, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.464 · Cited by 25
Towards Interpretable Natural Language Understanding with Explanations as Latent VariablesPreprint
Zhou, W., Hu, J., Zhang, H., Liang, X., Sun, M., Xiong, C., & Tang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05268 · Cited by 26
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and GenerationPreprint
Liu, J., Zhu, X., Liu, F., Guo, L., Zhao, Z., Sun, M., Wang, N., Lu, H., Zhou, S., Zhang, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00249 · Cited by 40
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation
Narayan, S., Simões, G., Zhao, Y., Maynez, J., Das, D., Collins, M., & Lapata, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.94 · Cited by 27
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Gao, J., Xu, H., Han, S., Ren, X., Yu, P. L. H., Liang, X., Jiang, X., & Li, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21311 · Cited by 32
Self-citationAutomatic Product Copywriting for E-commerce
Zhang, X., Zou, Y., Zhang, H., Zhou, J., Diao, S., Chen, J., Ding, Z., He, Z., He, X., Xiao, Y., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21508 · Cited by 24
LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models
Wang, Z., Chen, Y., Xie, L., Tian, Q., & Wang, Y. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3308474 · Cited by 24
Self-citationMathematical Reasoning via Self-supervised Skip-tree TrainingPreprint
Rabe, M. N., Lee, D. M., Bansal, K., & Szegedy, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04757 · Cited by 38
Esale: Enhancing Code-Summary Alignment Learning for Source Code Summarization
Fang, C., Sun, W., Chen, Y., Chen, X., Wei, Z., Zhang, Q., You, Y., Luo, B., Liu, Y., & Chen, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3422274 · Cited by 25
CAiRE-COVID: A Question Answering and Multi-Document Summarization System for COVID-19 Research.
Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E. J., & Fung, P. (2020) · Europe PMC (PubMed Central) · Cited by 24
Analyzing the Potential of Pre-Trained Embeddings for Audio Classification Tasks
Grollmisch, S., Cano, E., Kehling, C., & Taenzer, M. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287743 · Cited by 24
Investigating Learning Dynamics of BERT Fine-Tuning
Hao, Y., Dong, L., Wei, F., & Xu, K. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.11 · Cited by 25
Self-citationYang, S., Zhang, R., Erfani, S., & Lau, J. H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/548 · Cited by 23
Hark: A Deep Learning System for Navigating Privacy Feedback at Scale
Harkous, H., Peddinti, S. T., Khandelwal, R., Srivastava, A., & Taft, N. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833729 · Cited by 28
KPT++: Refined knowledgeable prompt tuning for few-shot text classification
Ni, S., & Kao, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110647 · Cited by 23
MVP: Multi-task Supervised Pre-training for Natural Language Generation
Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.558 · Cited by 24
Evaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 41
Metadata
- Authors
- Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03197
- OpenAlex
- W2945260553
- Cited by
- 1,225
- References
- 48
- Citations by year
- 2026: 1 · 2025: 61 · 2024: 146 · 2023: 253 · 2022: 206 · 2021: 289 · 2020: 240 · 2019: 28
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-training, natural language understanding, natural language generation, sequence-to-sequence prediction, abstractive summarization, question answering, question generation, GLUE benchmark, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
