The DOI matches its DataCite record; 60 of the 60 references it cites were found in a registry or an index.
Unified Language Model Pre-training for Natural Language Understanding and Generation
Open accessAbstract
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“A difference is that UniLM predictions are conditionally independent, whereas BART's are autoregressive.”
From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Dong et al., 2019
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This integration leverages the linguistic capabilities of Llama 3, utilizing both its textual output and intermediate embeddings at the token rate (Devlin et al., 2018; Dong et al., 2019; Raffel et al., 2020; Guo et al., 2023) to enhance the prediction of prosody features, thus reducing the lookahead required by the model.”
From The Llama 3 Herd of Models (2024) · cites this paper as Dong et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This approach has recently been used to obtain state-of-the-art results in many of the most common NLP benchmarks (Devlin et al., 2018; Yang et al., 2019; Dong et al., 2019; Liu et al., 2019c; Lan et al., 2019).”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Dong et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2004) · Cited by 9,718
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Research works printed but not linked: 3
- [2][2] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 01 2006. -
- [14][14] Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. Grounded response generation task at dstc7. In AAAI Dialog System Technology Challenges Workshop , 2019. -
- [22][22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , San Diego, CA, 2015. -
Cited by
Sorted by citations
Items 201–300 of 1,008
Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction
Bae, I., Lee, J., & Jeon, H.-G. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.00078 · Cited by 93
SCARF: Self-Supervised Contrastive Learning using Random Feature CorruptionPreprint
Bahri, D., Jiang, H., Tay, Y., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15147 · Cited by 161
Table Fact Verification with Structure-Aware Transformer
Zhang, H., Wang, Y., Wang, S., Cao, X., Zhang, F., & Wang, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.126 · Cited by 53
Self-citationLong Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
Guan, J., Mao, X., Fan, C., Liu, Z., Ding, W., & Huang, M. (2021) · doi:10.18653/v1/2021.acl-long.499 · Cited by 75
A Survey on Transfer Learning in Natural Language ProcessingPreprint
Alyafeai, Z., Al-shaibani, M. S., & Ahmad, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04239 · Cited by 120
BiasAsker: Measuring the Bias in Conversational AI System
Wan, Y., Wang, W., He, P., Gu, J., Bai, H., & Lyu, M. R. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616310 · Cited by 54
Self-citationMultimodal Pretraining for Dense Video Captioning
Huang, G., Pang, B., Zhu, Z., Rivera, C. E., & Soricut, R. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.48 · Cited by 56
Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization
Laskar, M. T. R., Hoque, E., & Huang, J. X. (2022) · Computational Linguistics · doi:10.1162/coli_a_00434 · Cited by 58
InterBERT: Vision-and-Language Interaction for Multi-modal PretrainingPreprint
Lin, J., An, Y., Zhang, Y., Liu, J., Zhou, J., & Yang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13198 · Cited by 86
Yang, M., Huang, L., Huang, H., Tang, H., Zhang, N., Yang, H., Wu, J., & Mu, F. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac326 · Cited by 56
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Controlling the Amount of Verbatim Copying in Abstractive Summarization
Song, K., Wang, B., Feng, Z., Liu, R., & Liu, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6420 · Cited by 50
Pre-training Methods in Information Retrieval
Fan, Y., Xie, X., Cai, Y., Chen, J., Ma, X., Li, X., Zhang, R., & Guo, J. (2022) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000100 · Cited by 69
HTSS: A novel hybrid text summarisation and simplification architecture
Zaman, F., Shardlow, M., Hassan, S., Aljohani, N. R., & Nawaz, R. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102351 · Cited by 51
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Yang, Z., Zhu, C., Gmyr, R., Zeng, M., Huang, X., & Darve, E. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.168 · Cited by 51
Ma, S., Dong, L., Huang, S., Zhang, D., Muzio, A. F. V., Singhal, S., Awadalla, H. H., Song, X., & Wei, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13736 · Cited by 63
Self-citationSyn-QG: Syntactic and Shallow Semantic Rules for Question Generation
Dhole, K., & Manning, C. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.69 · Cited by 48
On Multi-Modal Learning of Editing Source Code
Chakraborty, S., & Ray, B. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678559 · Cited by 55
A Survey of Pretrained Language Models
Sun, K., Luo, X., & Luo, M. Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10986-7_36 · Cited by 48
Friendly Topic Assistant for Transformer Based Abstractive Summarization
Wang, Z., Duan, Z., Zhang, H., Wang, C., Tian, L., Chen, B., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.35 · Cited by 49
Self-citationFine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning
Grießhaber, D., Maucher, J., & Vu, N. T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.100 · Cited by 60
Text Compression-aided Transformer Encoding
Li, Z., Zhang, Z., Zhao, H., Wang, R., Chen, K., Utiyama, M., & Sumita, E. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3058341 · Cited by 50
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
Zellers, R., Holtzman, A., Peters, M. E., Mottaghi, R., Kembhavi, A., Farhadi, A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.159 · Cited by 69
EQG-RACE: Examination-Type Question Generation
Jia, X., Zhou, W., Sun, X., & Wu, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17553 · Cited by 48
Esteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09595 · Cited by 48
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296
Discourse-Aware Unsupervised Summarization for Long Scientific Documents
Dong, Y., Mircea, A., & Cheung, J. C. K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.93 · Cited by 73
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Tang, Z., Lei, J., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.193 · Cited by 48
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Zhang, Z., Chen, S., Zhou, L., Wu, Y., Ren, S., Liu, S., Yao, Z., Gong, X., Dai, L.-R., Li, J., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3379877 · Cited by 72
Self-citationLearning to Ask Unanswerable Questions for Machine Reading Comprehension
Zhu, H., Dong, L., Wei, F., Wang, W., Qin, B., & Liu, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1415 · Cited by 48
Self-citationProgram Enhanced Fact Verification with Verbalization and Graph Attention Network
Yang, X., Nie, F., Feng, Y., Liu, Q., Chen, Z., & Zhu, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.628 · Cited by 53
I-MAD: Interpretable malware detector using Galaxy Transformer
Li, M. Q., Fung, B. C. M., Charland, P., & Ding, S. H. H. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102371 · Cited by 54
One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
Wu, C., Wu, F., & Huang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.387 · Cited by 62
Controllable Abstractive Dialogue Summarization with Sketch Supervision
Wu, C.-S., Liu, L., Liu, W., Stenetorp, P., & Xiong, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.454 · Cited by 53
ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented DialoguesPreprint
Wu, C.-S., Hoi, S. C. H., Socher, R., & Xiong, C. (2020) · arXiv (Cornell University) · Cited by 47
Ladhak, F., Durmus, E., He, H., Cardie, C., & McKeown, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.100 · Cited by 46
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better
Wu, C., Wu, F., Qi, T., & Huang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.76 · Cited by 59
A Framework and Operational Procedures for Metaverses-Based Industrial Foundation Models
Wang, J., Tian, Y., Wang, Y., Yang, J., Wang, X., Wang, S., & Kwan, O. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3226755 · Cited by 48
Self-citationOne Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
Wang, D., Chen, B., Li, S., Luo, W., Peng, S., Dong, W., & Liao, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00013 · Cited by 57
A Closer Look at Data Bias in Neural Extractive Summarization Models
Zhong, M., Wang, D., Liu, P., Qiu, X., & Huang, X. (2019) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/d19-5410 · Cited by 48
Zhang, Z., Zhou, L., Ao, J., Liu, S., Dai, L.-R., Li, J., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.108 · Cited by 55
Self-citationDetCLIPv3: Towards Versatile Generative Open-Vocabulary Object Detection
Yao, L., Pi, R., Han, J., Liang, X., Xu, H., Zhang, W., Li, Z., & Xu, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02586 · Cited by 54
Unsupervised Multi-hop Question Answering by Question Generation
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.469 · Cited by 55
Self-citationMachine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 63
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Fu, T.-J., Wang, W. Y., McDuff, D., & Song, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19943 · Cited by 63
Self-citationGupta, D. K., Ekbal, A., & Bhattacharyya, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.206 · Cited by 48
Cross-Lingual Abstractive Summarization with Limited Parallel Resources
Bai, Y., Gao, Y., & Huang, H. (2021) · doi:10.18653/v1/2021.acl-long.538 · Cited by 51
PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
Hua, X., & Wang, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.57 · Cited by 48
Pre-training Entity Relation Encoder with Intra-span and Inter-span Information
Wang, Y., Sun, C., Wu, Y., Yan, J., Gao, P., & Xie, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.132 · Cited by 43
Self-citationZero-shot Fact Verification by Claim Generation
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · doi:10.18653/v1/2021.acl-short.61 · Cited by 43
Self-citationSimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.125 · Cited by 111
Self-citationGenerating fact checking briefs
Fan, A., Piktus, A., Petroni, F., Wenzek, G., Saeidi, M., Vlachos, A., Bordes, A., & Riedel, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.580 · Cited by 42
Simplifying Paragraph-Level Question Generation via Transformer Language Models
Lopez, L. E., Cruz, D. K., Cruz, J. C. B., & Cheng, C. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-89363-7_25 · Cited by 41
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41
Better Document-Level Machine Translation with Bayes’ Rule
Yu, L., Sartran, L., Stokowiec, W., Wang, L., Kong, L., Blunsom, P., & Dyer, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00319 · Cited by 42
Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Xu, S., Zhang, X., Wu, Y., Wei, F., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.161 · Cited by 47
Self-citationDiscern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Reading
Gao, Y., Wu, C.-S., Li, J., Joty, S., Hoi, S. C. H., Xiong, C., King, I., & Lyu, M. R.-T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.191 · Cited by 42
Parallel Refinements for Lexically Constrained Text Generation with BART
He, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.681 · Cited by 54
Leaf: Multiple-Choice Question Generation
Vachev, K., Hardalov, M., Karadzhov, G., Georgiev, G., Koychev, I., & Nakov, P. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-99739-7_41 · Cited by 45
Knowledge-based dynamic prompt learning for multi-label disease diagnosis
Xie, J., Li, X., Yuan, Y., Guan, Y., Jiang, J., Guo, X., & Peng, X. X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111395 · Cited by 39
PHCG: PLC Honeypoint Communication Generator for Industrial IoT
Liu, H., Zhou, Y., Fang, B., Sun, Y., Hu, N., & Tian, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3455564 · Cited by 43
Pre-training for Abstractive Document Summarization by Reinstating Source Text
Zou, Y., Zhang, X., Lu, W., Wei, F., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.297 · Cited by 46
Self-citationContrastive Information Extraction With Generative Transformer
Zhang, N., Ye, H., Deng, S., Tan, C., Chen, M., Huang, S., Huang, F., & Chen, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3110126 · Cited by 42
Pre-training Text-to-Text Transformers for Concept-centric Common SensePreprint
Zhou, W., Lee, D., Selvam, R. K., Lee, S., & Ren, X. (2021) · arXiv (Cornell University) · Cited by 42
Multi-turn intent determination and slot filling with neural networks and regular expressions
Abro, W. A., Qi, G., Ali, Z., Feng, Y., & Aamir, M. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106428 · Cited by 39
Improving Truthfulness of Headline Generation
Matsumaru, K., Takase, S., & Okazaki, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.123 · Cited by 42
Multi-hop Question Generation with Graph Convolutional Network
Su, D., Xu, Y., Dai, W., Ji, Z., Yu, T., & Fung, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.416 · Cited by 38
Goodwin, T. R., Savery, M. E., & Demner‐Fushman, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.494 · Cited by 42
BASS: Boosting Abstractive Summarization with Unified Semantic Graph
Wu, W., Li, W., Xiao, X., Liu, J., Cao, Z., Li, S., Wu, H. R., & Wang, H. (2021) · doi:10.18653/v1/2021.acl-long.472 · Cited by 45
Encoder-Agnostic Adaptation for Conditional Language GenerationPreprint
Ziegler, Z. M., Melas-Kyriazi, L., Gehrmann, S., & Rush, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06938 · Cited by 60
End-to-End Video Question-Answer Generation With Generator-Pretester Network
Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., & Hsu, W. H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3051277 · Cited by 38
Self-citationTesting Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 43
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 41
Multimodal Colearning Meets Remote Sensing: Taxonomy, State of the Art, and Future Works
Kieu, N., Nguyen, K., Nazib, A., Fernando, T., Fookes, C., & Sridharan, S. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3378348 · Cited by 38
Zhou, Y., Shen, T., Geng, X., Long, G., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.183 · Cited by 47
Natural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 41
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 45
StableMoE: Stable Routing Strategy for Mixture of Experts
Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.489 · Cited by 108
Self-citationChung, H.-L., Chan, Y.-H., & Fan, Y.-C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.393 · Cited by 47
Defending against Backdoor Attacks in Natural Language Generation
Sun, X., Li, X., Meng, Y., Ao, X., Lyu, L., Li, J., & Zhang, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25656 · Cited by 48
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01909 · Cited by 85
All NLP Tasks Are Generation Tasks: A General Pretraining FrameworkPreprint
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · Cited by 38
Self-citationRefSum: Refactoring Neural Summarization
Liu, Y., Dou, Z.-Y., & Liu, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.113 · Cited by 36
HGPrompt: Bridging Homogeneous and Heterogeneous Graphs for Few-Shot Prompt Learning
Yu, X., Fang, Y., Liu, Z., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29596 · Cited by 42
Guo, J., Wang, M., Yin, H., Song, B., Chi, Y., Yu, F. R., & Yuen, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3496491 · Cited by 42
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
Liu, D., Gong, Y., Fu, J., Yan, Y., Chen, J., Lv, J., Duan, N., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.467 · Cited by 37
Self-citationSpeech-Language Pre-Training for End-to-End Spoken Language Understanding
Qian, Y., Bianv, X., Shi, Y., Kanda, N., Shen, L., Xiao, Z. Y., & Zeng, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414900 · Cited by 40
Sun, Y., Zheng, Y., Hao, C., & Qiu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03564 · Cited by 57
RepsNet: Combining Vision with Language for Automated Medical Reports
Tanwani, A. K., Barral, J. K., & Freedman, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_68 · Cited by 39
Material transformers: deep learning language models for generative materials design
Fu, N., Wei, L., Song, Y., Li, Q., Xin, R., Omee, S. S., Dong, R., Siriwardane, E. M. D., & Hu, J. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acadcd · Cited by 41
Rethinking Perturbations in Encoder-Decoders for Fast Training
Takase, S., & Kiyono, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.460 · Cited by 39
TreeBERT: A Tree-Based Pre-Trained Model for Programming LanguagePreprint
Jiang, X., Zheng, Z., Lyu, C., Li, L., & Lyu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12485 · Cited by 43
DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition With Limited Annotations
Hu, P., Sun, X., Sclaroff, S., & Saenko, K. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346405 · Cited by 42
Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs
Yu, X., Liu, Z., Fang, Y., Liu, Z., Chen, S., & Zhang, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3419109 · Cited by 41
Xiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.136 · Cited by 45
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Lyu, C., Shang, L., Graham, Y., Foster, J., Jiang, X., & Liu, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.340 · Cited by 37
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Yao, B., Wang, D., Wu, T., Zhang, Z., Li, T., Yu, M., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.54 · Cited by 51
A Review of Transformer-Based Approaches for Image Captioning
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2023) · Applied Sciences · doi:10.3390/app131911103 · Cited by 39
Instance-Dictionary Learning for Open-World Object Detection in Autonomous Driving Scenarios
Ma, Z., Zheng, Z., Wei, J., Yang, Y., & Shen, H. T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3322465 · Cited by 34
Metadata
- Authors
- Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.03197
- OpenAlex
- W2945260553
- Cited by
- 1,225
- References
- 48
- Citations by year
- 2026: 1 · 2025: 61 · 2024: 146 · 2023: 253 · 2022: 206 · 2021: 289 · 2020: 240 · 2019: 28
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-training, natural language understanding, natural language generation, sequence-to-sequence prediction, abstractive summarization, question answering, question generation, GLUE benchmark, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
