The DOI matches its Crossref record; 78 of the 82 references it cites were found in a registry or an index.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Open accessAbstract
We propose pre-finetuning, an additional largescale learning stage between language model pre-training and fine-tuning.Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks.We show that prefinetuning consistently improves performance for pretrained discriminators (e.g.RoBERTa) and generation models (e.g.BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning.We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2014) · Cited by 33,419
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Self-citationBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Research works printed but not linked: 7
- Hector Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Princi- ples of Knowledge Representation and Reasoning. Citeseer.
- Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alterna- tives: An evaluation of commonsense causal reason- ing. In 2011 AAAI Spring Symposium Series.
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Re- thinking the inception architecture for computer vi- sion. corr abs/1512.00567 (2015). · arXiv
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
- Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In NIPS.
Cited by
Sorted by citations
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Self-citationRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755
Self-citationCross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269
Xie, T., Wu, C. H., Shi, P., Zhong, R., Scholak, T., Yasunaga, M., Wu, C.-S., Zhong, M., Yin, P., Wang, S. I., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.39 · Cited by 271
Self-citationParameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169
Self-citationMulti-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System
Su, Y., Shu, L., Mansimov, E., Gupta, A., Cai, D., Lai, Y.-A., & Zhang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.319 · Cited by 173
Self-citationYang, S., Zhao, H., Zhu, S., Zhou, G., Hongfei, X., Jia, Y., & Zan, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29907 · Cited by 179
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Mishra, S. R., Mitra, A., Varshney, N., Sachdeva, B., Clark, P. A., Baral, C. R., & Kalyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.246 · Cited by 168
CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation
Shao, Y., Geng, Z., Liu, Y., Dai, J., Yan, H., Yang, F., Li, Z., Bao, H., & Qiu, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-021-3536-5 · Cited by 183
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 152
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_30 · Cited by 105
Zhang, Z., Yu, W., Yu, M., Guo, Z., & Jiang, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.66 · Cited by 108
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Singh, A., D’Arcy, M., Cohan, A., Downey, D., & Feldman, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.338 · Cited by 67
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 132
Meta-learning via Language Model In-context Tuning
Chen, Y., Zhong, R., Zha, S., Karypis, G., & He, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.53 · Cited by 143
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts
Asai, A., Salehi, M., Peters, M. E., & Hajishirzi, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.446 · Cited by 103
FitzGerald, J., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Delli Bovi, C., Cao, J., Chada, R., Chauhan, A., Chen, L., et al. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539173 · Cited by 68
BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems
Asyrofi, M. H., Yang, Z., Yusuf, I. N. B., Kang, H. J., Thung, F., & Lo, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3136169 · Cited by 81
A Comprehensive Survey of Abstractive Text Summarization Based on Deep Learning
Zhang, M., Zhou, G., Yu, W., Huang, N., & Liu, W. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7132226 · Cited by 75
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research
Gruetzemacher, R., & Paradice, D. (2022) · ACM Computing Surveys · doi:10.1145/3505245 · Cited by 68
Zero-Shot Text Classification with Self-Training
Gera, A., Halfon, A., Shnarch, E., Perlitz, Y., Ein‐Dor, L., & Slonim, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.73 · Cited by 68
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Xu, Z., Shen, Y., & Huang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.641 · Cited by 102
Alba, C., Xue, B., Abraham, J., Kannampallil, T., & Lu, C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01489-2 · Cited by 52
Zhang, C., Tian, L., & Chu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103507 · Cited by 62
Domain-matched Pre-training Tasks for Dense Retrieval
Oğuz, B., Lakhotia, K., Gupta, A., Lewis, P., Karpukhin, V., Piktus, A., Chen, X., Riedel, S., Yih, S., Gupta, S., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.114 · Cited by 50
Self-citationA taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.699 · Cited by 38
Bridging Remote Sensors with Multisensor Geospatial Foundation Models
Han, B., Zhang, S., Shi, X., & Reichstein, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02631 · Cited by 53
DesPrompt: Personality-descriptive prompt tuning for few-shot personality recognition
Wen, Z., Cao, J., Yang, Y., Wang, H., Yang, R., & Liu, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103422 · Cited by 31
PromptST: Prompt-Enhanced Spatio-Temporal Multi-Attribute Prediction
Zhang, Z., Zhao, X., Liu, Q., Zhang, C., Ma, Q., Wang, W., Zhao, H., Wang, Y., & Liu, Z. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615016 · Cited by 36
Qibo: A Large Language Model for traditional Chinese medicine
Jia, Y., Ji, X., Wang, X., Zhang, H., Meng, Z., Zhang, J., Chen, Z., Zhuang, P., Xu, D., Guo, W., et al. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127672 · Cited by 68
Assemble Foundation Models for Automatic Code Summarization
Gu, J., Salza, P., & Gall, H. C. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00112 · Cited by 34
Joint reasoning with knowledge subgraphs for Multiple Choice Question Answering
Zhang, Q., Chen, S., Fang, M., & Chen, X. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103297 · Cited by 26
Self-citationA survey of text classification based on pre-trained language model
Wu, Y., & Wan, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128921 · Cited by 43
Summarization, simplification, and generation: The case of patents
Casola, S., & Lavelli, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117627 · Cited by 30
Improving In-Context Few-Shot Learning via Self-Supervised Training
Chen, M., Du, J., Pasunuru, R., Mihaylov, T., Iyer, S. V., Stoyanov, V., & Kozareva, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.260 · Cited by 31
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 95
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
Vu, T., Luong, M.-T., Le, Q. V., Simon, G., & Iyyer, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.462 · Cited by 45
MVP: Multi-task Supervised Pre-training for Natural Language Generation
Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.558 · Cited by 24
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
Jia, R., Lewis, M., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.59 · Cited by 29
Self-citationA Thorough Evaluation of Task-Specific Pretraining for Summarization
Rothe, S., Maynez, J., & Narayan, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.12 · Cited by 25
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Parmar, M., Mishra, S. R., Purohit, M., Luo, M., Mohammad, M., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.10 · Cited by 38
Torbarina, L., Ferkovic, T., Roguski, Ł., Mihelčić, V., Šarlija, B., & Kraljević, Ž. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100076 · Cited by 29
Tc-llama 2: fine-tuning LLM for technology and commercialization applications
Yeom, J., Lee, H., Byun, H., Kim, Y., Byun, J., Choi, Y., Kim, S., & Song, K. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00963-0 · Cited by 21
IndicXNLI: Evaluating Multilingual Inference for Indian Languages
Aggarwal, D., Gupta, V., & Kunchukuttan, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.755 · Cited by 38
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
Padmakumar, V., Lausen, L., Ballesteros, M., Zha, S., He, H., & Karypis, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.183 · Cited by 19
Materials Informatics: Emergence to Autonomous Discovery in the Age of AI
Lookman, T., Liu, Y., & Gao, Z. (2026) · Advanced Materials · doi:10.1002/adma.202515941 · Cited by 20
Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences
Bensemann, J., Peng, A. Y., Benavides‐Prado, D., Chen, Y., Tan, N. Ö., Corballis, P. M., Riddle, P., & Witbrock, M. (2022) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2022.cmcl-1.9 · Cited by 17
Bajestani, M. S., Mun, D., & Kim, D. B. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.04.027 · Cited by 21
Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization
Liu, P., Huang, C., & Mou, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.545 · Cited by 16
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
Hou, Z., Salazar, J., & Polovets, G. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00517 · Cited by 18
Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity
Kung, P.-N., Yin, S.-S., Chen, Y., Yang, T.-H., & Chen, Y.-N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.34 · Cited by 16
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
Don-Yehiya, S., Venezian, E., Raffel, C., Slonim, N., & Choshen, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.46 · Cited by 44
Hu, X., Yu, S., Xiong, C., Liu, Z., Liu, Z., & Yu, G. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531786 · Cited by 14
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Huber, P., Aghajanyan, A., Oğuz, B., Okhonko, D., Yih, S., Gupta, S., & Chen, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.184 · Cited by 16
Self-citationGeographic Adaptation of Pretrained Language Models
Hofmann, V., Glavaš, G., Ljubešić, N., Pierrehumbert, J. B., & Schütze, H. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00652 · Cited by 22
A Comprehensive Survey on Summarization Techniques
Uppalapati, P. J., Dabbiru, M., & Rao, K. V. (2023) · SN Computer Science · doi:10.1007/s42979-023-02007-5 · Cited by 15
Single-Document Abstractive Text Summarization: A Systematic Literature Review
Rao, A. B., Aithal, S. G., & Singh, S. (2024) · ACM Computing Surveys · doi:10.1145/3700639 · Cited by 16
Probing BERT for Ranking Abilities
Wallat, J., Beringer, F., Anand, A., & Anand, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_17 · Cited by 17
Reducing the Cost: Cross-Prompt Pre-finetuning for Short Answer Scoring
Funayama, H., Asazuma, Y., Matsubayashi, Y., Mizumoto, T., & Inui, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_7 · Cited by 13
Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking
Moghe, N., Steedman, M. J., & Birch, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.87 · Cited by 13
Few-shot fine-tuning SOTA summarization models for medical dialogues
Navarro, D. F., Dras, M., & Berkovsky, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.32 · Cited by 12
MIGA: A Unified Multi-Task Generation Framework for Conversational Text-to-SQL
Fu, Y., Ou, W., Zhou, Y., & Lin, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26504 · Cited by 15
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
Yang, P., Wang, J., Gan, R., Zhu, X., Zhang, L., Wu, Z., Gao, X., Zhang, J., & Sakai, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.474 · Cited by 14
Soft Prompt Decoding for Multilingual Dense Retrieval
Huang, Z., Zeng, H., Zamani, H., & Allan, J. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591769 · Cited by 13
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Zhang, Z., Wang, S., Yi‐chong, X., Fang, Y., Yu, W., Liu, Y., Zhao, H. Y., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.416 · Cited by 17
Where to start? Analyzing the potential value of intermediate models
Choshen, L., Venezian, E., Don-Yehiya, S., Slonim, N., & Katz, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.90 · Cited by 23
Exploring Universal Intrinsic Task Subspace for Few-Shot Learning via Prompt Tuning
Qin, Y., Wang, X., Su, Y., Lin, Y., Ding, N., Yi, J., Chen, W., Liu, Z., Li, J., Hou, L., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430545 · Cited by 72
Sezgın, E., Sirrianni, J., & Kranz, K. (2023) · medRxiv · doi:10.1101/2023.12.06.23299573 · Cited by 10
Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead
Parthasarathi, S. H. K., Zeng, L., & Hakkani-Tur, D. Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096170 · Cited by 11
Bayrak, G., & Issifu, A. M. (2022) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/2022.wanlp-1.43 · Cited by 9
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple-Choice Perspective
Wang, J., Yang, P., Gan, R., Zhang, Y., Zhang, J., & Sakai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3343123 · Cited by 10
Design of superalloys with multiple properties via multi-task learning
Wang, W., Jiang, X. F., Li, W., Zhang, C., Liu, P., Tian, S., Lookman, T., & Su, Y. (2025) · Acta Materialia · doi:10.1016/j.actamat.2025.121161 · Cited by 11
Leveraging QA Datasets to Improve Generative Data Augmentation
Mekala, D., Vu, T., Schick, T., & Shang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.660 · Cited by 10
Multi-task Learning for Features Extraction in Financial Annual Reports
Montariol, S., Martinc, M., Pelicon, A., Pollak, S., Koloski, B., Lončarski, I., Valentinčič, A., Šuštar, K. S., Ichev, R., & Žnidaršič, M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23633-4_1 · Cited by 10
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts
Ye, Q., Zha, J., & Ren, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.189 · Cited by 19
CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations
Menon, R. R., Ghosh, S., & Srivastava, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.451 · Cited by 11
Analyzing Multi-Task Learning for Abstractive Text Summarization
Kirstein, F., Wahle, J. P., Ruas, T., & Gipp, B. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.5 · Cited by 9
Uncovering Inconsistencies and Contradictions in Financial Reports using Large Language Models
Deußer, T., Leonhard, D., Hillebrand, L., Berger, A., Khaled, M. B., Heiden, S., Dilmaghani, T., Kliem, B., Loitz, R., Bauckhage, C., et al. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386673 · Cited by 8
Wu, B., Razuvayevskaya, O., Heppell, F., Leite, J. A., Scarton, C., Bontcheva, K., & Song, X. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.275 · Cited by 14
Exploring Mode Connectivity for Pre-trained Language Models
Qin, Y., Qian, C., Yi, J., Chen, W., Lin, Y., Xu, H., Liu, Z., Sun, M., & Zhou, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.451 · Cited by 20
Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPPreprint
Rozner, J., Potts, C., & Mahowald, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08620 · Cited by 11
Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning
Zhao, H., Fu, J., & He, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.280 · Cited by 11
Enhancing Low-resource Fine-grained Named Entity Recognition by Leveraging Coarse-grained Datasets
Lee, S. J., Oh, S., & Jung, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.197 · Cited by 6
A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction
Hu, Z., Ni, Z., Shi, J. Y., Xu, S., & Xu, B. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1461-5 · Cited by 6
Penha, G., & Hauff, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28241-6_9 · Cited by 6
Multitask Pretraining with Structured Knowledge for Text-to-SQL Generation
Giaquinto, R. A., Zhang, D., Kleiner, B., Li, Y., Tan, M., Bhatia, P., Nallapati, R. M., & Ma, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.620 · Cited by 7
Hexatagging: Projective Dependency Parsing as Tagging
Amini, A., Liu, T., & Cotterell, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.124 · Cited by 6
Multi Self-Supervised Pre-Finetuned Transformer Fusion for Better Vehicle Detection
Zheng, J., & Ren, J. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3374759 · Cited by 6
DaCy: A Unified Framework for Danish NLPPreprint
Enevoldsen, K., Hansen, L., & Nielbo, K. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05295 · Cited by 13
ParaNet:Parallel Networks with Pre-trained Models for Text Classification
Wu, Y., Guo, X. Y., Wei, Y., & Chen, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46671-7_9 · Cited by 6
Self-citationImproving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task
Chen, C.-C., Takamura, H., Kobayashi, I., & Miyao, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.4 · Cited by 5
DFM: Dialogue foundation model for universal large-scale dialogue-oriented task learning
Chen, Z., Ma, D., Li, H., Chen, L., Ji, J., Liu, Y., Chen, B., Wu, M., Zhu, S., Dong, X., et al. (2025) · AI Open · doi:10.1016/j.aiopen.2025.04.001 · Cited by 11
Locally Aggregated Feature Attribution on Natural Language Model Understanding
Zhang, S., Wang, J., Jiang, H., & Song, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.159 · Cited by 5
Pre-Finetuning for Few-Shot Emotional Speech Recognition
Chen, M., & Zhou, Y. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-136 · Cited by 7
Learning to Generalize Compositionally by Transferring Across Semantic Parsing TasksPreprint
Wang, Z., Shaw, P. M., Linzen, T., & Sha, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05013 · Cited by 12
Metadata
- Authors
- Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S.
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 5799-5811
- OpenAlex
- W3124687886
- Cited by
- 183
- References
- 82
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 23 · 2023: 64 · 2022: 55 · 2021: 13
- FWCI
- 17.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multi-task learning, representation learning, language model pretraining, fine-tuning, sample efficiency, RoBERTa, BART, commonsense reasoning, machine reading comprehension, sentence prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
