The DOI matches its DataCite record.
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Open accessAbstract
Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A consistent finding across studies is that fine-tuning LMs on a range of NLP tasks, with instructions, improves their downstream performance on held-out tasks, both in the zero-shot and few-shot settings.
“A consistent finding across studies is that fine-tuning LMs on a range of NLP tasks, with instructions, improves their downstream performance on held-out tasks, both in the zero-shot and few-shot settings.”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Mishra et al., 2021
Section 5.2 and the results on instruction-based generalization report that BART was fine-tuned on seen tasks and performed better on held-out tasks when given instructions. But §5.2 describes GPT-3 as the model used for few-shot experiments, and the GPT3 section says it was not fine-tuned. This paper therefore does not establish the claimed fine-tuning benefit in both zero-shot and few-shot settings. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 38
- Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of EMNLP, pages 5799-5811.
- Ali M Ali. 1981. The use of positive and negative ex- amples during instruction. Journal of instructional development, 5(1):2-7.
- Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS, volume 33, pages 1877-1901.
- Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
- Pradeep Dasigi, Nelson F Liu, Ana Marasovic, Noah A Smith, and Matt Gardner. 2019. Quoref: A read- ing comprehension dataset with questions requiring coreferential reasoning. In Proceedings of EMNLP- IJCNLP, pages 5927-5934.
- Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of NAACL, pages 2368-2378.
- Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982. · arXiv
- Tanmay Gupta, A. Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2021. Towards general purpose vi- sion systems. ArXiv, abs/2104.00743. · arXiv
- Peter Hase and Mohit Bansal. 2021. When can mod- els learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201. · arXiv
- Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of EMNLP-IJCNLP, pages 2391-2401.
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking be- yond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of NAACL, pages 252-262.
- Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what is essential in ques- tions. In Proceedings of CoNLL, pages 80-89.
- Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UnifiedQA: crossing format boundaries with a single qa system. In Proceedings of EMNLP: Findings, pages 1896-1907.
- Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence compo- sition. In Proceedings of AAAI.
- Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? In Proceedings of NAACL-HLT, pages 2627-2636.
- Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of ACL.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019. Reasoning over paragraph effects in situ- ations. In Proceedings of the 2nd Workshop on Ma- chine Reading for Question Answering, pages 58- 62.
- Winston Lin, Roman Yangarber, and Ralph Grishman. 2003. Bootstrapped learning of semantic classes from positive and negative examples. In Proceed- ings of ICML Workshop on The Continuum from La- beled to Unlabeled Data, volume 1, page 21.
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. · arXiv
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcom- ing few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. · arXiv
- Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. 2020. To- wards question format independent numerical rea- soning: A set of prerequisite tasks. arXiv preprint arXiv:2005.08516. · arXiv
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of NAACL-HLT, pages 2227-2237.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
- Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Com- puting Systems, pages 1-7.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. Winogrande: An adver- sarial winograd schema challenge at scale. In Pro- ceedings of the AAAI.
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Tae- woon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Ab- heesht Sharma, Andrea Santilli, Thibault Fevry, Ja- son Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexan- der M Rush. 2022. Multitask prompted training en- ables zero-shot task generalization. In Proceedings of ICLR.
- Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. In Proceedings of EMNLP.
- Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text gen- eration. In Proceedings of ACL, pages 7881-7892.
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In Proceedings of ICLR.
- Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew Peters. 2020. Learning from task descrip- tions. In Proceedings of EMNLP, pages 1361-1375.
- Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. 2020. Hard negative examples are hard, but useful. In Proceedings of ECCV, pages 126-142. Springer.
- Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross- task generalization in nlp. In Proceedings of EMNLP.
- Qinyuan Ye and Xiang Ren. 2021. Zero-shot learning by generating task-specific adapters. arXiv preprint arXiv:2101.00420. · arXiv
- Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of ICML, pages 12697-12706.
- Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero- shot learning by meta-tuning on dataset and prompt collections. In Proceedings of EMNLP: Findings, pages 2856-2878.
- Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "going for a walk": A study of temporal common- sense understanding. In Proceedings of EMNLP- IJCNLP, pages 3354-3360.
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“Most recently, Mishra et al. (2021) finetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks—this is very similar to our few-shot instruction tuning result from §4.3.”
MethodResultInfluentialRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755
Self-citationA Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationDo Prompt-Based Models Really Understand the Meaning of Their Prompts?Preprint
Webson, A., & Pavlick, E. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.01247 · Cited by 522
“At a high-level, our findings contradict Mishra et al. (2021)’s claim that models benefit from elaborate instructions adapted from crowdsourcing annotation guides.”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
PandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16355 · Cited by 340
“With alignment techniques such as supervised instruction tuning [13, 21, 28] and reinforcement learning from human feedback [16, 23], LLMs exhibit surprisingly effective zero-and few-shot generalization abilities to perform almost any NLP tasks.”
MethodContinual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2025) · ACM Computing Surveys · doi:10.1145/3735633 · Cited by 282
“After discussing general observations on CFT in Section 4.3.1, we will delve into each sub-category in detail.”
InfluentialZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J.-H., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2202.07922 · Cited by 324
“This motivates researchers to investigate better ways to automatically search and/or manually construct a proper prompt (Jiang et al., 2020a; Shin et al., 2020; Reynolds and McDonell, 2021; Mishra et al., 2021b).”
BackgroundReframing Instructional Prompts to GPTk’s LanguagePreprint
Mishra, S., Khashabi, D., Baral, C., Choi, Y., & Hajishirzi, H. (2021) · Findings · doi:10.48550/arxiv.2109.07830 · Cited by 287
“On the other hand, task instructions written by non-expert users are often long and contain abstractive descriptions (Mishra et al., 2021) which”
BackgroundMethodInfluentialSelf-citationToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2203.09509 · Cited by 276
“While there are several approaches for prompting pretrained LLMs (Liu et al., 2021b), a recent and promising direction is demonstration-based prompting (Gao et al., 2021; Mishra et al., 2021).”
BackgroundLIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275
“To enable this transfer, various methods for aligning language models have thus been proposed, primarily focusing on instruction tuning [Mishra et al., 2021, Wei et al., 2022a, Sanh et al., 2022] over large multi-million-example datasets [Chung et al., 2022, Beeching et al., 2023, Köpf et al.,…”
MethodAdapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 152
“Our approach is complementary to that of Weller et al. (2020), which asks turkers to generate tasks, and that of Mishra et al. (2021), which generates tasks by decomposing existing templates used to construct reading comprehension datasets.”
BackgroundPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160
“For the latter, examples of reasoning are typically used in the larger context of instruction tuning (Mishra et al., 2021; Sanh et al., 2022; Wang et al., 2022d; Ouyang et al., 2022), where, more generally, an LM’s ability to follow instructions is improved based on human-labeled examples.”
BackgroundMethodK2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization
Deng, C., Zhang, T., He, Z., Chen, Q., Shi, Y., Xu, Y. X., Fu, L., Zhang, W., Wang, X., Zhou, C., et al. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635772 · Cited by 104
“• Natural Instruction: Natural Instruction [33] maintains many We select objective tasks elaborately from the v2.x dataset and organize them into <instruction, input, output> format.”
BackgroundCan language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2024) · 2024 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00018 · Cited by 114
Large language models for chemistry robotics
Yoshikawa, N., Skreta, M., Darvish, K., Arellano-Rubach, S., Ji, Z., Bjørn Kristensen, L., Li, A. Z., Zhao, Y., Xu, H., Kuramshin, A., et al. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10136-2 · Cited by 112
Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering
Luo, M., Zeng, Y., Banerjee, P., & Baral, C. R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.517 · Cited by 74
Self-citationWang, Y.-F., Liu, Y., Shi, C., Li, H.-L., Chen, C., Lu, H., & Yang, Y.-J. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2403.11435 · Cited by 90
“After being unsupervised pre-trained on large-scale raw text, LLMs are further trained via instruction tuning to generate appropriate outputs based on the given input instructions (Sanh et al., 2021; Mishra et al., 2021; Chung et al., 2022).”
MethodMeasuring Inductive Biases of In-Context Learning with Underspecified Demonstrations
Si, C., Friedman, D., Joshi, N., Feng, S., Chen, D., & He, H. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.13299 · Cited by 72
“We format our instructions mostly following prior work such as Natural Instructions (Mishra et al., 2021; Wang et al., 2022), and we list all our instructions in Tables 7 and 8.”
MethodRAFT: A Real-World Few-Shot Text Classification BenchmarkPreprint
Alex, N., Lifland, E., Tunstall, L., Thakur, A., Maham, P., Riedel, C. J., Hine, E., Ashurst, C., Sedille, P., Carlier, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14076 · Cited by 62
“The most closely related few-shot NLP benchmarks are FLEX [5], FewGLUE [29], CrossFit [37], and NaturalInstructions [21].”
InfluentialInstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskénazi, M., & Bigham, J. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.33 · Cited by 73
“Most recent work (Mishra et al., 2021; Sanh et al., 2022; Wei et al., 2022) on instruction tuning has focused on general NLP tasks such as para-phrase detection and reading comprehension, but not specifically on dialogue.”
BackgroundMethodInfluentialCognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61
“Inspired by the basic rule of machine learning models "Garbage input, garbage output", [147] proposes that data quality and diversity outweigh the importance of fine-tuning large-scale instructions [148, 3, 149] and RLHF [6, 2].”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
Zhao, J.-Y., Khashabi, D., Khot, T., Sabharwal, A., & Chang, K.-W. (2021) · Findings · doi:10.48550/arxiv.2106.01465 · Cited by 61
“This is also remotely relevant to the literature on learning from instructions which expect a model to adapt its behavior according declarative instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021).”
BackgroundSelf-citationCINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems
Mi, F., Li, Y., Wang, Y., Jiang, X., & Liu, Q. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2109.04645 · Cited by 47
“As “prompts” are often short and concise, Mishra et al. (2021) proposed to encode extra task-specific instructions to generalize to new tasks.”
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Yang, D.-K., Wei, J.-J., Xiao, D., Wang, S.-L., Wu, T., Li, G., Li, M.-C., Wang, S., Chen, J., Jiang, Y., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.19266 · Cited by 41
Learning to Generate Task-Specific Adapters from Task Description
Ye, Q., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.82 · Cited by 26
“, 2020) and instructions (Mishra et al., 2021) can be considered as a sub-category in zeroshot learning, with the goal of generalizing to unseen tasks during inference.”
BackgroundExploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Shi, C., Su, Y.-X., Yang, C., Yang, Y., & Cai, D. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.15326 · Cited by 38
“…capabilities) is predominantly acquired during large-scale unsupervised pre-training; and (2) instruction tuning (Sanh et al., 2022; Wei et al., 2022a; Mishra et al., 2021; Ouyang et al., 2022) is an incredibly data-efficient method for unleashing the power of LLMs to complete realistic NLP tasks.”
BackgroundMethodData-Efficient Finetuning Using Cross-Task Nearest Neighbors
Ivison, H., Smith, N. A., Hajishirzi, H., & Dasigi, P. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.00196 · Cited by 38
Self-citationQiu, X., Hao, T., Shi, S., Tan, X., & Xiong, Y.-J. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3377590 · Cited by 28
“Thisoptimiza-tion process is identical to the typical IF, in which we provide instructions as input and obtain the corresponding output from LLMs [17], [18].”
MethodCasModaTest: A Cascaded and Model-agnostic Self-directed Framework for Unit Test Generation
Ni, C., Wang, X., Chen, L., Zhao, D.-H., Cai, Z.-G., Wang, S.-H., & Yang, X. (2024) · arXiv.org · doi:10.48550/arxiv.2406.15743 · Cited by 24
“The few-shot learning [23], [61], [62] and the instruction tuning [63]–[65] are the most effective ways in prompt engineering.”
BackgroundAre Language Models Worse than Humans at Following Prompts? It's Complicated
Webson, A., Loo, A., Yu, Q., & Pavlick, E. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2301.07085 · Cited by 21
Are the BERT family zero-shot learners? A study on their potential and limitations
Wang, Y., Wu, L., Li, J., Liang, X., & Zhang, M. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103953 · Cited by 13
Rosa, G., Bonifacio, L., Jeronymo, V., Abonizio, H., Lotufo, R., & Nogueira, R. (2022) · arXiv.org · doi:10.48550/arxiv.2205.15172 · Cited by 15
“[13] fine-tuned a BART model with 140 million parameters using samples containing instructions written in natural language and evaluated the model few-shot ability on unseen tasks.”
MethodDeconstructing In-Context Learning: Understanding Prompts via Corruption
Shivagunde, N., Lialin, V., Muckatira, S., & Rumshisky, A. (2024) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2404.02054 · Cited by 9
“For example, Webson and Pavlick (2021) suggest that relevant and irrelevant instructions in the prompt yield similar model performance, whereas Mishra et al. (2021b) and Reynolds and McDonell (2021) argued the opposite.”
BackgroundSong, C., Shao, T., Lin, K., Liu, D., Wang, S., & Chen, H. (2022) · Applied Sciences · doi:10.3390/app122111117 · Cited by 8
A Cascaded Pipeline for Self-Directed, Model-Agnostic Unit Test Generation via LLMs
Ni, C., Wang, X., Yin, X., Chen, L., & Ma, G. (2025) · IEEE International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00037 · Cited by 5
You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language ModelsPreprint
He, W., Li, P., Jiang, Z., & Go, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.11103 · Cited by 3
“Additionally, several studies have emphasized the importance of instruction-based methods (Mishra et al., 2021), (Wei et al., 2022).”
Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models
Albalak, A., Shrivastava, A., Sankar, C., Sagar, A., & Ross, M. (2022) · arXiv.org · doi:10.48550/arxiv.2210.03871 · Cited by 4
“"topic" for topic classification) they have expanded to include answer templates, examples, and instructions (Raffel et al., 2020; Albalak et al., 2022; Wei et al., 2022a; Mishra et al., 2021; Ouyang et al., 2022).”
BackgroundCommonsense Reasoning with Implicit Knowledge in Natural Language
Banerjee, P., Mishra, S., Pal, K. K., Mitra, A., & Baral, C. (2021) · Conference on Automated Knowledge Base Construction · doi:10.24432/c57c7h · Cited by 4
“We hope our findings will help design models that respond better to instructions [Mishra et al., 2021] containing knowledge expressed in natural language.”
MethodSelf-citationBosley, M. (2025) · Journal of Political Institutions and Political Economy · doi:10.1561/113.00000128 · Cited by 3
“Researchers have also developed techniques for optimizing prompts to achieve specific behaviors, such as reducing harmful outputs (Ouyang et al., 2022), improving truthfulness (Lin et al., 2021), and following instructions (Mishra et al., 2022).”
Wang, H.-H., Qi, J., Chen, J.-P., Wu, J., Huang, L., Zheng, L., Choi, K., Veeramani, B., Bowen, E., Hu, A., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.13640 · Cited by 2
“We use the Natural-Instructions dataset [45] as the real dataset and the Magpie-Pro-1M dataset [46] as the synthetic dataset.”
RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection
Yang, Y.-X., Dong, Q., Yao, L.-L., Zhu, F., Luo, W.-L., Wang, B., & Sui, Z.-F. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2505.05327 · Cited by 2
“Instruction tuning (Liu et al. 2023a; Xu et al. 2024b) has emerged as a powerful approach to enhance the performance of such models by finetuning them to better follow human instructions.”
On Conditional and Compositional Language Model Differentiable Prompting
Pilault, J., Liu, C., Bansal, M., & Dreyer, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2307.01446 · Cited by 2
Prompt-Based Learning for Thread Structure Prediction in Cybersecurity Forums
Kashihara, K., Pal, K. K., Baral, C., & Trevino, R. P. (2023) · Intelligent Systems with Applications · doi:10.48550/arxiv.2303.05400 · Cited by 2
“In contrast, the recent works [16,17] give instructions encode detailed instructions as they were used to collect the dataset.”
ResultSelf-citationThe Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
Multi-Feature Enhanced Multimodal Action Recognition
Zhu, Q., Gu, Z., He, H., Yu, Z., Li, C., Zhao, T., & Xu, Z. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394285 · Cited by 1
“Mishra et al. [20] expands the method to perform training for 61 tasks on 9 datasets.”
MethodWhat is Not in the Context? Evaluation of Few-shot Learners with Informative Demonstrations
Štefánik, M., & Kadlcík, M. (2022) · arXiv.org · doi:10.48550/arxiv.2212.01692 · Cited by 1
“Such task description can take the form of instructions in natural language, similar or identical to the instructions given to human annotators (Mishra et al., 2021), or a set of demonstrations , i.e. the input-output mappings of the task to learn (Brown et al., 2020).”
BackgroundExploring the Capabilities of LLMs for Code Change Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2024) · arXiv · Cited by 41
Memory-assisted prompt editing to improve GPT-3 after deployment
Madaan, A., Tandon, N., Clark, P., & Yang, Y.-M. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2201.06009
“Learning from instruction has become important for large LMs that can perform a task based on direct instruction rather than examples (Wei et al., 2021; Mishra et al., 2021).”
BackgroundBeyond One-Size-Fits-All: Personalized Harmful Content Detection with In-Context LearningPreprint
Zhang, R., Zhang, L., & Mi, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.05532
“Finally, the addition of rationales—natural language explanations paired with labels—has been shown to enhance reasoning and reduce error propagation [36, 44, 58], providing interpretive signals that guide the model beyond surface-level pattern recognition.”
Cui, J.-J., Huang, M., Xu, Y., & Guo, N.-W. (2025) · Other Conferences · doi:10.1117/12.3060306
Taxonomy-based CheckList for Large Language Model EvaluationPreprint
Zhang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.10899
“Clark, Tafjord, and Richardson demonstrate models’ ability to take in rules and engage in soft reasoning, relevant to model adapting its behavior according to declarative instructions (Weller et al. 2020; Efrat and Levy 2020; Mishra et al. 2021).”
BackgroundInterviewer-Candidate Role Play: Towards Developing Real-World NLP SystemsPreprint
Varshney, N., Mishra, S. R., & Baral, C. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00315
“…al., 2020; Schick and Schütze, 2021b; Ye et al., 2021; Tam et al., 2021), Prompting (Shin et al., 2020; Jiang et al., 2020; Le Scao and Rush, 2021; Mishra et al., 2021), etc. Selective Prediction enables a system to maintain high accuracy by abstaining on instances where it is likely to be…”
Self-citation
Metadata
- Authors
- Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H.
- Affiliations
- Arizona State University (US)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3470-3487
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2104.08773
- OpenAlex
- W3196731672
- Cited by
- 269
- References
- 0
- Citations by year
- 2026: 21 · 2025: 59 · 2024: 55 · 2023: 95 · 2022: 38 · 2021: 1
- FWCI
- 34.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2044660)
- Defense Advanced Research Projects Agency
- Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- cross-task generalization, natural language instructions, instruction tuning, meta-dataset, generalization to unseen tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
