CoChat
Sign Up
PreprintCited by 612Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

Multitask Prompted Training Enables Zero-Shot Task Generalization

Open access
Sanh, Victor, Webson, Albert, Raffel, Colin, Bach, Stephen H., Sutawika, Lintang, Alyafeai, Zaid, Chaffin, Antoine, Stiegler, Arnaud +33 more·2021·arXiv
DOI 10.48550/arxiv.2110.08207arXiv 2110.08207
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2110.08207

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Designing effective prompting strategies requires identifying contexts in which LLM errors arise, devising prompting strategies to overcome them, and systematically assessing those strategies' effectiveness.

    “While prompting LLMs can appear efortless, designing efective prompting strategies requires identifying the contexts in which these LLMs' errors arise, devising prompting strategies to overcome them, and systematically assessing those strategies' efectiveness.”

    From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [42]

    Section 6.2 (p. 7) tests whether training with more prompts improves robustness to prompt wording, and the Experimental Setup assesses performance across prompts. This supports systematically evaluating prompt choices, but the paper does not describe a process of identifying contexts in which LLM errors arise and devising prompts to overcome those errors. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Prompt design tools to date have focused on supporting professional programmers and NLP practitioners rather than non-AI experts, non-programmers, and other potential end-users.

    “Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”

    From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [42]

    Appendix C describes PromptSource as a tool for writing prompts for NLP datasets, using Jinja templates and Python-dictionary examples. That supports citing this paper as an example of an NLP-oriented prompt-design tool, but it does not explicitly identify NLP practitioners as its intended users or establish that such tools generally exclude non-experts and other end-users. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • The scaling up of network and training data leads to leap-forward capabilities in LLMs, including emergent abilities such as in-context learning, instruction following, and step-by-step reasoning.

    “The scaling up of network and training data lead to the leap-forward of large language models' capability.”

    From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [36]

    The Introduction links large pretraining corpora to possible task exposure, and the Abstract reports that large language models can generalize to unseen tasks. But the paper does not establish that scaling both network size and training data causes a leap in capability, nor does it support the claim’s specific list of in-context learning, instruction following, and step-by-step reasoning. Its Conclusion instead emphasizes gains from multitask prompted training, including gains over larger models. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Context-aware modeling faces challenges such as handling dynamically changing contexts, balancing the significance of different modality data, and preventing overfitting.

    “Context-aware modeling faces several challenges, such as handling dynamically changing contexts [223], balancing the significance of different modality data [224], and preventing overfitting.”

    From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [224]

    Citation [223] is attached to the challenge of handling dynamically changing contexts. The Related Work section discusses multitask NLP prompting, and the retrieved QA prompt templates use “context” as an input field; neither addresses contexts changing dynamically. Searches for dynamically changing and adapting to changing contexts found no supporting passage. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 79

  1. Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785, 2019. · arXiv
  2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pages 6–4. Venice, 2006.
  3. Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678, 2020. doi: 10.1162/tacl a 00338. URL https://doi.org/10.1162/tacl a 00338. · DOI
  4. Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In EMNLP, 2019.
  5. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1160.
  6. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
  7. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Informa- tion Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. CoRR, abs/1905.10044, 2019. URL http://arxiv.org/abs/1905.10044. · arXiv
  9. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Filnand, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 160–167. ACM, 2008. doi: 10.1145/1390156.1390177. URL https: //doi.org/10.1145/1390156.1390177. · DOI · Paper page
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop, pages 177–190. Springer, 2005.
  11. Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. arXiv:1908.05803v2, 2019. · arXiv
  12. Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11–20, Brussels, Belgium, October 2018. Association for Computational Linguis- tics. doi: 10.18653/v1/W18-5102. URL https://www.aclweb.org/anthology/W18-5102. · DOI
  13. William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
  14. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenodo.5371628. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf https://doi.org/10.1023/A:1007379606734 https://aclanthology.org/D18-1241 http://arxiv.org/abs/1905.10044 https://doi.org/10.1145/1390156.1390177 https://doi.org/10.1145/1390156.1390177 https://www.aclweb.org/anthology/W18-5102 https://doi.org/10.5281/zenodo.5371628 · DOI
  15. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics, 2007.
  16. Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009.
  17. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Con- sortium, Philadelphia, 4(1):34, 2003.
  18. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/1611.015collin87, 2016. URL http://arxiv.org/abs/1611.01587. · arXiv
  19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701, 2015.
  20. Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL https://aclanthology.org/H01-1069.
  21. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In arXiv:1909.00277v2, 2019. · arXiv
  22. Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. arXiv:1707.06209v1, 2017. · arXiv
  23. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Pro- ceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
  24. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online, November 2020b. Association for Computational Linguistics. doi: 10.18653/v1/2020. findings-emnlp.171. URL https://aclanthology.org/2020.findings-emnlp.171. · DOI
  25. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2, 2020. · arXiv
  26. Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. What changes can large- scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. arXiv preprint arXiv:2109.04650, 2021. · arXiv
  27. Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with appli- cation to the biography domain. CoRR, abs/1603.07771, 2016. URL http://arxiv.org/abs/1603. 07771. · arXiv
  28. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, S"oren Auer, et al. Dbpedia–a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195, 2015.
  29. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
  30. Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https://aclanthology.org/C02-1150.
  31. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.findings-emnlp.165. URL https://aclanthology.org/2020.findings-emnlp.165.
  32. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. In MRQA@EMNLP, 2019.
  33. Robert L Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353, 2021. · arXiv
  34. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142– 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www.aclweb.org/anthology/P11-1015.
  35. Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172, 2013.
  36. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730, 2018. URL http: //arxiv.org/abs/1806.08730. · arXiv
  37. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. CoRR, abs/1902.01007, 2019. URL http://arxiv.org/abs/ 1902.01007. · arXiv
  38. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. http://arxiv.org/abs/1603.07771 http://arxiv.org/abs/1603.07771 https://arxiv.org/abs/2104.08691 https://aclanthology.org/C02-1150 https://aclanthology.org/2020.findings-emnlp.165 http://www.aclweb.org/anthology/P11-1015 http://www.aclweb.org/anthology/P11-1015 http://arxiv.org/abs/1806.08730 http://arxiv.org/abs/1806.08730 http://arxiv.org/abs/1902.01007 http://arxiv.org/abs/1902.01007 · arXiv
  39. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Natural instruc- tions: Benchmarking generalization to new tasks from natural language instructions. CoRR, abs/2104.08773, 2021. URL https://arxiv.org/abs/2104.08773. · arXiv
  40. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
  41. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
  42. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, 2016.
  43. David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. · arXiv
  44. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. CoRR, abs/2105.11447, 2021. URL https://arxiv.org/abs/2105.11447. · arXiv
  45. Mohammad Taher Pilehvar and os'e Camacho-Collados. Wic: 10, 000 example pairs for evaluating context-sensitive representations. CoRR, abs/1808.09121, 2018. URL http://arxiv.org/abs/1808. 09121. · arXiv
  46. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 67–81, Brussels, Belgium, October-November 2018. Associa- tion for Computational Linguistics. doi: 10.18653/v1/D18-1007. URL https://aclanthology.org/ D18-1007. · DOI
  47. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.467. URL https: //aclanthology.org/2020.acl-main.467. · DOI
  48. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  49. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 5418–5426, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.437. URL https://aclanthology.org/2020.emnlp-main.437. · DOI · Paper page
  50. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
  51. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to AI com- plete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial In- telligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artifi- cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8722–8731. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6398.
  52. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  53. Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015. doi: 10.18653/v1/d15-1044. URL http://dx.doi.org/10.18653/v1/ D15-1044. · DOI
  54. Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. In Meeting of the Association for Computational Linguistics (ACL), 2018.
  55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. URL http://arxiv. org/abs/1907.10641. · arXiv
  56. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54–63, 2020.
  57. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer- generator networks. CoRR, abs/1704.04368, 2017. URL http://arxiv.org/abs/1704.04368. · arXiv
  58. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018.
  59. Reddy Siva, Chen Danqi, and Manning Christopher D. Wikiqa: A challenge dataset for open-domain question answering. arXiv, 2018.
  60. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. Release strategies and the social impacts of language models. CoRR, abs/1908.09203, 2019. URL http://arxiv.org/abs/1908.09203. · arXiv
  61. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, 2019. https://arxiv.org/abs/2102.07350 https://aclanthology.org/2020.emnlp-main.437 https://aaai.org/ojs/index.php/AAAI/article/view/6398 http://dx.doi.org/10.18653/v1/D15-1044 http://dx.doi.org/10.18653/v1/D15-1044 http://arxiv.org/abs/1907.10641 http://arxiv.org/abs/1907.10641 https://aclanthology.org/2021.eacl-main.20 http://arxiv.org/abs/1704.04368 http://arxiv.org/abs/1908.09203 · DOI
  62. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 2019. URL https://arxiv.org/abs/1902.00164v1. · arXiv
  63. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. "quartz: An open-domain dataset of qualitative relationship questions". EMNLP, "2019".
  64. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: A dataset and models for answering questions about qualitative relationships. CoRR, abs/1811.08048, 2018. URL http://arxiv.org/abs/1811.08048. · arXiv
  65. Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7882–7926, Online, November 2020. Association for Com- putational Linguistics. doi: 10.18653/v1/2020.emnlp-main.635. URL https://aclanthology.org/ 2020.emnlp-main.635. · DOI
  66. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537, 2019a. URL http://arxiv.org/abs/1905.00537. · arXiv
  67. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2019b. In the Proceedings of ICLR.
  68. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner. CoRR, abs/2104.14690, 2021. URL https://arxiv.org/abs/2104.14690. · arXiv
  69. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. · arXiv
  70. Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts?, 2021. URL https://arxiv.org/abs/2109.01247. · arXiv
  71. Adina Williams, Tristan Thrush, and Douwe Kiela. Anlizing the adversarial natural language infer- ence dataset. arXiv preprint arXiv:2010.12729, 2020. · arXiv
  72. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835, 2021. URL https://arxiv.org/abs/2104. 08835. · arXiv
  73. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. · arXiv
  74. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classi- fication. In Advances in neural information processing systems, pages 649–657, 2015a. https://arxiv.org/abs/1902.00164v1 http://arxiv.org/abs/1811.08048 https://aclanthology.org/2020.emnlp-main.635 https://aclanthology.org/2020.emnlp-main.635 http://arxiv.org/abs/1905.00537 https://arxiv.org/abs/2104.14690 https://arxiv.org/abs/2109.01247 https://arxiv.org/abs/2104.08835 https://arxiv.org/abs/2104.08835 · arXiv
  75. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015b.
  76. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scram- bling. In Proc. of NAACL, 2019.
  77. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2003. URL https://aclanthology.org/N18-2003. · DOI · Paper page
  78. Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. CoRR, abs/2104.04670, 2021. URL https://arxiv.org/abs/2104.04670. · arXiv
  79. 4https://bigscience.huggingface.co/ https://aclanthology.org/N18-2003 https://arxiv.org/abs/2104.04670 https://bigscience.huggingface.co/ · arXiv

Cited by

Sorted by citations

Authors
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Bers, T., Biderman, S., Gao, L., Wolf, T., & Rush, A. M.
Affiliations
  • Brown University (US)
  • King Fahd University of Petroleum and Minerals (SA)
  • Institut de Recherche en Informatique et Systèmes Aléatoires (FR)
  • Hugging Face (US)
  • Institute for Infocomm Research (SG)
  • Systems, Applications & Products in Data Processing (Germany) (DE)
  • Nanyang Technological University (SG)
  • University of California San Diego (US)
  • Vrije Universiteit Amsterdam (NL)
  • University of Virginia (US)
  • Asus (Taiwan) (TW)
  • New York University (US)
  • IBM Research - Zurich (CH)
  • University of California, Berkeley (US)
  • ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnelles (FR)
  • Birla Institute of Technology and Science, Pilani (IN)
  • Stanford University (US)
  • Charles River Analytics (United States) (US)
  • Booz Allen Hamilton (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2110.08207
arXiv
2110.08207
OpenAlex
W3205068155
Cited by
612
References
26
Citations by year
2026: 6 · 2025: 57 · 2024: 132 · 2023: 326 · 2022: 90 · 2021: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • IRIS Research product catalog (Sapienza University of Rome) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Agence Nationale de la Recherche (ANR‐19, ANR-19-P3IA-0001)
  • Centre National de la Recherche Scientifique
  • Grand Équipement National De Calcul Intensif
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
zero-shot generalization, multi-task learning, prompt engineering, large language models, BIG-Bench, task generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review