CoChat
Sign Up
PreprintCited by 226Domain Adaptation and Few-Shot LearningTopic ModelingNatural Language Processing Techniques

True Few-Shot Learning with Language Models

Open access
Perez, Ethan, Kiela, Douwe, Cho, Kyunghyun·2021·arXiv
DOI 10.48550/arxiv.2105.11447arXiv 2105.11447
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.11447

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Task disclosure conditions model behavior on task identity, encouraging task-specific response strategies instead of implicit recall from prior dialogue.

    “This formulation conditions model behavior on task identity, encouraging task-specific response strategies instead of implicit recall from prior dialogue.”

    From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Perez et al., 2021

    Section 3 (p. 4) says the model conditions its completion on earlier words in a task prompt, and §3.8 (p. 8) describes using different prompts for different tasks. These passages support task-dependent prompting, but they do not discuss task disclosure as a way to encourage response strategies instead of implicit recall from prior dialogue. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The transferability of prompts is poor when the model sizes are quite different in both tuned and true few-shot learning scenarios.

    “The transferability is poor when the model sizes are quite different in both scenarios.”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [96]

    Section 3.7 (p. 8) says prompts selected by test accuracy generalize reasonably well between similarly sized models, and CV-selected prompts transfer better to similar sizes. But it says MDL-selected prompts perform similarly across many model sizes, so poor transfer between very different sizes does not hold for every true few-shot selection method. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • A development set is not desirable in a few-shot setting where the size of the development set is very limited, even unavailable.

    “However, this is not desirable in a few-shot setting where the size of the development set is very limited, even unavailable (Perez et al., 2021) .”

    From Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2022) · cites this paper as Perez et al., 2021

  • Previous work overestimates the few-shot ability of language models in the true few-shot learning setting.

    “Experimental result suggested that previous work overestimate the few-shot ability of LMs in this (true few-shot learning) setting.”

    From Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2022) · cites this paper as Perez et al. (2021)

  • This practice is arguably not true zero-shot learning with respect to the downstream task.

    “As noted by Perez et al. [96], this is arguably not true zero-shot learning with respect to the downstream task.”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [96]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 66

  1. [5][5] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 , 2020. - · arXiv
  2. [8][8] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv , abs/2101.06804, 2021. - · arXiv
  3. [9][9] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. Computing Research Repository , arXiv:2009.07118, 2020. - · arXiv
  4. [10][10] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. Rissanen data analysis: Examining dataset characteristics via description length. In ICML , 2021. -
  5. [11][11] Timo Schick and H. Schutze. Few-shot text generation with pattern-exploiting training. arXiv , abs/2012.11926, 2020. - · arXiv
  6. [12][12] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2103.11955 , 2021. - · arXiv
  7. [17][17] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In EMNLP , pages 2463-2473, Hong Kong, China, November 2019. ACL. -
  8. [20][20] S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In ICLR , 2017. -
  9. [21][21] Ke Li and Jitendra Malik. Learning to optimize. arXiv , abs/1606.01885, 2017. - · arXiv
  10. [23][23] M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series A (Methodological) , 36:111-133, 1974. -
  11. [25][25] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001. -
  12. [26][26] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , volume 70 of ICML'17 , pages 1126-1135. JMLR.org, 2017. -
  13. [27][27] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, NeurIPS , volume 32. Curran Associates, Inc., 2019. -
  14. [28][28] J. Rissanen. Modeling by shortest data description. Automatica , 14(5):465 - 471, 1978. -
  15. [29][29] J. Rissanen. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory , 30(4):629-636, 1984. -
  16. [31][31] Peter Grünwald. A tutorial introduction to the minimum description length principle. CoRR , math.ST/0406077, 06 2004. - · arXiv
  17. [32][32] Léonard Blier and Yann Ollivier. The description length of deep learning models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeuRIPS , volume 31, pages 2216-2226. Curran Associates, Inc., 2018. -
  18. [33][33] Alselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Occam's razor. Inf. Process. Lett. , 24(6):377-380, April 1987. -
  19. [34][34] Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. CoRR , abs/2104.06644, 2021. - · arXiv
  20. [35][35] P. J. Phillips, Hyeonjoon Moon, S. A. Rizvi, and P. J. Rauss. The feret evaluation methodology for face-recognition algorithms. TPAMI , 22(10):1090-1104, 2000. -
  21. [36][36] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Fairness, Accountability and Transparency , volume 81 of PMLR , pages 77-91, New York, NY, USA, 23-24 Feb 2018. PMLR. -
  22. [37][37] Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In AAAI/ACM Conference on AI, Ethics, and Society , AIES '18, page 123-129, New York, NY, USA, 2018. Association for Computing Machinery. -
  23. [38][38] Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King, and Rohit Prasad. Advancing the state of the art in open domain dialog systems through the alexa prize. CoRR , abs/1812.10757, 2018. - · arXiv
  24. [39][39] Javier García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. JMLR , 16(42):1437-1480, 2015. -
  25. [40][40] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR , abs/1606.06565, 2016. - · arXiv
  26. [41][41] Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. JMLR , 11(116):3571-3594, 2010. -
  27. [42][42] H. Akaike. A new look at the statistical model identification. TACON , 19(6):716-723, 1974. -
  28. [43][43] C. L. Mallows. Some comments on cp. Technometrics , 15(4):661-675, 1973. -
  29. [44][44] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization , pages 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. -
  30. [47][47] Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Chapter 15 - evolving deep neural networks. In Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito, editors, Artificial Intelligence in the Age of Neural Networks and Brain Computing , pages 293-312. Academic Press, 2019. -
  31. [48][48] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI , 33(01):4780-4789, Jul. 2019. -
  32. [49][49] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR . OpenReview.net, 2017. -
  33. [50][50] J. Larsen, L.K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Neural Networks for Signal Processing VI. IEEE Signal Processing Society Workshop , pages 62-71, 1996. -
  34. [51][51] Yoshua Bengio. Gradient-Based Optimization of Hyperparameters. Neural Computation , 12(8):1889-1900, 08 2000. -
  35. [53][53] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR , 2019. -
  36. [54][54] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In EMNLP , pages 4222-4235, Online, November 2020. ACL. -
  37. [58][58] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-ofthe-art natural language processing. In EMNLP: System Demonstrations , pages 38-45, Online, October 2020. ACL. -
  38. [59][59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, NeuRIPS , pages 8024-8035. Curran Associates, Inc., 2019. -
  39. [60][60] Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In EMNLP , pages 183-196, Online, November 2020. ACL. -
  40. [62][62] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR , 2020. -
  41. [64][64] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR , 2020. -
  42. [65][65] Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in NLP. CoRR , abs/2104.08835, 2021. - · arXiv
  43. [66][66] Rich Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, NeuRIPS , volume 7. MIT Press, 1995. -
  44. [68][68] Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR , abs/1811.01088, 2018. - · arXiv
  45. [69][69] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL , pages 4487-4496, Florence, Italy, July 2019. ACL. -
  46. [70][70] Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for the Winograd schema challenge. In ACL , pages 4837-4842, Florence, Italy, July 2019. ACL. -
  47. [71][71] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeuRIPS , volume 33, pages 6256-6268. Curran Associates, Inc., 2020. -
  48. [72][72] Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL , pages 2147-2157, Online, July 2020. ACL. -
  49. [73][73] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, JiPing Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP , pages 1008-1025, Online, November 2020. ACL. -
  50. [74][74] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In ICLR , 2018. -
  51. [75][75] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR , 2018. -
  52. [76][76] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP , pages 8864-8880, Online, November 2020. ACL. -
  53. [77][77] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. CoRR , abs/1804.09170, 2018. - · arXiv
  54. [78][78] Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP , pages 2185-2194, Hong Kong, China, November 2019. ACL. -
  55. [79][79] Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. TACL , 8:34-48, 2020. -
  56. [80][80] Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leaveone-out cross-validation and waic. Statistics and Computing , 27(5):1413-1432, September 2017. -
  57. [81][81] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR , abs/2002.06305, 2020. - · arXiv
  58. [82][82] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL , pages 2924-2936, Minneapolis, Minnesota, June 2019. ACL. -
  59. [83][83] Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. Sinn und Bedeutung , 23(2):107-124, July 2019. -
  60. [84][84] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d'Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment , pages 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. -
  61. [85][85] Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Second PASCAL Challenges Workshop on Recognising Textual Entailment , 2006. -
  62. [86][86] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing , RTE '07, page 1-9, USA, 2007. ACL. -
  63. [88][88] Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In NAACL . ACL, 2019. -
  64. [89][89] Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR , KR'12, page 552-561. AAAI Press, 2012. -
  65. [90][90] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL . ACL, 2018. -
  66. [91][91] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Perez, E., Kiela, D., & Cho, K.
Affiliations
New York University (US)
ORCID
Cho, Kyunghyun
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2105.11447
arXiv
2105.11447
OpenAlex
W3164972323
Cited by
226
References
12
Citations by year
2026: 5 · 2025: 34 · 2024: 37 · 2023: 76 · 2022: 60 · 2021: 14
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
pre-trained language models, prompt selection, hyperparameter optimization, cross-validation, minimum description length, model selection, uncertainty estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review