CoChat
Sign Up
PreprintCited by 1,264Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Emergent Abilities of Large Language Models

Open access
Wei, Jason, Tay, Yi, Bommasani, Rishi, Raffel, Colin, Zoph, Barret, Borgeaud, Sebastian, Yogatama, Dani, Bosma, Maarten +8 more·2022·arXiv
DOI 10.48550/arxiv.2206.07682arXiv 2206.07682
Open this paper in CoChat — free accountOpen PDF

Abstract

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.07682

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs have achieved significant performance improvements upon previous PLMs across a diverse array of NLP tasks, and have demonstrated many emerging abilities and strong steerability after instruction tuning and RLHF.

    “The resulting Large Language Models (LLMs) not only have achieved significant performance improvements upon previous PLMs across a diverse array of NLP tasks, but have also demonstrated many emerging abilities and strong steerability after instruction tuning and Reinforcement Learning from Human Feedback (RLHF) [276, 320].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [276]

    The Conclusions (p. 11) describe emergent abilities across varied models and tasks. Section 4 (p. 5) reports improved instruction following after fine-tuning, and §5.2 (p. 7) reports favorable human evaluations of a model trained with fine-tuning and RLHF. The cited paper does not establish the sentence’s broad comparison with previous PLMs across NLP tasks or characterize the resulting steerability as strong. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • None of those capabilities were engineered or anticipated by their creators; instead, they emerged as LLMs were trained to achieve other goals.

    “Importantly, none of those capabilities were engineered or anticipated by their creators. Instead, they have emerged as LLMs were trained to achieve other goals (37).”

    From Evaluating large language models in theory of mind tasks (2024) · cites this paper as [37]

    The Discussion (p. 6) says emergent few-shot tasks were not explicitly included in pre-training, and the definition (p. 2) says emergent abilities could not be predicted by extrapolating smaller models’ performance. These passages support the general idea of unplanned capabilities, but they do not establish that none of the capabilities were engineered or anticipated by their creators, or that all arose while training for other goals. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Given the pace of technological development, future LLMs and generative AI systems may not be prone to mistakes.

    “Given the pace of technological development, future LLMs and generative AI systems may not be prone to mistakes (Wei et al., 2022).”

    From ChatGPT and the Technology-Education Tension: Applying Contextual Virtue Epistemology to a Cognitive Artifact (2024) · cites this paper as Wei et al., 2022

    Section 5.2 (pp. 7–8) says improved training and architectures may unlock abilities, which supports the possibility of future improvement. It does not suggest that future LLMs or generative AI systems may cease to be prone to mistakes; it cautions that some abilities may never emerge and that scaling does not guarantee a desired level of performance. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The TruthfulQA dataset has become widely used for evaluating the factuality of LLMs.

    “This dataset has become widely used for evaluating the factuality of LLMs [89, 146, 192, 220].”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [220]

    Section 3 (p. 3) uses TruthfulQA to evaluate whether language models answer questions truthfully, and §5.4 (p. 9) discusses results for GPT-3 and Gopher. These passages support its use as an evaluation benchmark, but do not establish that it had become widely used. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • LLMs excel at learning patterns from their vast training data and reproducing them, even in novel contexts.

    “Indeed, LLMs excel at learning patterns from their vast training data and reproducing them, even in novel contexts (Wei et al., 2022).”

    From Does It Make Sense to Speak of Introspection in Large Language Models? (2025) · cites this paper as Wei et al., 2022

    The paper’s “Data scaling” discussion (p. 10) says that training on a large-enough dataset helps language models acquire syntactic, semantic, and world knowledge. Its Introduction (p. 1) also reports improved performance on downstream tasks. Neither passage establishes the more specific claim that LLMs excel at reproducing learned patterns in novel contexts. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 1,264

Research works printed but not linked: 95

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022. URL https://arxiv.org/ abs/2204.01691 . · arXiv
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS , 2022. URL https://arxiv.org/abs/2204.14198 . · arXiv
  3. Philip W. Anderson. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science , 177(4047):393-396, 1972. URL http://www.lanais.famaf.unc.edu.ar/cursos/ em/Anderson-MoreDifferent-1972.pdf .
  4. Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441 , 2022. URL https://arxiv.org/abs/2210.02441 . · arXiv
  5. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684 , 2021. URL https://arxiv.org/abs/2112.10684 . · arXiv
  6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021. URL https://arxiv.org/abs/2112.00861 . · arXiv
  7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , 2021. URL https://dl.acm.org/doi/pdf/10.1145/3442188. 3445922 . · DOI · Paper page
  8. BIG-Bench. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. URL https://arxiv.org/abs/2206.04615 . · arXiv
  9. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. URL https://arxiv.org/abs/2108. 07258 . · arXiv
  10. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426 , 2021. URL https: //arxiv.org/abs/2112.04426 . · arXiv
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 2020. URL https://papers.nips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html .
  12. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. USENIX Security , 2021. URL https://www.usenix.org/conference/usenixsecurity21/ presentation/carlini-extracting .
  13. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 , 2022. URL https://arxiv.org/abs/2202.07646 . · arXiv
  14. Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055 , 2022. URL https://arxiv.org/abs/2205.05055 . · arXiv
  15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311 , 2022. URL https://arxiv.org/abs/2204.02311 . · arXiv
  16. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. URL https://arxiv.org/abs/2210.11416 . · arXiv
  17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. URL https://arxiv.org/abs/2110.14168 . · arXiv
  18. Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. Wordcraft: A human-AI collaborative editor for story writing. arXiv preprint arXiv:2107.07430 , 2021. URL https://arxiv.org/abs/2107. 07430 . · arXiv
  19. Antonella Corradini and Timothy O'Connor. Emergence in science and philosophy , volume 6. Routledge, 2010. URL https://books.google.com/books?hl=en&lr=&id=55RaBwAAQBAJ&oi= fnd&pg=PP1&dq=Emergence+in+science+and+philosophy&ots=2\_8VNDXLfv&sig=1aisq\_ WouF95Cx58WWMZ0Gq3RNk .
  20. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. URL https://arxiv.org/abs/1807.03819 . · arXiv
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL , 2019. URL https://aclanthology.org/ N19-1423 .
  22. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mixture-of-experts. ICML , 2021. URL https://arxiv.org/abs/2112.06905 . · arXiv
  23. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 , 2021. URL https://arxiv.org/ abs/2101.03961 . · arXiv
  24. Stephanie Forrest. Emergent computation: Self-organizing, collective, and cooperative phenomena in natural and artificial computing networks. Physica D: Nonlinear Phenomena , 42(1-3):1-11, 1990. URL https://www.sciencedirect.com/science/article/abs/pii/016727899090063U .
  25. Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, et al. Predictability and surprise in large generative models. arXiv preprint arXiv:2202.07785 , 2022. URL https://arxiv.org/abs/2202.07785 . · arXiv
  26. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. ACL , 2021. doi: 10.18653/v1/2021.acl-long.295. URL https://aclanthology.org/2021.acl-long.295 . · DOI · Paper page
  27. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP , 2020. doi: 10.18653/v1/ 2020.findings-emnlp.301. URL https://aclanthology.org/2020.findings-emnlp.301 . · DOI
  28. Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983 , 2016. URL https://arxiv.org/abs/1603.08983 . · arXiv
  29. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ICML , 2020. URL https://arxiv.org/abs/2002.08909 . · arXiv
  30. David A. Harper and Paul A. Lewis. New perspectives on emergence in economics. New Perspectives on Emergence in Economics , pp. 2-3, 2012. URL https://www.sciencedirect. com/science/article/pii/S0167268112000200?casa\_token=fLs2nCYo\_64AAAAA: H2sSpSygJmEqXgmpM4jLyeppph3C4TgEsaSXm5RkOpT0r4q2A1x9Su3u4uycK4sIC6a8NdLiSw .
  31. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR , 2021a. URL https://openreview.net/ forum?id=d7KBjmI3GmQ .
  32. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916 , 2021b. URL https://arxiv.org/abs/2109.13916 . · arXiv
  33. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. NeurIPS , 2022. URL https://arxiv.org/abs/2203.15556 . · arXiv
  34. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't always right. EMNLP , 2021. URL https://aclanthology. org/2021.emnlp-main.564 .
  35. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 , 2022. URL https://arxiv.org/pdf/2201.07207 . · arXiv
  36. Bernardo A. Huberman and Tad Hogg. Phase transitions in artificial intelligence systems. Artificial Intelligence , 33(2):155-171, 1987. URL https://www.sciencedirect.com/science/article/ abs/pii/0004370287900336 .
  37. Harold Y. Hwang, Yoh Iwasa, Masashi Kawasaki, Bernhard Keimer, Naoto Nagaosa, and Yoshinori Tokura. Emergent phenomena at oxide interfaces. Nature Materials , 11(2):103-113, 2012. URL https://www. nature.com/articles/nmat3223 .
  38. Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. ICML , 2017. URL https://arxiv.org/abs/1608.05343 . · arXiv
  39. Dan Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition . Prentice Hall series in Artificial Intelligence. Pearson Prentice Hall, 2009. ISBN 9780131873216. URL https://books.google.com/books?id= fZmj5UNK8AQC .
  40. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022. URL https://arxiv.org/abs/2207.05221 . · arXiv
  41. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. ICML , 2022. URL https://arxiv.org/abs/2202.06539 . · arXiv
  42. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS , 2022. URL https://arxiv.org/abs/2205.11916 . · arXiv
  43. Andrew K. Lampinen, Ishita Dasgupta, Stephanie C.Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. Can language models learn from explanations in context? Findings of EMNLP , 2022. URL https://arxiv.org/abs/2204.02329 . · arXiv
  44. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. ACL , 2022a. URL https://arxiv.org/abs/2107.06499 . · arXiv
  45. Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. CHI , 2022b. URL https://arxiv.org/abs/2201.06796 . · arXiv
  46. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ICLR , 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb .
  47. Percy Liang. Semi-supervised learning for natural language . PhD thesis, Massachusetts Institute of Technology, 2005. URL https://www-cs.stanford.edu/~pliang/papers/meng-thesis.pdf .
  48. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021. URL https://arxiv.org/abs/2109.07958 . · arXiv
  49. Christopher D. Manning. Human language understanding & reasoning. Daedalus , 151(2):127-138, 2022. URL https://www.amacad.org/publication/human-language-understanding-reasoning .
  50. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences , 117(48):30046-30054, 2020. URL https://www.pnas.org/doi/10.1073/pnas. 1907367117 . · DOI
  51. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 , 2018. URL https://arxiv. org/abs/1806.08730 . · arXiv
  52. Scott Miller, Jethran Guinness, and Alex Zamanian. Name tagging with word clusters and discriminative training. In NAACL , 2004. URL https://aclanthology.org/N04-1043 .
  53. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. ACL , 2022a. URL https://arxiv.org/abs/2108.04106 . · arXiv
  54. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022b. URL https://arxiv.org/abs/2202.12837 . · arXiv
  55. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021. URL https://openreview.net/forum?id=iedYJm92o0a . · arXiv
  56. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. URL https://arxiv.org/abs/2203.02155 . · arXiv
  57. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL , 2002. URL https://aclanthology.org/P02-1040.pdf .
  58. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of ACL , 2022. URL https://arxiv.org/abs/2110.08193 . · arXiv
  59. Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. ICLR , 2022. URL https://openreview.net/forum?id=gJcEM8sxHK .
  60. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286 , 2022. URL https://arxiv.org/abs/2202.03286 . · arXiv
  61. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. NAACL , 2019. URL https://aclanthology.org/ N19-1128 .
  62. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8), 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language\_models\_ are\_unsupervised\_multitask\_learners.pdf .
  63. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446 , 2021. URL https://arxiv.org/abs/ 2112.11446 . · arXiv
  64. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020. URL https://jmlr.org/papers/v21/20-074.html .
  65. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. URL https://arxiv.org/ abs/2204.06125 . · arXiv
  66. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206 , 2022. URL https://arxiv.org/ abs/2202.07206 . · arXiv
  67. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021. URL https://arxiv.org/abs/2102.07350 . · arXiv
  68. Rachel Rudinger, Chandler May, and Benjamin Van Durme. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing , 2017. URL https://aclanthology.org/W17-1609 .
  69. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 .
  70. Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. ICLR , 2021. URL https://arxiv.org/abs/2010.03648 . · arXiv
  71. Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. NAACL , June 2021. URL https://aclanthology.org/2021.naacl-main.185 .
  72. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 , 2022. URL https://arxiv.org/abs/ 2210.03057 . · arXiv
  73. Mirac Suzgun, Nathan Scales, Nathaneal Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny ZHou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022. URL https: //arxiv.org/abs/2210.09261 . · arXiv
  74. Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131 , 2022a. URL https://arxiv.org/abs/2205.05131 . · arXiv
  75. Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06991 , 2022b. URL https://arxiv.org/abs/2202.06991 . · arXiv
  76. Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399 , 2022c. URL https://arxiv.org/abs/2210.11399 . · arXiv
  77. Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. In ACL Big Science Workshop , 2022. URL https://aclanthology.org/2022.bigscience-1.11/ .
  78. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022. URL https://arxiv.org/abs/2201.08239 . · arXiv
  79. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847 , 2018. URL https://arxiv.org/abs/1806.02847 . · arXiv
  80. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? ICML , 2022a. URL https://arxiv.org/abs/2204.05832 . · arXiv
  81. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022b. URL https: //arxiv.org/abs/2203.11171 . · arXiv
  82. Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning. NeurIPS , 2021a. URL https://openreview.net/ forum?id=MDMV2SxCboX .
  83. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. Frequency effects on syntactic rule learning in transformers. EMNLP , 2021b. doi: 10.18653/v1/2021.emnlp-main.72. URL https://aclanthology. org/2021.emnlp-main.72 . · DOI
  84. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ICLR , 2022a. URL https: //openreview.net/forum?id=gEZrGCozdqR .
  85. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. NeurIPS , 2022b. URL https://arxiv. org/abs/2201.11903 .
  86. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021. URL https://arxiv.org/abs/2112.04359 . · arXiv
  87. Tongshuang Wu, Michael Terry, and Carrie J. Cai. AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. arXiv preprint arXiv:2110.01691 , 2021. URL https://arxiv.org/abs/2110.01691 . · arXiv
  88. Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J. Cai. PromptChainer: Chaining large language model prompts through visual programming. arXiv preprint arXiv:2203.06566 , 2022a. URL https://arxiv.org/abs/2203.06566 . · arXiv
  89. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913 , 2022b. · arXiv
  90. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ICLR , 2022. URL https://arxiv.org/abs/2111.02080 . · arXiv
  91. Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022. URL https://arxiv. org/abs/2204.00598 . · arXiv
  92. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. When do you need billions of words of pretraining data? In ACL , 2021. doi: 10.18653/v1/2021.acl-long.90. URL https://aclanthology. org/2021.acl-long.90 . · DOI · Paper page
  93. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. ICML , 2021. URL https://arxiv.org/abs/2102.09690 . · arXiv
  94. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022. URL https://arxiv.org/abs/2205.10625 . · arXiv
  95. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906 , 2022. URL https: //arxiv.org/abs/2202.08906 . · arXiv

Authors
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., & Fedus, W.
Affiliations
  • Google (United States) (US)
  • University of North Carolina at Chapel Hill (US)
  • Google DeepMind (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.07682
arXiv
2206.07682
OpenAlex
W4283026156
Cited by
1,264
References
0
Citations by year
2026: 48 · 2025: 326 · 2024: 511 · 2023: 362 · 2022: 16 · 2020: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
emergent capabilities, large language models, model scaling, model size, scaling laws

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review