Large language models are zero-shot reasoners
Open accessAbstract
Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding "Let's think step by step" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The zero-shot DNA prompting technique described by Kojima et al. [14] was carefully followed to reproduce the original setup as faithfully as possible.
“In our study, we carefully follow the zero-shot DNA (Descriptive, Narrative, and Aspects) prompting technique described by Kojima et al. [14] to reproduce the original setup as faithfully as possible.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [14]
Section 3 (p. 3) identifies Kojima et al.’s technique as Zero-shot-CoT, which prompts a model with “Let’s think step by step” to elicit reasoning. Section 3.1 (pp. 3–4) describes a two-stage reasoning-and-answer procedure, not the attributed Descriptive, Narrative, and Aspects (DNA) technique. Whether the citing authors followed a procedure faithfully is their own claim, not something this source can establish. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredZero-shot CoT approaches have now become more prevalent than early few-shot CoT methods.
“Although early research primarily focused on fewshot CoT methods, zero-shot CoT approaches [241] have now become more prevalent.”
From A Survey of Large Language Models (2026) · cites this paper as [241]
The Background (p. 3) describes the original CoT prompting method as few-shot, and the Conclusion (p. 10) presents this paper’s zero-shot approach as a contrast to previous few-shot work. Neither passage says that zero-shot CoT approaches have since become more prevalent than few-shot methods. (LLM, conf=0.87)
Experimental — accuracy not yet measuredChain-of-Thought and its variants encourage LLMs to reason prior to arriving at a final reply, which improves performance and reduces hallucinations by leveraging the model's internal knowledge.
“Chain-of-Thought and its variants encourage LLMs to reason prior to arriving at a final reply, which improves performance and reduces hallucinations by leveraging the model's internal knowledge [46, 110, 123, 138, 232, 273].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [123]
The Abstract (p. 1) reports that Zero-shot-CoT elicits step-by-step reasoning and improves accuracy on several reasoning benchmarks; it also describes prompting as a way to explore knowledge hidden inside LLMs. The paper does not establish that CoT reduces hallucinations, so that part of the attributed claim is unsupported. (LLM, conf=0.87)
Experimental — accuracy not yet measuredLLMs have the ability to understand tasks described in natural language and generate plans to solve them.
“Large language models (LLMs) have the ability to understand tasks described in natural language and generate plans to solve them [20, 27, 49, 60].”
From AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (2024) · cites this paper as [27]
Section 3.1 (p. 4) describes giving models a natural-language question and prompting them to generate a chain of thought; the Discussion (p. 9) reports understandable chains of thought across diverse reasoning tasks. This supports generating steps toward an answer, but the paper does not establish that LLMs generally understand task descriptions or generate plans without the specified prompting. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe limitations resemble those of natural language models, which have improved through scaling, labeled data, prompt engineering, and human-preference alignment.
“These limitations mirror the constraints of natural language models, which have been improved over time with increased scale, labeled data, prompt engineering, and alignment with human preferences (39, 83–85, 87).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [87]
The Introduction (p. 1) and Background (p. 3) describe improvements associated with scaling and prompting. Discussion and Related Work (p. 9) also reports improved zero-shot abilities from instruction fine-tuning, but our copy does not establish the claimed improvements through labeled data or alignment with human preferences. Its discussion of shortcomings (p. 10) concerns prompting inheriting biases from language-model training, not the unspecified limitations in the citing sentence. (LLM, conf=0.84)
Experimental — accuracy not yet measured
LLMs possess significant capabilities in reasoning and problem-solving, particularly in zero-shot learning within resource-limited domains.
“Authorship attribution is a complex reasoning task, and LLMs possess significant capabilities in reasoning and problem-solving, particularly in zero-shot learning within resource-limited domains [Kojima et al., 2022].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Kojima et al., 2022
The Abstract (p. 1) reports substantial gains on diverse reasoning tasks using zero-shot prompting, and the Discussion (p. 9) describes LLMs as decent zero-shot reasoners with potential for generic problem solving. The paper does not establish these capabilities specifically within resource-limited domains; its zero-shot method avoids handcrafted examples and time-consuming fine-tuning, which is a narrower point. (LLM, conf=0.84)
Experimental — accuracy not yet measuredZero-sample CoT is considered to be the best inference prompt currently; using 'Let's think step by step' significantly improved multiple inference-related zero-shot tasks.
“In addition, zero-sample CoT [19] is also considered to be the best inference prompt currently; by using a simple prompt: 'Let's think step by step', the specific prompt and the corresponding two-stage key answer extraction prompt technology has significantly improved multiple inference-related zero-shot tasks.”
From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [19]
The paper describes the two-stage reasoning and answer-extraction prompts (Section 3.1, p. 3) and reports substantial gains over standard zero-shot prompting on multiple reasoning tasks (p. 2). Its conclusion calls Zero-shot-CoT the strongest zero-shot baseline for difficult multi-step reasoning tasks (p. 10), but does not establish it as the best inference prompt generally or currently; the paper also reports that it underperforms carefully crafted few-shot CoT. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLLMs can be used as decent planners to decompose complex tasks into simpler ones, and this capability can be triggered without incorporating examples in prompts.
“Previous work has demonstrated that LLMs can be used as decent planers to decompose complex task into simpler ones [45].Such capability of LLMs can be even triggered without incorporating examples in the prompts [46].”
From A survey on large language model based autonomous agents (2024) · cites this paper as [46]
Early work on few-shot learning and prompt engineering found that the LLM could be further biased towards high-quality productions by pre-processing the input string.
“Early work on few-shot learning (Brown et al., 2020) and prompt engineering (Wei et al., 2022b; Kojima et al., 2022; Xu et al., 2023c) found that the LLM could be further biased towards high-quality productions by pre-processing the input string.”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Kojima et al., 2022
Step-by-step prompting drastically improves performance on reasoning benchmarks.
“a prompting technique drastically improving performance on reasoning benchmarks (Kojima et al., 2022).”
From Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure (2025) · cites this paper as Kojima et al., 2022
Prompt engineering demonstrated that model behavior could be substantially altered without touching the weights.
“Prompt engineering demonstrated that model behavior could be substantially altered without touching the weights: few-shot examples, role descriptions, chain-of-thought decomposition, and self-consistency traces all changed how the same model performed on the same underlying task [Wei et al., 2022, Wang et al., 2023b, Kojima et al., 2022].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Kojima et al., 2022
Chain-of-Thought prompting elicits step-by-step reasoning traces that could mimic clinical differential diagnosis processes.
“Chain-of-Thought (CoT) prompting [24,25] elicits step-by-step reasoning traces that could mimic clinical differential diagnosis processes [62].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [25]
Chain-of-thought prompting showed that eliciting intermediate reasoning steps substantially improves performance on arithmetic, logical, and symbolic benchmarks.
“Chain-of-thought (CoT) prompting showed that eliciting intermediate reasoning steps substantially improves performance on arithmetic, logical, and symbolic benchmarks (Wei et al., 2022;Kojima et al., 2022).”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Kojima et al., 2022
CoT helps guide the model to output the final answer step by step.
“It helps guide the model to output the final answer step by step (Wei et al., 2022;Peng et al., 2023;Kojima et al., 2022).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Kojima et al., 2022
Zero-shot chain-of-thought prompting did not yield any gains in the paper's preliminary experiments.
“We experimented with the zero-shot CoT approach (Kojima et al., 2022) in our preliminary experiments, but it did not yield any gains (Arcuschin et al., 2025).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Kojima et al., 2022
LLMs have the capability to solve problems using step-by-step reasoning.
“Large language models (LLMs) have the capability to solve problems using step-by-step reasoning (Kojima et al., 2022)”
From iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planning (2025) · cites this paper as Kojima et al., 2022
Natural language models frequently have difficulty producing coherent and diverse text over long sequences.
“Natural language models often struggle to maintain coherent and diverse generation over long sequences”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [87]
The discussion of Few-shot-CoT (p. 2) says language models have struggled with multi-step reasoning, not with maintaining coherence or diversity in long text generation. Searches for long-sequence generation, coherence, diversity, and repetition found no passage in our copy addressing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Solving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275 · cited 18×
“While our Zero-shot-CoT underperforms Few-shot-CoT with carefully-crafted and task-specific step-by-step examples, Zero-shot-CoT achieves enormous score gains compared to the zero-shot baseline, e.g. from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K with large-scale InstructGPT model (text-davinci-002).”
BackgroundInfluential
Flagged first, then by citations
Research works printed but not linked: 43
- Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https://arxiv.org/abs/ 2204.01691 .
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in NeurIPS , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .
- François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547 , 2019. URL https://arxiv.org/abs/1911.01547 . · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https://arxiv.org/abs/2204.02311 . · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/ abs/2110.14168 . · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL , pages 4171-4186, 2019. URL https://aclanthology.org/N19-1423 .
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv: Arxiv-2101.00027 , 2020.
- Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of ACL-IJCNLP , pages 3816-3830, 2021. URL https://aclanthology. org/2021.acl-long.295 .
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. TACL , 9:346-361, 2021. URL https://aclanthology.org/2021.tacl-1.21/ .
- Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In EMNLP , volume 523533. Citeseer, 2014. URL https://aclanthology.org/D14-1058/ .
- Wendy Johnson and Thomas J Bouchard Jr. The structure of human intelligence: It is verbal, perceptual, and image rotation (vpr), not fluid and crystallized. Intelligence , 33(4):393-416, 2005.
- Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. TACL , 3:585-597, 2015. URL https: //aclanthology.org/Q15-1042 .
- Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of NAACL , pages 1152-1157, 2016. URL https://aclanthology.org/N16-1136 .
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of ACL , pages 158-167, 2017. URL https://aclanthology.org/P17-1015 .
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 , 2021a. URL https://arxiv.org/abs/2101.06804 . · arXiv
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 , 2021b. URL https://arxiv.org/abs/2107. 13586 . · arXiv
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of ACL , pages 8086-8098, 2022. URL https://aclanthology.org/2022.acl-long.556 .
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: Arxiv-1609.07843 , 2016. URL https://arxiv.org/abs/1609. 07843 .
- Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022. URL https://arxiv.org/pdf/2202.12837.pdf . · arXiv
- Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop , 2022. URL https://openreview.net/forum? id=HBlx2idbkbq .
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155 . · arXiv
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in NeurIPS , 32:8026-8037, 2019. URL https://papers.nips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html .
- Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of NAACL , pages 2080-2094, 2021. URL https:// aclanthology.org/2021.naacl-main.168 .
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2021. URL https://arxiv.org/abs/2112.11446 . · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .
- Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of ACL , pages 4932-4942, 2019. URL https://aclanthology.org/P19-1487 .
- Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-7, 2021. URL https://arxiv.org/pdf/2102.07350.pdf . · arXiv
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In ICLR , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 .
- Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also fewshot learners. In Proceedings of NAACL , pages 2339-2352, 2021. URL https://aclanthology. org/2021.naacl-main.185 .
- Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of EMNLP , pages 4222-4235, 2020. URL https://aclanthology.org/2020. emnlp-main.346 .
- Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of EMNLP , pages 4615-4629, 2020. URL https://aclanthology.org/2020.emnlp-main.373 .
- Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, 2022. URL https: //arxiv.org/abs/2201.11990 .
- Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. URL https://arxiv.org/abs/2206.04615 . · arXiv
- Keith E Stanovich and Richard F West. Individual differences in reasoning: Implications for the rationality debate? Behavioral and brain sciences , 23(5):645-665, 2000.
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT , pages 4149-4158, 2019. URL https://aclanthology.org/N19-1421/ .
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022. URL https: //arxiv.org/abs/2201.08239 .
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in NeurIPS , 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022. URL https://arxiv.org/abs/2203.11171 . · arXiv
- Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2300-2344. Association for Computational Linguistics, July 2022. URL https://aclanthology.org/2022.naacl-main. 167 .
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022. URL https: //arxiv.org/abs/2201.11903 .
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP , 2020. URL https://aclanthology.org/ 2020.emnlp-demos.6 .
- Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465 . · arXiv
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. URL https://arxiv.org/abs/2205.01068 . · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113 · see citing works
DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025) · Nature · doi:10.1038/s41586-025-09422-z · Cited by 1,701 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
“Prompt with correct chain-of-thought 17.1 Prompt with imperfect chain-of-thought 14.9 + Self-consistency (40 paths) 23.4 Prompt with equations 5.0 + Self-consistency (40 paths) 6.5 PaLM-540B Zero-shot CoT (Kojima et al., 2022) 43.0 + Self-consistency (40 paths) 69.2 Table 8: Self-consistency works under imperfect prom…”
ResultInfluentialCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · arXiv · doi:10.48550/arxiv.2212.10509 · Cited by 349 · see citing works
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344 · see citing works
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
“Specifically, we ask language models to ‘think step-by-step’ [Kojima et al., 2022] and write out an argument explaining why one AI assistant response would be more harmless than another, before actually choosing the less harmful response.”
MethodEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“In the zero-shot CoT setting, this corresponds to the setup described in Kojima et al., 25 and the direct setting corresponds to Brown et al. 7”
MethodMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · see citing works
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3641204.3641221 · Cited by 342 · see citing works
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 274 · see citing works
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · see citing works
“This kind of baseline includes: —Chain-of-thought (CoT) [57] generates a chain of thought for each question and then generates the corresponding code.”
InfluentialMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Self-citationHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“…language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023a; Chu et al., 2023).”
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210 · see citing works
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202 · see citing works
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., & Awadalla, H. H. (2023) · arXiv · doi:10.48550/arxiv.2302.09210 · Cited by 202 · see citing works
“…learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022; Kojima et al., 2022; Anonymous, 2023a).”
BackgroundLarge Language Models can Accurately Predict Searcher Preferences
Thomas, P., Spielman, S., Craswell, N., & Mitra, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657707 · Cited by 168 · see citing works
NetLLM: adapting large language models for networking
Wu, D., Wang, X., Qiao, Y., Wang, Z., Jiang, J., Cui, S., & Wang, F. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672268 · Cited by 167 · see citing works
“These problems, in fact, are not unique to networking and have been successfully addressed in other domains (e.g., NLP [10, 45] and robotics [23, 112]) by introducing LLMs as the foundation models to solve various downstream tasks.”
BackgroundApplying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186 · see citing works
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155 · see citing works
Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D. (2024) · Proceedings of the 17th ACM International Conference on Web Search and Data Mining · doi:10.1145/3616855.3635752 · Cited by 155 · see citing works
Expel: LLM agents are experiential learners
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29936 · Cited by 153 · see citing works
Is GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“, these models are able to answer questions with explicit reasoning steps (Wei et al., 2022b; Kojima et al., 2022).”
BackgroundEight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137 · see citing works
“The chain-of-thought reasoning strategies mentioned in Section 2 are an especially clear example of this: Simply prompting a model to “think step by step” can lead it to perform well on entire categories of math and reasoning problems that it would otherwise fail on (Kojima et al., 2022).”
Biomedical knowledge graph-optimized prompt generation for large language models
Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B., Peetoom, B., Villouta-Reyes, C., Cerono, G., Shi, Y., Rizk-Jackson, A., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae560 · Cited by 135 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“C O T; Kojima et al., 2022) is a variation of C O T where the prompt does not contain any exemplar.”
BackgroundInfluentialLevy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119 · see citing works
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110 · see citing works
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106 · see citing works
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
Semantic anomaly detection with large language models
Elhafsi, A., Sinha, R., Agia, C., Schmerling, E., Nesnas, I. A. D., & Pavone, M. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10132-6 · Cited by 101 · see citing works
Deception abilities emerged in large language models
Hagendorff, T. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317967121 · Cited by 100 · see citing works
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · see citing works
“In Figure 6, we illustrate the performance of multi-agent debate with and without zero-shot chain-ofthought prompting [11] on GSM8K.”
BackgroundMethodMcCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96 · see citing works
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120 · see citing works
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · see citing works
Zheng, M., Pei, J., Logeswaran, L., Lee, M., & Jurgens, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.888 · Cited by 82 · see citing works
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79 · see citing works
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 77 · see citing works
“Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
BackgroundOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74 · see citing works
“It has shown impressive performance in complex reasoning tasks (e.g., arithmetic and symbolic reasoning) [19, 47], and has therefore been applied to code generation [17, 26].”
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“Zero-shot Chain-of-Thought (Kojima et al., 2022) Q −→ QStep-by-step ∼∼∼∼▸ LLM QStep-by-stepA”
BackgroundTowards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective
Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.15408 · Cited by 69 · see citing works
“Practically, this can be achieved by either adding special phrases such as “let’s think step by step” or by giving few-shot CoT demonstrations [29, 56, 51, 38, 63, 58].”
BackgroundHagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
A Practical Survey on Zero-shot Prompt Design for In-context Learning
Columbia University, N. Y. C., & Li, Y. (2023) · Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings · doi:10.26615/978-954-452-092-2_069 · Cited by 68 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Chen, J., Saha, S., & Bansal, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.381 · Cited by 64 · see citing works
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63 · see citing works
Liu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642698 · Cited by 61 · see citing works
Yuan, H., Yu, K., Xie, F., Liu, M., & Sun, S. (2024) · Medicine Advances · doi:10.1002/med4.75 · Cited by 61 · see citing works
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 60 · see citing works
“This is fueling an exploding space of “prompting” techniques—and lightweight finetuning techniques—for adapting LMs to new tasks (Kojima et al., 2022), eliciting systematic reasoning from them (Wei et al., 2022; Wang et al., 2022b), and augmenting them with retrieved sources (Guu et al., 2020;…”
MethodOlausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57 · see citing works
“…are able to flexibly and systematically reason (Huang and Chang, 2022), especially when using prompting strategies that explicitly encourage verbalizing intermediate reasoning steps before generating the final answer (Nye et al., 2021; Wei et al., 2022; Kojima et al., 2022; Wang et al., 2023b).”
BackgroundMethodAssessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57 · see citing works
<scp>PLeak:</scp> Prompt Leaking Attacks against Large Language Model Applications
Hui, B., Yuan, H., Gong, N., Burlina, P., & Cao, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670370 · Cited by 55 · see citing works
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58 · see citing works
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A., & Saffari, A. (2023) · Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023) · doi:10.18653/v1/2023.matching-1.7 · Cited by 51 · see citing works
“Retrieval is the Top-1 result of the MPNet (Song et al., 2020). augments them as the form of the prompt (Liu et al., 2022b), which is similar to Kojima et al. (2022).”
BackgroundShojaee, P., Mirzadeh, I., Alizadeh vahid, K., Horton, M., Bengio, S., & Farajtabar, M. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3603 · Cited by 47 · see citing works
Large Language Models Can Learn Temporal Reasoning
Xiong, S., Payani, A., Kompella, R., & Fekri, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.563 · Cited by 47 · see citing works
Open Science at the generative AI turn: An exploratory analysis of challenges and opportunities
Hosseini, M., Horbach, S. P. J. M., Holmes, K., & Ross-Hellauer, T. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00337 · Cited by 40 · see citing works
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51 · see citing works
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“Authorship attribution is a complex reasoning task, and LLMs possess significant capabilities in reasoning and problem-solving, particularly in zero-shot learning within resource-limited domains (Kojima et al., 2022).”
Hallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38 · see citing works
“Recently, the emergence of large language models (LLMs) [29, 81, 85, 99, 132] has dominated a wide range of tasks in natural language processing (NLP), achieving unprecedented progress in language understanding [39, 47], generation [128, 140] and reasoning [20, 58, 87, 107, 115].”
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“Following Kojima et al. (2022); Kim et al. (2023), we utilize the dev set for our evaluation, which encompasses 1,221 questions.”
MethodRanking LLM-Generated Loop Invariants for Program Verification
Chakraborty, S., Lahiri, S., Fakhoury, S., Lal, A., Musuvathi, M., Rastogi, A., Senthilnathan, A., Sharma, R., & Swamy, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.614 · Cited by 37 · see citing works
Generating dermatopathology reports from gigapixel whole slide images with HistoGPT
Tran, M., Schmidle, P., Guo, R. R., Wagner, S. J., Koch, V., Lupperger, V., Novotny, B., Murphree, D. H., Hardway, H. D., D’Amato, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60014-x · Cited by 36 · see citing works
Generative approach to Aspect Based Sentiment Analysis with GPT Language Models
Chumakov, S., Kovantsev, A., & Surikov, A. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.12.030 · Cited by 35 · see citing works
POSIX: A Prompt Sensitivity Index For Large Language Models
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.852 · Cited by 34 · see citing works
Beattie, J., Neufeld, S., Yang, D., Chukwuma, C., Gul, A., Desai, N., Jiang, S., & Dohopolski, M. (2024) · Cureus · doi:10.7759/cureus.60044 · Cited by 30 · see citing works
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31 · see citing works
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2309.16797 · Cited by 31 · see citing works
Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., & Li, Y.-F. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.77 · Cited by 31 · see citing works
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33 · see citing works
The language of prompting: What linguistic properties make a prompt successful?
Leidinger, A., van Rooij, R., & Shutova, E. (2023) · arXiv · doi:10.48550/arxiv.2311.01967 · Cited by 29 · see citing works
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 25 · see citing works
“Chain of Thought (CoT) (Wei et al., 2022; Kojima et al., 2022) prompting techniques represent a linear problem-solving approach where each step builds upon the previous one.”
BackgroundMindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17066 · Cited by 24
- Authors
- Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y.
- ORCID
- Iwasawa, Yusuke
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 22199-22213
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133231369
- Cited by
- 2,013
- References
- 1
- Citations by year
- 2026: 534 · 2025: 489 · 2024: 588 · 2023: 385 · 2022: 17
- FWCI
- 204.48
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Institute of Advanced Industrial Science and Technology
- University of Tokyo (MbSC2030)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, zero-shot reasoning, natural language reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
