CoChat
Sign Up
PreprintCited by 182Logic, Reasoning, and KnowledgeAI-based Problem Solving and PlanningNatural Language Processing Techniques

STaR: Bootstrapping Reasoning With Reasoning

Open access
Zelikman, Eric, Wu, Yuhuai, Mu, Jesse, Goodman, Noah D.·2022·arXiv
DOI 10.48550/arxiv.2203.14465arXiv 2203.14465
Open this paper in CoChat — free accountOpen PDF

Abstract

Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using only few-shot inference. We propose a technique to iteratively leverage a small number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning. This technique, the "Self-Taught Reasoner" (STaR), relies on a simple loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; fine-tune on all the rationales that ultimately yielded correct answers; repeat. We show that STaR significantly improves performance on multiple datasets compared to a model fine-tuned to directly predict final answers, and performs comparably to fine-tuning a 30$\times$ larger state-of-the-art language model on CommensenseQA. Thus, STaR lets a model improve itself by learning from its own generated reasoning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.14465

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • STaR iteratively fine-tunes models on self-generated solutions but struggles with compounding errors due to the absence of explicit verification.

    “STaR (Zelikman et al., 2022) iteratively fine-tunes models on self-generated solutions but struggles with compounding errors due to the absence of explicit verification.”

    From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Zelikman et al., 2022

    The Conclusion (p. 9) confirms that STaR repeatedly fine-tunes on model-generated rationales. It also notes that poor rationales can confound the approach, but does not describe compounding errors as the cause of its difficulties. Algorithm 1 (p. 4) explicitly filters rationales using ground-truth answers, so the claim that STaR lacks explicit verification is too broad, even though that filter does not verify the reasoning itself. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Common approaches rely on variants of expert iteration or rejection finetuning, where multiple solutions are sampled, and an external signal selects positive ones for model fine-tuning.

    “Common approaches rely on variants of expert iteration or rejection finetuning (Polu et al., 2022;Zelikman et al., 2022;Yuan et al., 2023;Lin et al., 2024), where multiple solutions are sampled, and an external signal selects the positive ones for model fine-tuning.”

    From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Zelikman et al., 2022

    The Introduction (p. 1) and §3.2 (p. 4) support filtering generated rationales by whether they lead to the correct answer and fine-tuning on the retained examples. But the claim’s description of sampling multiple solutions is not how this paper presents STaR: §5 (p. 9) describes more, higher-temperature sampling as a counterproductive alternative to its rationalization step. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Even when answers are present, the solver still engages in nontrivial reasoning to reach a solution, potentially benefiting from this exposure.

    “Moreover, we believe that even when answers are present, the solver still engages in nontrivial reasoning to reach a solution, potentially benefiting from this exposure.”

    From Absolute Zero: Reinforced Self-play Reasoning with Zero Data (2025) · cites this paper as Zelikman et al., 2022

  • Forms of weak supervision that would normally require task-specific or hand-built heuristics are now done by LMs.

    “Similarly, forms of weak supervision that would normally require task-specific (Khattab et al., 2021a;b) or hand-built (Ratner et al., 2016; Hancock et al., 2018) heuristics are now done by LMs (Wang et al., 2022b; Zelikman et al., 2022; Zhang et al., 2022; Shao et al., 2023).”

    From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as Zelikman et al., 2022

  • The term 'self-improvement' is used for fundamentally different ambitions-inference-time output revision, training loops on self-generated data, agents that rewrite their own code, and systems that autonomously conduct AI research-and the literature's 'self-X' vocabulary obscures these differences.

    “First, 'self-improvement' is not one thing. The term is used for inference-time output revision (Self-Refine [6]), training loops on self-generated data (STaR [7], Self-Rewarding LMs [8]), agents that rewrite their own code (Gödel Agent [9], Darwin Gödel Machine [10]), and systems that autonomously conduct AI research [11]. These differ enormously in ambition and risk profile, yet the literature's”

    From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 182

Research works printed but not linked: 32

  1. [2][2] K Anders Ericsson and Herbert A Simon. Protocol analysis: Verbal reports as data. the MIT Press, 1984. -
  2. [3][3] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. ACL , 2019. -
  3. [4][4] Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. EMNLP 2020 , 2020. -
  4. [5][5] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021. - · arXiv
  5. [6][6] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. - · arXiv
  6. [7][7] Ana Marasovi´ c, Iz Beltagy, Doug Downey, and Matthew E Peters. Few-shot self-rationalization with natural language prompts. arXiv preprint arXiv:2111.08284 , 2021. - · arXiv
  7. [8][8] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 , 2022. - · arXiv
  8. [9][9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. - · arXiv
  9. [10][10] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4149-4158, 2019. -
  10. [11][11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  11. [12][12] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ICLR 2022 , 2021. -
  12. [13][13] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 , 2021. - · arXiv
  13. [15][15] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP 2021 , 2021. -
  14. [16][16] Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. Formal mathematics statement curriculum learning. arXiv preprint arXiv:2202.01344 , 2022. - · arXiv
  15. [17][17] Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. The lean theorem prover (system description). In International Conference on Automated Deduction , pages 378-388. Springer, 2015. -
  16. [18][18] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393 , 2020. - · arXiv
  17. [19][19] Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. Towards interpretable natural language understanding with explanations as latent variables. Advances in Neural Information Processing Systems , 33:6803-6814, 2020. -
  18. [20][20] Noam Wies, Yoav Levine, and Amnon Shashua. Sub-task decomposition enables learning in sequence to sequence tasks. arXiv preprint arXiv:2204.02892 , 2022. - · arXiv
  19. [21][21] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems , 30, 2017. -
  20. [22][22] Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville. Iterated learning for emergent systematicity in vqa. ICLR 2021 , 2021. -
  21. [23][23] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems , 31, 2018. -
  22. [24][24] Hanxiong Chen, Xu Chen, Shaoyun Shi, and Yongfeng Zhang. Generate natural language explanations for recommendation. arXiv preprint arXiv:2101.03392 , 2021. - · arXiv
  23. [25][25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  24. [27][27] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022. - · arXiv
  25. [28][28] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. singh 2002 (2016). arXiv preprint arxiv:1612.03975 , 2016. - · arXiv
  26. [29][29] Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. arXiv:2112.03254 , December 2021. human parity result on CommonsenseQA. - · arXiv
  27. [30][30] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022. - · arXiv
  28. [31][31] Stefan Palan and Christian Schitter. Prolific. ac-a subject pool for online experiments. Journal of Behavioral and Experimental Finance , 17:22-27, 2018. -
  29. [33][33] Bernease Herman. The promise and peril of human evaluation for model interpretability. arXiv preprint arXiv:1711.07414 , 2017. - · arXiv
  30. [34][34] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? ACL 2020 , 2020. -
  31. [35][35] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. - · arXiv
  32. [36][36] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv

Authors
Zelikman, E., Wu, Y., Mu, J., & Goodman, N. D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
15476-15488
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2203.14465
arXiv
2203.14465
OpenAlex
W7133192525
Cited by
182
References
0
Citations by year
2026: 45 · 2025: 36 · 2024: 34 · 2023: 59 · 2022: 8
FWCI
18.94
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review