The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Let's Verify Step by Step
Open accessAbstract
In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Constitutional AI (Bai et al., 2022) uses pre-specified rules; reward design methods (PBRS (Ng et al., 1999), process rewards (Lightman et al., 2023), rule-based rewards (Shao et al., 2024)) encode L 3 -type knowledge but are humandesigned, not learned.”
From Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents (2026) · cites this paper as Lightman et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current multi-step agents remain limited in process-level control. Although they can generate subgoals or action sequences, they usually lack an explicit mechanism to determine whether the current step is consistent with the task context and whether it effectively supports the current subgoal.”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Lightman et al. (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Interestingly, while [186] did not find any major downstream performance improvement with their method, [97] used similar methodology but instead trained larger models on a larger feedback dataset coupled with evaluation on a more difficult task found segment-level feedback to be significantly more useful.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [97]
The cited source says what the claim attributes to it.
In the citing paper
“A more granular assessment can be particularly valuable because it (1) not only allows users to pinpoint where inaccuracies occur (Liu et al., 2021) but also (2) serves as a reward model for developers to refine their generative systems (Lightman et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Lightman et al. [130], whose finding that process rewards outperform outcome rewards seeded today's process-reward-model (PRM) literature”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [130]
The cited source says what the claim attributes to it.
In the citing paper
“Let's Verify Step by Step Lightman et al. [2023] showed that process-level signals outperform outcome-only supervision on mathematical reasoning”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Lightman et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Additionally, our dataset explores the more general case of prompting API-based LLMs, whereas PRM800K uses a generator LLM that was heavily fine-tuned for math-specific problems.”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“process reward models [26] extend this to step-level supervision over reasoning traces.”
From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Step-level supervision has also been shown to be beneficial for reasoning verification [9].”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“In addition to Table 1, we also examine the accuracy at the 5 different difficulty levels explicitly defined by MATH-500 (Lightman et al., 2023) in Appendix B.2, where the observations remain consistent.”
From Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models (2025) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Lightman et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent studies [35, 41] have explored this question with established math benchmarks by comparing the upper-bound capabilities (pass@k) of RL-based reasoning (thinking) models with their non-reasoning (non-thinking) standard LLM counterparts. They have shown that under equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable to thinking models on benchmarks”
From The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025) · cites this paper as [42]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 23
- A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. · arXiv
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. · arXiv
- T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017. · arXiv
- L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti- mization. arXiv preprint arXiv:2210.10760, 2022. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. · arXiv
- A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra- masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. · arXiv
- Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022. · arXiv
- J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. · arXiv
- R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
- E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. In Proceedings of the 13th ACM SIGGRAPH Con- ference on Motion, Interaction and Games, pages 1-10, 2020.
- M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. · arXiv
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021. · arXiv
- N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feed- back. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
- J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. · arXiv
- X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. · arXiv
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. · arXiv
- E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reason- ing with reasoning. Advances in Neural Information Processing Systems, 35: 15476-15488, 2022.
- D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human pref- erences. arXiv preprint arXiv:1909.08593, 2019. Problem · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Metadata
- Authors
- Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.20050
- OpenAlex
- W4379089709
- Cited by
- 65
- References
- 0
- Citations by year
- 2026: 3 · 2025: 23 · 2024: 20 · 2023: 19
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- process supervision, multi-step reasoning, large language models, active learning (machine learning), reward modeling, mathematical problem solving, logical errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
