The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Let's Verify Step by Step
Open accessAbstract
In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Constitutional AI (Bai et al., 2022) uses pre-specified rules; reward design methods (PBRS (Ng et al., 1999), process rewards (Lightman et al., 2023), rule-based rewards (Shao et al., 2024)) encode L 3 -type knowledge but are humandesigned, not learned.”
From Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents (2026) · cites this paper as Lightman et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current multi-step agents remain limited in process-level control. Although they can generate subgoals or action sequences, they usually lack an explicit mechanism to determine whether the current step is consistent with the task context and whether it effectively supports the current subgoal.”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Lightman et al. (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Interestingly, while [186] did not find any major downstream performance improvement with their method, [97] used similar methodology but instead trained larger models on a larger feedback dataset coupled with evaluation on a more difficult task found segment-level feedback to be significantly more useful.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [97]
The cited source says what the claim attributes to it.
In the citing paper
“A more granular assessment can be particularly valuable because it (1) not only allows users to pinpoint where inaccuracies occur (Liu et al., 2021) but also (2) serves as a reward model for developers to refine their generative systems (Lightman et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Lightman et al. [130], whose finding that process rewards outperform outcome rewards seeded today's process-reward-model (PRM) literature”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [130]
The cited source says what the claim attributes to it.
In the citing paper
“Let's Verify Step by Step Lightman et al. [2023] showed that process-level signals outperform outcome-only supervision on mathematical reasoning”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Lightman et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Additionally, our dataset explores the more general case of prompting API-based LLMs, whereas PRM800K uses a generator LLM that was heavily fine-tuned for math-specific problems.”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“process reward models [26] extend this to step-level supervision over reasoning traces.”
From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Step-level supervision has also been shown to be beneficial for reasoning verification [9].”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“In addition to Table 1, we also examine the accuracy at the 5 different difficulty levels explicitly defined by MATH-500 (Lightman et al., 2023) in Appendix B.2, where the observations remain consistent.”
From Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models (2025) · cites this paper as Lightman et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Lightman et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent studies [35, 41] have explored this question with established math benchmarks by comparing the upper-bound capabilities (pass@k) of RL-based reasoning (thinking) models with their non-reasoning (non-thinking) standard LLM counterparts. They have shown that under equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable to thinking models on benchmarks”
From The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025) · cites this paper as [42]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 23
- A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. · arXiv
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. · arXiv
- T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017. · arXiv
- L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti- mization. arXiv preprint arXiv:2210.10760, 2022. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. · arXiv
- A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra- masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. · arXiv
- Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022. · arXiv
- J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. · arXiv
- R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
- E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. In Proceedings of the 13th ACM SIGGRAPH Con- ference on Motion, Interaction and Games, pages 1-10, 2020.
- M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. · arXiv
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021. · arXiv
- N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feed- back. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
- J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. · arXiv
- X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. · arXiv
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. · arXiv
- E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reason- ing with reasoning. Advances in Neural Information Processing Systems, 35: 15476-15488, 2022.
- D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human pref- erences. arXiv preprint arXiv:1909.08593, 2019. Problem · arXiv
Cited by
Sorted by citations
Items 101–122 of 122
Stealing Reasoning Traces from Proprietary LLM APIs
Panfilov, A., Schmotz, D., Shumailov, I., Beurer-Kellner, L., Schaeffer, J., Prabhu, A., Geiping, J., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2608.09867
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
Decentralized Multi-Agent Systems with Shared Context
Mao, Y., Gu, J., Chauhan, A., Zhang, Q., Kang, H., & Mirhoseini, A. (2026) · arXiv · doi:10.48550/arxiv.2606.10662
“If a sub-agent receives insufficient context, it return control to the main agent, triggering additional retrieval or another delegation round.”
SKYNET 2023 Conception of the Artificial Super Intelligence Project. A System Approach. ver. 3Preprint
Novikov, A. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/kqt9p
An Inference Method for Professional Texts with Computational Expressions Under Few-Shot Scenarios
Yang, L., Zheng, W., Yuan, F., & Sun, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-2373-0_15
Detection avoidance techniques for large language models
Schneider, S., Steuber, F., Schneider, J., & Rodosek, G. D. (2025) · Data & Policy · doi:10.1017/dap.2025.6
TopoRefine: Iterative Refinement with Reasoning Topology as High-Level Feedback
Liao, H., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10888346
Kadlubsky, A. (2025) · Advances in wireless technologies and telecommunication book series · doi:10.4018/979-8-3693-7147-3.ch005
Human Rights in the Internet Engineering Task Force: Changing Norm EntrepreneurshipPreprint
Morrell, S., & Riedl, C. (2025) · doi:10.33774/apsa-2025-q9gdj-v2
TestFlow: Advancing Mobile UI Testing through Multi-Step Reinforcement Learning
Tang, X., Chen, X., Chen, D., Zhou, S., Jiang, W., & Li, Y. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732930
Tian, X. (2025) · Asian Journal of Law and Economics · doi:10.1515/ajle-2025-0037 · Cited by 1
Madzhumder, M. S., & Begunova, D. D. (2025) · Scientific and Technical Information Processing · doi:10.3103/s0147688225700479 · Cited by 1
"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration
Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536 · Cited by 1
Son, B., Jung, M., & Lee, S. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.7.81
Beyond Outcomes: Toward Process-Aware Software Agents
Liu, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3844569
MARFT: Multi-Agent Reinforcement Fine-Tuning
Liao, J., Wen, M., Wang, J., & Zhang, W. (2025) · arXiv · doi:10.48550/arxiv.2504.16129
A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
Khan, S., Madhavan, S., & Natarajan, K. (2025) · arXiv · doi:10.48550/arxiv.2506.18957
OmniGen2: Towards Instruction-Aligned Multimodal Generation
Wu, C., Zheng, P., Yan, R., Xiao, S., Luo, X., Wang, Y., Li, W., Jiang, X., Liu, Y., Zhou, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.18871
TTRL: Test-Time Reinforcement Learning
Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Zhang, Y., Long, X., Hua, E., et al. (2025) · arXiv · Cited by 7
“Subsequently, an aggregation strategy is applied to integrate these candidates, commonly using process reward models [24, 46, 60].”
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
Miao, N., Teh, Y. W., & Rainforth, T. (2023) · arXiv · doi:10.48550/arxiv.2308.00436 · Cited by 18
Metadata
- Authors
- Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.20050
- OpenAlex
- W4379089709
- Cited by
- 65
- References
- 0
- Citations by year
- 2026: 3 · 2025: 23 · 2024: 20 · 2023: 19
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- process supervision, multi-step reasoning, large language models, active learning (machine learning), reward modeling, mathematical problem solving, logical errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
