Let's Verify Step by Step
Open accessAbstract
In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Constitutional AI uses pre-specified rules; reward design methods encode L3-type knowledge but are human-designed, not learned.
“Constitutional AI (Bai et al., 2022) uses pre-specified rules; reward design methods (PBRS (Ng et al., 1999), process rewards (Lightman et al., 2023), rule-based rewards (Shao et al., 2024)) encode L 3 -type knowledge but are humandesigned, not learned.”
From Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents (2026) · cites this paper as Lightman et al., 2023
The Introduction and §2.6 describe process rewards based on human feedback about intermediate reasoning steps, so the citation supports identifying Lightman et al. with process rewards. But §2.6 explicitly says the process reward model is trained to predict step correctness; it is not simply human-designed rather than learned. Our copy also does not characterize the feedback as “L3-type knowledge.” (LLM, conf=0.86)
Experimental — accuracy not yet measuredCurrent multi-step agents remain limited in process-level control; they usually lack an explicit mechanism to determine whether the current step is consistent with the task context and whether it effectively supports the current subgoal.
“current multi-step agents remain limited in process-level control. Although they can generate subgoals or action sequences, they usually lack an explicit mechanism to determine whether the current step is consistent with the task context and whether it effectively supports the current subgoal.”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
The Introduction and §6.1 say that multi-step reasoning models make errors and that outcome supervision does not identify which step went wrong. But the paper studies mathematical solutions, not multi-step agents generally, and does not establish that agents usually lack a mechanism for checking task-context consistency or support for a current subgoal. Indeed, §2.6 describes a reward model that predicts the correctness of each step. (LLM, conf=0.84)
Experimental — accuracy not yet measuredCobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Lightman et al. (2023)
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredWhile [186] did not find any major downstream performance improvement with segment-level feedback, [97] found segment-level feedback to be significantly more useful when training larger models on a larger feedback dataset coupled with evaluation on a more difficult task.
“Interestingly, while [186] did not find any major downstream performance improvement with their method, [97] used similar methodology but instead trained larger models on a larger feedback dataset coupled with evaluation on a more difficult task found segment-level feedback to be significantly more useful.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [97]
A more granular assessment can be particularly valuable because it allows users to pinpoint where inaccuracies occur and serves as a reward model for developers to refine their generative systems.
“A more granular assessment can be particularly valuable because it (1) not only allows users to pinpoint where inaccuracies occur (Liu et al., 2021) but also (2) serves as a reward model for developers to refine their generative systems (Lightman et al., 2023).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Lightman et al., 2023
Process rewards outperform outcome rewards.
“Lightman et al. [130], whose finding that process rewards outperform outcome rewards seeded today's process-reward-model (PRM) literature”
From Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) · cites this paper as [130]
Let's Verify Step by Step showed that process-level signals outperform outcome-only supervision on mathematical reasoning.
“Let's Verify Step by Step Lightman et al. [2023] showed that process-level signals outperform outcome-only supervision on mathematical reasoning”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Lightman et al. [2023]
One iteration of MAGICORE outperforms Best-of-k by 3.2% when Best-of-k uses k=120.
“just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Lightman et al., 2023
PRM800K uses a generator LLM that was heavily fine-tuned for math-specific problems.
“Additionally, our dataset explores the more general case of prompting API-based LLMs, whereas PRM800K uses a generator LLM that was heavily fine-tuned for math-specific problems.”
From LLMs cannot find reasoning errors, but can correct them given the error location (2024) · cites this paper as Lightman et al., 2023
Process reward models extend this to step-level supervision over reasoning traces.
“process reward models [26] extend this to step-level supervision over reasoning traces.”
From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [26]
Step-level supervision is beneficial for reasoning verification.
“Step-level supervision has also been shown to be beneficial for reasoning verification [9].”
From A Closed-Loop Modular Language Agent with Step Verification and Local Correction for Multi-Step Task Solving (2026) · cites this paper as [9]
The observation that more difficult tasks incur more severe performance drop remains consistent when examining accuracy at the five difficulty levels defined by MATH-500.
“In addition to Table 1, we also examine the accuracy at the 5 different difficulty levels explicitly defined by MATH-500 (Lightman et al., 2023) in Appendix B.2, where the observations remain consistent.”
From Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models (2025) · cites this paper as Lightman et al., 2023
When using model-generated chains of thought, the intermediate steps may not always be correct.
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Lightman et al., 2023
Recent studies have shown that under equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable to thinking models on benchmarks like MATH500 and AIME24.
“Recent studies [35, 41] have explored this question with established math benchmarks by comparing the upper-bound capabilities (pass@k) of RL-based reasoning (thinking) models with their non-reasoning (non-thinking) standard LLM counterparts. They have shown that under equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable to thinking models on benchmarks”
From The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025) · cites this paper as [42]
Section 3 (Large-scale Supervision) compares process-supervised and outcome-supervised reward models as they search over increasing numbers of solutions; Appendix C describes evaluation on 500 held-out MATH problems. Neither passage compares thinking with non-thinking LLMs under equivalent inference-token budgets or reports comparable performance on MATH500 and AIME24 (searched: pass@k, inference budget, MATH500, AIME24). (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 24
- A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. · arXiv
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. · arXiv
- T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017. · arXiv
- L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti- mization. arXiv preprint arXiv:2210.10760, 2022. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. · arXiv
- A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra- masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. · arXiv
- Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022. · arXiv
- J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661, 2020. · arXiv
- R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
- E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. In Proceedings of the 13th ACM SIGGRAPH Con- ference on Motion, Interaction and Games, pages 1-10, 2020.
- M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. · arXiv
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. · arXiv
- J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021. · arXiv
- N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feed- back. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
- J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. · arXiv
- X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. · arXiv
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. · arXiv
- E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reason- ing with reasoning. Advances in Neural Information Processing Systems, 35: 15476-15488, 2022.
- D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human pref- erences. arXiv preprint arXiv:1909.08593, 2019. Problem · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Hicks, M. T., Humphries, J., & Slater, J. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09775-5 · Cited by 456 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Pre-Trained Language Models for Text Generation: A Survey
Li, J., Tang, T., Zhao, W. X., Nie, J., & Wen, J.-R. (2024) · ACM Computing Surveys · doi:10.1145/3649449 · Cited by 304 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230 · see citing works
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Large language models streamline automated machine learning for clinical studies
Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J. N., Truhn, D., & Nebelung, S. (2024) · Nature Communications · doi:10.1038/s41467-024-45879-8 · Cited by 114 · see citing works
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Large Language Models are Better Reasoners with Self-Verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., & Zhao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.167 · Cited by 110 · see citing works
Pardos, Z. A., & Bhandari, S. (2024) · PLOS ONE · doi:10.1371/journal.pone.0304013 · Cited by 97 · see citing works
“The 32% error rate is very close to the 27% recently reported by OpenAI using ChatGPT 4 to answer AP Calculus, Physics, and Chemistry questions [64].”
MethodOpen Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“To address this, some works have trained LLMs to better solve multi-step math problems with process supervision [Uesato et al., 2022, Lightman et al., 2023].”
MethodPan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Harnessing LLMs for Temporal Data - A Study on Explainable Financial Time Series Forecasting
Yu, X., Chen, Z., & Lu, Y.-B. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-industry.69 · Cited by 75 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76 · see citing works
Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective
Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.15408 · Cited by 69 · see citing works
ChatGPT hallucinating: can it get any more humanlike?
Siontis, K. C., Attia, Z. I., Asirvatham, S. J., & Friedman, P. A. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad766 · Cited by 54 · see citing works
Shojaee, P., Mirzadeh, I., Alizadeh vahid, K., Horton, M., Bengio, S., & Farajtabar, M. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3603 · Cited by 47 · see citing works
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023) train a verifier or a critique model on a high-quality dataset to verify or refine LLMs’ outputs.”
BackgroundSiepmann, R. M., Huppertz, M., Rastkhiz, A., Reen, M., Corban, E., Schmidt, C., Wilke, S., Schad, P., Yüksel, C., Kuhl, C. K., et al. (2024) · European Radiology · doi:10.1007/s00330-024-10727-2 · Cited by 38 · see citing works
On large language models safety, security, and privacy: A survey
Zhang, R., Li, H., Qian, X., Jiang, W., & Chen, H.-X. (2025) · Journal of Electronic Science and Technology · doi:10.1016/j.jnlest.2025.100301 · Cited by 45 · see citing works
Domain-specific chatbots for science using embeddings
Yager, K. G. (2023) · Digital Discovery · doi:10.1039/d3dd00112a · Cited by 31 · see citing works
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29 · see citing works
“A more granular assessment can be particularly valuable because it (1) not only allows users to pinpoint where inaccuracies occur (Liu et al., 2021) but also (2) serves as a reward model for developers to refine their generative systems (Lightman et al., 2023).”
BackgroundLLMs cannot find reasoning errors, but can correct them given the error location
Tyen, G., Mansoor, H., Carbune, V., Chen, P., & Mak, T. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.826 · Cited by 24 · see citing works
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 26 · see citing works
Critique of impure reason: Unveiling the reasoning behaviour of medical large language models
Sim, S. Z. Y., & Chen, T. (2025) · eLife · doi:10.7554/elife.106187 · Cited by 24 · see citing works
Duan, J. (2024) · International Journal of New Developments in Education · doi:10.25236/ijnde.2024.060205 · Cited by 23 · see citing works
Repeatability of Fine-Tuning Large Language Models Illustrated Using QLoRA
Alahmari, S. S., Hall, L., Mouton, P. R., & Goldgof, D. B. (2024) · IEEE Access · doi:10.1109/access.2024.3470850 · Cited by 27 · see citing works
Transforming agency: On the mode of existence of large language models
Barandiaran, X. E., & Almendros, L. S. (2025) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-025-10094-3 · Cited by 24 · see citing works
Nakamoto, R., Flanagan, B., Yamauchi, T., Dai, Y., Takami, K., & Ogata, H. (2023) · Computers · doi:10.3390/computers12110217 · Cited by 17 · see citing works
Agent Lumos: Unified and Modular Training for Open-Source Language Agents
Yin, D., Brahman, F., Ravichander, A., Chandu, K., Chang, K.-W., Choi, Y., & Lin, B. Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.670 · Cited by 17 · see citing works
Generation Next: Experimentation with AI
Charness, G. B., Jabarian, B., & List, J. A. (2023) · National Bureau of Economic Research · doi:10.3386/w31679 · Cited by 13 · see citing works
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12 · see citing works
“[...] This confirms the energy spectrum is: ) .”
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.538 · Cited by 12 · see citing works
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models
Liu, J., Huang, Z., Liu, Q., Ma, Z., Zhai, C., & Chen, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3556367 · Cited by 11 · see citing works
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.11018 · Cited by 35 · see citing works
“We use the MATH500 test set from Light-4 man et al. [32] to report performance, and our 4-shot MATH prompt for data generation can be found in Figure A.12.”
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 10 · see citing works
“Lightman et al. (2023) constructed a large dataset with step-wise correctness labels from human annotators, and fine-tuned a GPT-4 (OpenAI, 2024) model on it.”
MethodLehmann, J., Bhandiwad, D., Gattogi, P., & Vahdati, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00671 · Cited by 11 · see citing works
The Hallucination Tax of Reinforcement Finetuning
Song, L., Shi, T., & Zhao, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.112 · Cited by 10 · see citing works
Legally Enforceable Hate Speech Detection for Public Forums
Luo, C., Bhambhoria, R. V., Dahan, S., & Zhu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.730 · Cited by 9 · see citing works
Democratizing Reasoning Ability: Tailored Learning from Large Language Model
Wang, Z., Huang, S., Liu, Y., Wang, J., Song, M., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.120 · Cited by 14 · see citing works
Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
Winata, G. I., Zhao, H., Das, A., Tang, W., Yao, D. D. W., Zhang, S.-X., & Sahu, S. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.17541 · Cited by 11 · see citing works
Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning
Yen, A.-Z., & Hsu, W.-L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.201 · Cited by 8 · see citing works
Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Liu, Z., Hao, Z., Chen, Y., Wang, H., Hou, J., Ding, R., Yang, Y., Ji, W., Xia, W., & Liu, F. (2026) · arXiv · Cited by 7 · see citing works
Revolutionizing High School Physics Education: A Novel Dataset
Anand, A., Addala, K., Baghel, K. K. S., Goel, A., Hira, M., Gupta, R., & Shah, R. R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49601-1_5 · Cited by 8 · see citing works
Evaluation of Medium-large Language Models at Zero-shot Closed Book Generative Question Answering
Peinl, R., & Wirth, J. (2024) · doi:10.5121/csit.2024.140106 · Cited by 8 · see citing works
GPT-in-the-Loop: Supporting Adaptation in Multiagent Systems
Nascimento, N., Alencar, P. S. C., & Cowan, D. D. (2023) · BigData Congress [Services Society] · doi:10.1109/bigdata59044.2023.10386490 · Cited by 9 · see citing works
XAI for medicine by ChatGPT code interpreter
Kitamura, K., Irvan, M., & Shigetomi Yamaguchi, R. (2023) · Proceedings of the 2023 5th International Conference on Big-data Service and Intelligent Computation · doi:10.1145/3633624.3633629 · Cited by 5 · see citing works
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 17 · see citing works
Assessing Hallucination in Large Language Models Under Adversarial Attacks
Tripathi, S., Griffith, H., & Rathore, H. (2024) · International Conference on Mobile and Secure Services (MobiSecServ) · doi:10.1109/mobisecserv63327.2024.10759969 · Cited by 5 · see citing works
Gentopia.AI: A Collaborative Platform for Tool-Augmented LLMs
Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., & Xu, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.20 · Cited by 4 · see citing works
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
Wu, H., Hui, W., Chen, Y., Wu, W., Tu, K., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.427 · Cited by 4 · see citing works
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Chen, J., Prasad, A., Saha, S., Stengel-Eskin, E., & Bansal, M. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1660 · Cited by 3 · see citing works
MedS³: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision
Jiang, S., Liao, Y., Chen, Z., Zhang, Y., Wang, Y., & Wang, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i37.40395 · Cited by 4 · see citing works
Free-MAD: Consensus-Free Multi-Agent Debate
Cui, Y., Fu, H., Zhang, H., Wang, L., & Zuo, C. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1600 · Cited by 3 · see citing works
Faithful Self-Refinement in Mathematical Reasoning via Progressive Back-Translation
Liao, H., Zhu, Z., Hu, S., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890054 · Cited by 3 · see citing works
System Description: DeepLLM, Casting Dialog Threads into Logic Programs
Tarau, P. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2300-3_7 · Cited by 3 · see citing works
Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration
Wan, F., Huang, X., Yang, T., Quan, X., Bi, W., & Shi, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.587 · Cited by 7 · see citing works
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 2 · see citing works
“For instance, DeepSeek-R1 [33] undergoes multiple rounds of SFT and RL training, emphasizing structured thinking templates and rule-based reward mechanisms.”
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Zhai, Y., Liu, H., Zhang, Z., Lin, T., Xu, K., Yang, C., Feng, D., Ding, B., & Wang, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3729965 · Cited by 3 · see citing works
Selective Propositional Reasoning: A Cognitive Load-Aware Strategy for Enhanced Reasoning
Yue, Y., Lei, Y., Shi, W., & Zhou, Y. (2023) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase60015.2023.10439142 · Cited by 2 · see citing works
Evaluation of Medium-Sized Language Models in German and English Language
Peinl, R., & Wirth, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13101 · Cited by 2 · see citing works
Cultural and technological change in the future information environment
Slapakova, L., Fraser, A., Hughes, M., Aquilino, M., & Thue, K. (2024) · RAND Corporation eBooks · doi:10.7249/rra2662-1 · Cited by 2 · see citing works
ArithmeticGPT: empowering small-size large language models with advanced arithmetic skills
Liu, Z., Zheng, Y., Yin, Z., Chen, J., Liu, T., Mi, T., & Luo, W. (2025) · Machine Learning · doi:10.1007/s10994-024-06681-1 · Cited by 2 · see citing works
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1 · see citing works
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · see citing works
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 2 · see citing works
ELWARD : Empowering Language Model With World Insights and Human‐Aligned Reward Design
Yong-ping, D., Li, S., Yan, R., Hou, Y., & Han, H. (2025) · Expert Systems · doi:10.1111/exsy.70055 · Cited by 1 · see citing works
Embodied Reasoning with Self-Feedback
Kak, P., & Jain, S. (2024) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct62155.2024.10677317 · Cited by 1 · see citing works
Artificial Intelligence in Math Education of Engineers
Schönle, M. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016373 · Cited by 1 · see citing works
Advance Toward Artificial Superintelligence with OpenAI's O1 Reinforcement Learning and Ethics
Mirindi, D., Sinkhonde, D., Mirindi, F., & Bezabih, T. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077494 · Cited by 1 · see citing works
Calibrating Conservatism for Scalable Oversight
Overman, W., & Bayati, M. (2026) · arXiv · doi:10.48550/arxiv.2605.28807
“…demonstrates that written principles can partially substitute for continuous human judgment by enabling AI-generated feedback, while process supervision (Lightman et al., 2024) shows that supervising intermediate reasoning steps can outperform outcome-only supervision in mathematical reasoning.”
ARS: Adaptive Reasoning Suppression for Efficient Large Reasoning Language Models
Zheng, D. (2025) · arXiv · doi:10.48550/arxiv.2510.00071
“The adaptive mechanism ensures that suppression occurs only when true certainty exceeds the optimal threshold, with the error term diminishing as checkpoints increase. policy ← CoDFastPolicy(drafts=2, per_draft=10) soning benchmarks including MATH500 [12] and GSM8K.”
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Liu, R., Sun, Y., Zhang, M., Bai, H., Yu, X., Yu, T., Yuan, C., & Hou, L. (2025) · arXiv · doi:10.48550/arxiv.2504.04823
“In addition to Table 1, we also examine the accuracy at the 5 different difficulty levels explicitly defined by MATH-500 (Lightman et al., 2023) in Appendix B.2, where the observations remain consistent.”
Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents
Zhang, X., Wang, G., Cui, Y., Qiu, W., Li, Z., Zhu, B., & He, P. (2026) · arXiv
“Constitutional AI (Bai et al., 2022) uses pre-specified rules; reward design methods (PBRS (Ng et al., 1999), process rewards (Lightman et al., 2023), rule-based rewards (Shao et al., 2024)) encode L 3 -type knowledge but are human-designed, not learned.”
CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization
Shah, D., Badhe, S., Kathrotia, N., & Tiwari, P. (2026) · arXiv
ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Lee, H., Oh, S., Kim, J., Shin, J., & Tack, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14565
“As shown in Figure 8, in MATH-500, ReVISE trained on Llama-3.”
InfluentialCritique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Zhang, X., Zhang, Y., Sun, H., Feng, K., Lu, C., Yang, C., & Meng, H. (2025) · arXiv · doi:10.48550/arxiv.2506.03106
“Many studies [11, 12, 13, 14, 15] primarily use critiques for evaluation, transforming them into numerical rewards for model improvement via RL algorithms such as Proximal Policy Optimization (PPO) [16] or Group Relative Policy Optimization (GRPO) [17].”
MemGym: a Long-Horizon Memory Environment for LLM Agents
Xu, W., Wang, Y., Mei, K., Liang, K., Wang, Z., Jin, M., Zhang, H., Zhang, S.-X., Hua, W., Sahu, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20833
“Reward models trained from human or automated feedback have become a central tool for post-training LLMs [30], and process reward models [26] extend this to step-level supervision over reasoning traces.”
Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming
Chen, Z., & Ma, M. (2026) · arXiv · doi:10.48550/arxiv.2607.23019
“More broadly, process-level supervision has been motivated by the observation that final-answer correctness alone provides only a weak signal for reasoning quality Lightman et al. [2023].”
Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
Chen, H.-Y. (2026) · arXiv · doi:10.48550/arxiv.2604.21611
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
Chen, M., Wang, L., & Qu, B. (2026) · arXiv
Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K. (2026) · arXiv · doi:10.48550/arxiv.2607.08700
“Each LLM-judged criterion is an independent rubric judgment structurally similar to a process reward model (PRM) step [Lightman et al., 2024], a per-step judgment used as an intermediate reward signal, and the task is narrow enough that a well-specified rubric may compensate for model scale.”
Li, H., Feng, L., & Wang, D. (2026) · Electronics · doi:10.3390/electronics15102011
Li, Z., Li, Z., Shi, Y., Wang, R., Yang, J., Liu, Z., Wu, X., Li, A., Huang, C., Shang, E., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.08964
Stealing Reasoning Traces from Proprietary LLM APIs
Panfilov, A., Schmotz, D., Shumailov, I., Beurer-Kellner, L., Schaeffer, J., Prabhu, A., Geiping, J., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2608.09867
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
Decentralized Multi-Agent Systems with Shared Context
Mao, Y., Gu, J., Chauhan, A., Zhang, Q., Kang, H., & Mirhoseini, A. (2026) · arXiv · doi:10.48550/arxiv.2606.10662
“If a sub-agent receives insufficient context, it return control to the main agent, triggering additional retrieval or another delegation round.”
- Authors
- Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.20050
- OpenAlex
- W4379089709
- Cited by
- 65
- References
- 0
- Citations by year
- 2026: 3 · 2025: 23 · 2024: 20 · 2023: 19
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- process supervision, multi-step reasoning, large language models, active learning (machine learning), reward modeling, mathematical problem solving, logical errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
