The DOI matches its DataCite record; 29 of the 29 references it cites were found in a registry or an index.
Training Verifiers to Solve Math Word Problems
Open accessAbstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line of research utilizes majority voting to choose candidates based on their execution performance [89, 90].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [90]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Cobbe et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [14]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within tasks, feedback refines solutions or guides actions [1, 2].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GSM8K (Grade School Math 8K) is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Each problem is designed to be solvable using only basic arithmetic operations (+, -, ×, ÷) with a difficulty level appropriate for grade school students.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Knowledge-oriented benchmarks (e.g., MMLU [325] and C-Eval [335]) assess models' world knowledge retention, while reasoning-oriented benchmarks (e.g., GSM8K [318], BBH [336], and MATH [317]) evaluate their ability to solve complex reasoning tasks.”
From A Survey of Large Language Models (2026) · cites this paper as [318]
The cited source says what the claim attributes to it.
In the citing paper
“Best-ofN (Cobbe et al., 2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential.”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Notably, ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 → 31.1% on GSM8K (Maj@3) (Cobbe et al., 2021) with Llama3 1B (Dubey et al., 2024)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM8K (Cobbe et al., 2021) provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many benchmarks like MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al., 2021) and GSM-Hard (Gao et al., 2023) math tasks, respectively, after transferring across domains.”
From Automated Design of Agentic Systems (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Each GSM8K is paired with a numerical reference answer.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Given these criteria, the mathematical reasoning benchmark datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) foot_0 emerge as fitting choices for our test bed, which allow us to study data leakage in depth but also offer a relevant and challenging environment.”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“(i) Math Word Problems test reasoning on math problems similar to GSM8k (Cobbe et al., 2021) but with varying levels of relational complexity, particularly in their degree of variable interdependence, which we define as the number of other variables on which a single variable's value depends.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This category (21 problems) is inspired by the grade-school math problems in datasets such as GSM8k (Cobbe et al., 2021), but introduces an additional dimension of variable interdependence (i.e., higher relational complexity between the variables).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM1k is modelled after the GSM8k dataset (Cobbe et al. [2021]), released by OpenAI in 2021, which consists of 8.5k grade school math problems.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [18]
Experimental — accuracy not yet measuredIn the citing paper
“Multi-step reasoning in LLMs typically involves executing sequential logical operations-typically requiring 3-10 intermediate steps-to arrive at solutions that cannot be derived through single-step inference (Cobbe et al., 2021; Yang et al., 2018; Suzgun et al., 2022).”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Cobbe et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Specifically, we apply residual dropout (Vaswani et al., 2017) along the residual paths of each layer in the network.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
BackgroundSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”
BackgroundSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 822 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Deep Learning for Symbolic MathematicsPreprint
Lample, G., & Charton, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01412 · Cited by 405 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
Background
Cited by
Sorted by citations
Items 501–600 of 792
Gupta, P., Jain, N., & Bhat, A. (2024) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic60222.2024.10575336 · Cited by 2
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 3
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
Fan, H., Zhou, H., Huang, G., Raman, P., Fu, X., Gupta, G., Ram, D., Wang, Y., & Huan, J. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825098 · Cited by 9
Towards Reflexive AI: A Comprehensive Exploration of Enhancing Social Science Research Through NLP
Shan, S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-84460-7_49 · Cited by 4
Bootstrapping LLM-based Fact-checking via Iterative Rationalization Finetuning
Lyu, X., Cao, C., Sun, M., Liang, B., Yao, L., & Xu, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887888 · Cited by 3
Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning
Puccioni, L., Farshin, A., Scazzariello, M., Wang, C., Chiesa, M., & Kostić, D. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00028 · Cited by 2
Fostering collective intelligence in CPSS: an LLM-driven multi-agent cooperative tuning framework
Chen, R., & He, C. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1613499 · Cited by 4
Conversational LLM-Based Decision Support for Defect Classification in AFM Images
Biswas, A., Rade, J., Masud, N., Hasib, M. H. H., Balu, A., Zhang, J., Sarkar, S., Krishnamurthy, A., Ren, J., & Sarkar, A. (2025) · IEEE Open Journal of Instrumentation and Measurement · doi:10.1109/ojim.2025.3592284 · Cited by 4
Solving Math Word Problems Following Logically Consistent Template
Huang, Z., Zhang, X., Bai, J., Rong, W., Ouyang, Y., & Xiong, Z. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191776 · Cited by 2
FERMAT: An Alternative to Accuracy for Numerical Reasoning
Sivakumar, J., & Moosavi, N. S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.838 · Cited by 2
20Q: Overlap-Free World Knowledge Benchmark for Language Models
Bruyn, M. D., Lotfi, E., Buhmann, J., & Daelemans, W. M. P. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.46 · Cited by 2
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0003 · Cited by 2
AI and the UN Sustainable Development Goals
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0012 · Cited by 2
MWPRanker: An Expression Similarity Based Math Word Problem Retriever
Goel, M., Venktesh, V., & Goyal, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43430-3_33 · Cited by 2
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Xiong, J., Shen, J., Yuan, Y., Wang, H., Yin, Y., Liu, Z., Li, L., Guo, Z., Cao, Q., Huang, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.711 · Cited by 2
Tang, F., Gao, W., Peng, L., & Zhan, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-0316-6_9 · Cited by 4
Paraphrasing in the System of Automatic Solution of Planimetric Problems
Kurbatov, S. S. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-53552-9_20 · Cited by 2
DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation
Gu, Z., Sun, X., Lian, F., Kang, Z., Xu, C., & Fan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29768 · Cited by 3
CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation
You, W., Wang, P., Li, C., Ji, Z., & Bai, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29914 · Cited by 3
Distilling Mathematical Reasoning Capabilities into Small Language ModelsPreprint
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4782551 · Cited by 2
Lee, J., Sakaguchi, K., & Bak, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.528 · Cited by 2
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., & Zhao, T. (2024) · arXiv · doi:10.48550/arxiv.2403.05527 · Cited by 2
“However, a stark contrast emerges when applying these methods to complex generative tasks that require models to generate longer responses or involve reasoning, such as mathematical problem-solving (Cobbe et al., 2021) and chain-of-thought (CoT) reasoning (Wei et al., 2023).”
BackgroundCosmos-Reason1: From Physical Common Sense To Embodied Reasoning
NVIDIA, :, Azzolini, A., Bai, J., Brandon, H., Cao, J., Chattopadhyay, P., Chen, H., Chu, J., Cui, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.15558 · Cited by 187
Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks
University of Montreal, M. Q. A. I., & Hegazy, M. (2024) · Journal of Robotics and Automation Research · doi:10.33140/jrar.05.03.06 · Cited by 1
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“Mathematics reasoning : AIME (MAA, 2024), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b).”
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 21
“1. credentials. txt 2. installation_notes. txt 3. main. py 4. launcher_win. exe In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering…”
Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate
Wynn, A., Satija, H., & Hadfield, G. (2025) · arXiv · doi:10.48550/arxiv.2509.05396 · Cited by 96
“We describe the set of tasks, models and prompts used in our experiments in this section.”
Dedieu, A., Lehrach, W., Zhou, G., George, D., & Lázaro-Gredilla, M. (2024) · arXiv · doi:10.48550/arxiv.2401.05946 · Cited by 2
“Planning with LLMs: Despite their success on simple benchmarks including maths (Cobbe et al., 2021) and logics (Srivastava et al., 2022), growing evidence suggests that LLMs performance collapse on benchmarks that require stronger planning skills (Valmeekam et al., 2022).”
BackgroundFact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1
From Reasoning to Code: GRPO Optimization for Underrepresented Languages
PENNINO, F., RAIMONDI, B., RONDELLI, M., GURIOLI, A., & GABBRIELLI, M. (2026) · Theory and Practice of Logic Programming · doi:10.1017/s1471068426100489 · Cited by 1
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 197
KVDirect: Distributed Disaggregated LLM Inference
Chen, S., Jiang, R., Yu, D., Xu, J., Chao, M., Meng, F., Jiang, C., Xu, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2501.14743 · Cited by 1
“ShareGPT [8] is a collection of user-shared conversations with ChatGPT.”
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., & Stoica, I. (2025) · arXiv · doi:10.48550/arxiv.2504.07164 · Cited by 1
“In addition to the execution-based “testing agents”, we also explore the execution-free outcome-supervised reward models (a.k.a verifiers) (Cobbe et al., 2021).”
RewardBench 2: Advancing Reward Model Evaluation
Malik, S., Pyatkin, V., Land, S., Morrison, J., Smith, N. A., Hajishirzi, H., & Lambert, N. (2025) · arXiv · doi:10.48550/arxiv.2506.01937 · Cited by 1
“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0006 · Cited by 1
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 5
Mitigating Political Bias in Large Language Models Using Chain of thought Prompting Techniques
Poosarla, H. (2024) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2024.58057 · Cited by 1
In-Context Learning for Scalable and Online Hallucination Detection in RAGS
Albanese, N. C. (2024) · doi:10.5121/csit.2024.140703 · Cited by 1
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
MetaphorPrompt - An Analogical Reasoning Approach for Extracting Causal Links from Biological Text
Patel, P., Chiu, Y., Huang, Y., & Zhang, J. (2024) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3698587.3701384 · Cited by 1
SPRInT: Scaling Programmatic Reasoning for INstruction Tuning in mathematics
Yan, Y., Li, L., & Zhang, B.-W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129868 · Cited by 1
WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 21
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1
Evaluating Logical Reasoning Ability of Large Language Models
Chan, E., & Chan, A. C.-F. (2026) · doi:10.1109/icnlp69856.2026.11527900 · Cited by 1
Cost, Complexity, and Efficacy of Prompt Engineering Techniques for Large Language Models
Cherukuri, M. (2025) · International Journal on Science and Technology · doi:10.71097/ijsat.v16.i2.2584 · Cited by 1
The superalignment of superhuman intelligence with large language models
Huang, M., Wang, Y., Cui, S., Ke, P., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4348-6 · Cited by 3
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Code Ranking with Structure Awareness Contrastive Learning
Huang, H., Cao, L., Wang, J., Yu, T., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00053 · Cited by 1
KAFA-Merge: Model Merging Method With Layer-Based Bayes+Linear Search
Yuce, M. K., & Amasyalı, M. F. (2025) · IEEE Access · doi:10.1109/access.2025.3582015 · Cited by 2
Towards enhanced LLM pretraining: Dynamic checkpoint merging via generation quality
Wang, Z., Liu, D., Li, C., Chu, D., Chen, W., Wang, B., & Sui, D. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103415 · Cited by 1
Research on enhancing model performance by merging with Korean language models
Cho, T., Kim, R., & Choi, A. J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111686 · Cited by 1
GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models
Bandooni, A., & Subburaj, B. (2025) · International Conference on Recent Advances in Information Technology · doi:10.1109/rait65068.2025.11089442 · Cited by 1
Learning Optimal Agent Behavior From a Synthetic Reasoning Action Dataset
Dina, G.-A., & Ionescu, B. (2025) · International Symposium on Signals, Circuits and Systems · doi:10.1109/isscs66034.2025.11105702 · Cited by 1
Efficient Cross-Lingual Transfer for Language Models
Shah, A. F. M. S. (2025) · International Symposium on Innovative Approaches in Smart Technologies (ISAS) · doi:10.1109/isas66241.2025.11101933 · Cited by 1
AIPO: Automatic Instruction Prompt Optimization by model itself with “Gradient Ascent”
Park, K., & Kim, D. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101889 · Cited by 1
EMRDCM : An experience-enhanced multi-role debate method based on compromise mechanisms
Zhang, Y., Feng, J., Yiming, X., Wang, Z., Yang, T., & Wang, X. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01022-1 · Cited by 1
Kokate, R., & Onkar, P. S. (2026) · Design Science · doi:10.1017/dsj.2025.10044 · Cited by 4
Sheng, C. Y., Xin, C. Y., & Jie, T. G. (2026) · Cureus Journal of Computer Science. · doi:10.7759/s44389-026-00047-5 · Cited by 1
GradeSQL: Outcome reward models for intelligent Text-to-SQL generation from LLMs
Tritto, M., Farano, G., Palma, D. D., Rossiello, G., Subramanian, D., Narducci, F., & Noia, T. D. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01071-6 · Cited by 1
“Common approaches include Best-of-N (BoN), such as Cobbe et al. (2021), which samples N outputs and selects the highest-scoring candidate, and Majority Voting (Maj) (Sheng and Shuai, 2025), which chooses the most frequently generated output based on execution equivalence.”
Foundation Models for Text Generation
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_6 · Cited by 3
CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
Dong, Y., Martin, L. J., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.832 · Cited by 2
Soft Ethics and the Governance of AI
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0006 · Cited by 1
Is word order considered by foundation models? A comparative task-oriented analysis
Zhao, Q., Li, J., Liu, J., Kang, Z., & Zhou, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122700 · Cited by 3
A Generation-based Deductive Method for Math Word Problems
Hu, Y., Zhang, J., Li, H., Li, C., & Chen, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.108 · Cited by 1
Annotate French Clinical Data Using Large Language Model Predictions
Méoni, S., Ryffel, T., & de la Clergerie, É. V. (2023) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi57859.2023.00099 · Cited by 1
Li, W., & Pan, W. (2024) · International Conference on Electronic Information Engineering, Big Data and Computer Technology (EIBDCT) · doi:10.1117/12.3031176 · Cited by 1
ToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
Automatic Prompt Generation Based on Combinatorial Relationships in Reasoning Problems
Wu, T., & He, H. (2024) · doi:10.1109/icnlp60986.2024.10692467 · Cited by 1
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 2
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 1
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
ConfidenceCal: Enhancing LLMs Reliability through Confidence Calibration in Multi-Agent Debate
Bai, Y. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10808396 · Cited by 2
Benchmark Study for Evaluating the Korean Comprehension of Large Language Models
Seo, M., Jeong, Y., Lee, H., Im, H., & Jang, J. (2024) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2024.27.9.007 · Cited by 1
IDE: A Multi-Agent-Driven Iterative Framework for Dynamic Evaluation of LLMs
Tong, X., Jin, B., Wang, J., Wu, X., Xia, T., & Han, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890123 · Cited by 3
Alotaibi, N. N., Saberi, M., Bandara, M., Hussain, O. K., & Al-Najjar, H. A. H. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00087 · Cited by 2
Artificial Intelligence in Math Education of Engineers
Schönle, M. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016373 · Cited by 1
Explaining LLM Decisions: Counterfactual Chain-of-Thought Approach
Sarkar, P., Prakash, A. V., & Singh, J. B. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96473-2_18 · Cited by 1
Advance Toward Artificial Superintelligence with OpenAI's O1 Reinforcement Learning and Ethics
Mirindi, D., Sinkhonde, D., Mirindi, F., & Bezabih, T. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077494 · Cited by 1
Wei, Y., & Fan, J. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01236-0 · Cited by 1
GATmath and GATLc: Comprehensive benchmarks for evaluating Arabic large language models
AlBallaa, S., Al-Twairesh, N., Al‐Salman, A. S., & Alfarhood, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329129 · Cited by 3
Bueno, M. C., Gemmell, C., Dalton, J., Lotufo, R., & Nogueira, R. F. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.3 · Cited by 7
Empirical Investigation of Neural Symbolic Reasoning Strategies
Aoki, Y., Kudo, K., Kuribayashi, T., Brassard, A., Yoshikawa, M., Sakaguchi, K., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.86 · Cited by 1
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0003 · Cited by 1
Studies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language ModelsPreprint
Saqur, R., Kratsios, A., Horvath, B. N., Tian, J.-J., Willes, J., Krach, F., Limmer, Y., & Rudzicz, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4856254 · Cited by 1
Infinite-parameter Large Language ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-7khtw
Self-Pluralising Culture Alignment for Large Language ModelsPreprint
Xu, S., Leng, Y., Yu, L., & Xiong, D. (2024) · Qeios · doi:10.32388/q7ywfv · Cited by 1
Cao, J., & Lu, Y. (2025) · Qeios · doi:10.32388/mlaotg · Cited by 1
ECCOS: Efficient Capability and Cost Coordinated Scheduling for Multi-LLM ServingPreprint
Mei, K., Xu, W., Lin, S., & Zhang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5159339 · Cited by 1
Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity
Yang, Y., Qu, C., Wen, M., Shi, L., Wen, Y., Zhang, W., Wierman, A., & Gu, S. (2026) · arXiv · doi:10.48550/arxiv.2602.03794 · Cited by 1
Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Ahuja, K., Sclar, M., & Tsvetkov, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.11900 · Cited by 1
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning
Han, X., Jian, Y., Hu, X., Liu, H., Wang, Y., Fan, Q., Ai, Y., Huang, H., He, R., Yang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12568 · Cited by 1
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Beck, M., Pöppel, K., Lippe, P., Kurle, R., Blies, P. M., Klambauer, G., Böck, S., & Hochreiter, S. (2025) · arXiv · doi:10.48550/arxiv.2503.13427 · Cited by 9
“1 (Lambert et al., 2024), OpenHermes 2.5 (Teknium, 2023), GSM8K (Cobbe et al., 2021), and Smoltalk (subsets magpie-ultra, longalign, and self-oss-instruct) (Allal et al., 2024).”
InfluentialLatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
Zhang, G., Meng, F., Wan, G., Li, Z., Wang, K., Yin, Z., Bai, L., & Yan, S. (2025) · arXiv · doi:10.48550/arxiv.2509.24771
“…in the (I) explicit, natural-language space , as in self-correction (Shinn et al., 2023; Gou et al., 2024; Kang et al., 2025), feedback modeling (Cobbe et al., 2021; Yu et al., 2025), or repeated sampling (Gui et al., 2024; Ye et al., 2025); or it may operate in the (II) latent space , where…”
InfluentialiCLP: Large Language Model Reasoning with Implicit Cognition Latent Planning
Chen, S., & Niu, D. (2025) · arXiv · doi:10.48550/arxiv.2512.24014
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
Wunderlich, F. V., Kaesberg, L. B., Wahle, J. P., Ruas, T., & Gipp, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · doi:10.18653/v1/2026.acl-srw.1 · Cited by 5
“Huang et al. (2024) compare debate with self-consistency on the GMS8K benchmark (Cobbe et al., 2021) for an equal number of generations.”
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Liu, R., Sun, Y., Zhang, M., Bai, H., Yu, X., Yu, T., Yuan, C., & Hou, L. (2025) · arXiv · doi:10.48550/arxiv.2504.04823
“MATH-500 contains a mix of easy and hard prob-6 lems, followed by GSM8K with easy primary school-level questions.”
Influential
Metadata
- Authors
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J.
- ORCID
- Kosaraju, Vineet
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.14168
- OpenAlex
- W3210904070
- Cited by
- 575
- References
- 28
- Citations by year
- 2026: 30 · 2025: 137 · 2024: 156 · 2023: 216 · 2022: 34 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Education › Mathematics Education and Pedagogy
- Keywords
- mathematical word problems, large language models, solution verification, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
