The DOI matches its DataCite record; 29 of the 29 references it cites were found in a registry or an index.
Training Verifiers to Solve Math Word Problems
Open accessAbstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line of research utilizes majority voting to choose candidates based on their execution performance [89, 90].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [90]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Cobbe et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [14]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within tasks, feedback refines solutions or guides actions [1, 2].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GSM8K (Grade School Math 8K) is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Each problem is designed to be solvable using only basic arithmetic operations (+, -, ×, ÷) with a difficulty level appropriate for grade school students.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Knowledge-oriented benchmarks (e.g., MMLU [325] and C-Eval [335]) assess models' world knowledge retention, while reasoning-oriented benchmarks (e.g., GSM8K [318], BBH [336], and MATH [317]) evaluate their ability to solve complex reasoning tasks.”
From A Survey of Large Language Models (2026) · cites this paper as [318]
The cited source says what the claim attributes to it.
In the citing paper
“Best-ofN (Cobbe et al., 2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential.”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Notably, ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 → 31.1% on GSM8K (Maj@3) (Cobbe et al., 2021) with Llama3 1B (Dubey et al., 2024)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM8K (Cobbe et al., 2021) provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many benchmarks like MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al., 2021) and GSM-Hard (Gao et al., 2023) math tasks, respectively, after transferring across domains.”
From Automated Design of Agentic Systems (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Each GSM8K is paired with a numerical reference answer.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Given these criteria, the mathematical reasoning benchmark datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) foot_0 emerge as fitting choices for our test bed, which allow us to study data leakage in depth but also offer a relevant and challenging environment.”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“(i) Math Word Problems test reasoning on math problems similar to GSM8k (Cobbe et al., 2021) but with varying levels of relational complexity, particularly in their degree of variable interdependence, which we define as the number of other variables on which a single variable's value depends.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This category (21 problems) is inspired by the grade-school math problems in datasets such as GSM8k (Cobbe et al., 2021), but introduces an additional dimension of variable interdependence (i.e., higher relational complexity between the variables).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM1k is modelled after the GSM8k dataset (Cobbe et al. [2021]), released by OpenAI in 2021, which consists of 8.5k grade school math problems.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [18]
Experimental — accuracy not yet measuredIn the citing paper
“Multi-step reasoning in LLMs typically involves executing sequential logical operations-typically requiring 3-10 intermediate steps-to arrive at solutions that cannot be derived through single-step inference (Cobbe et al., 2021; Yang et al., 2018; Suzgun et al., 2022).”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Cobbe et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Specifically, we apply residual dropout (Vaswani et al., 2017) along the residual paths of each layer in the network.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
BackgroundSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”
BackgroundSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 822 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Deep Learning for Symbolic MathematicsPreprint
Lample, G., & Charton, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01412 · Cited by 405 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
Background
Cited by
Sorted by citations
Items 401–500 of 791
Ko, E., & Kang, P. (2025) · IEEE Access · doi:10.1109/access.2025.3553870 · Cited by 8
TDML — A Trustworthy Distributed Machine Learning Framework
Wang, Z., Wang, Q. L., Yu, G., & Chen, S. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107951 · Cited by 6
Interpretable Math Word Problem Solution Generation via Step-by-step Planning
Zhang, M., Wang, Z., Yang, Z., Feng, W., & Lan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.379 · Cited by 27
ALERT: Adapt Language Models to Reasoning Tasks
Yu, P., Wang, T., Golovneva, O., AlKhamissi, B., Verma, S. S., Jin, Z., Ghosh, G., Diab, M., & Çelikyılmaz, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.60 · Cited by 25
A Unified Framework of Ethical Principles for AI
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0004 · Cited by 9
The silicon service spectrum: warmth and competence explain people's preferences for AI assistants
Hernandez, I. A., & Chekili, A. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1396533 · Cited by 6
Mitigating Hallucination Issues in Small-Parameter LLMs through Inter-Layer Contrastive Decoding
Li, F., zhang, X., & Zhang, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650644 · Cited by 5
SALVe: Semantic Alignment Verification for Floorplan Reconstruction from Sparse Panoramas
Lambert, J., Li, Y., Boyadzhiev, I., Wixson, L. E., Narayana, M., Hutchcroft, W., Hays, J. H., Dellaert, F., & Kang, S. B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19821-2_37 · Cited by 5
Solving Probability and Statistics Problems by Program SynthesisPreprint
Tang, L., Ke, E., Singh, N., Verma, N., & Drori, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08267 · Cited by 5
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4
“The first is the In-Context Learning benchmark, for which we select GSM8K [27] and Jailbreakv [28, 35, 36] to evaluate the performance of ChunkKV, furthermore, we also include multi-step reasoning LLM DeepSeek-R1-Distill-Llama-8B [29] to evaluate the performance of ChunkKV.”
InfluentialSafetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10
Gentopia.AI: A Collaborative Platform for Tool-Augmented LLMs
Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., & Xu, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.20 · Cited by 4
LoRA 2 : Multi-scale low-rank approximations for fine-tuning large language models
Zhang, J.-C., Xiong, Y.-J., Xia, C.-M., Zhu, D.-H., & Zhan, H.-J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130859 · Cited by 7
Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
Bao, R., Xue, N., Sun, Y., & Chen, Z. (2025) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/icccworkshops67136.2025.11147210 · Cited by 4
Gignac, G. E., & Ilić, D. (2025) · Intelligence · doi:10.1016/j.intell.2025.101922 · Cited by 4
AutoMathKG: The Automated Mathematical Knowledge Graph Based on LLM and Vector Database
Bian, R., Geng, Y., Yang, Z., & Cheng, B. (2025) · Computational Intelligence · doi:10.1111/coin.70096 · Cited by 7
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
Wu, H., Hui, W., Chen, Y., Wu, W., Tu, K., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.427 · Cited by 8
Kadlčík, M., Štefánik, M., Sotolář, O., & Martinek, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.742 · Cited by 6
Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
Zheng, H., & Saparov, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.277 · Cited by 7
What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
Zhang, L., Zhai, X., Zhao, Z., Wen, X., & Zhao, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00497 · Cited by 5
Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models
Haroutunian, L., Li, Z., Galescu, L., Cohen, P. R., Tumuluri, R., & Haffari, G. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.69 · Cited by 8
Dual-View Visual Contextualization for Web Navigation
Kil, J., Song, C. H., Zheng, B., Deng, X., Su, Y., & Chao, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01369 · Cited by 12
Trippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 5
Zesch, T., Hanses, M., Seidel, N., Aggarwal, P., Veiel, D., & de Witt, C. (2024) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet61869.2024.10837635 · Cited by 5
Deng, Y., Qiu, X., Tan, X., Qu, C., Pan, J., Cheng, Y., Xu, Y., & Chu, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887899 · Cited by 6
Importance Weighting Can Help Large Language Models Self-Improve
Jiang, C., Chan, C.-M., Xue, W., Liu, Q., & Guo, Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34602 · Cited by 14
LLM-ProS: Analyzing Large Language Models’ Performance in Competitive Problem Solving
Hossain, M. S., Tabassum, A., Arefin, M. F., & Zaman, T. S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00015 · Cited by 4
Enhancing math reasoning ability of large language models via computation logic graphs
Zhao, D., Han, D., Wu, J. X., He, Z., Ning, B., Yuan, Y., Li, Y., Wang, C., & Song, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113905 · Cited by 4
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0002 · Cited by 6
CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Kim, S., Joo, S. J., Jang, Y., Chae, H., & Yeo, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.23 · Cited by 5
Wu, Y., Dong, G., & Xu, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.741 · Cited by 8
Unsupervised Explanation Generation via Correct Instantiations
Cheng, S., Wu, Z., Chen, J., Li, Z., Liu, Y., & Kong, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26494 · Cited by 5
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4
CiteME: Can Language Models Accurately Cite Scientific Claims?
Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., & Bethge, M. (2024) · arXiv · doi:10.48550/arxiv.2407.12861 · Cited by 4
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 804
“5-0.5B on GSM8K.”
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“Early model-based efforts like BERTScore (Zhang et al., 2020) sought to improve on traditional n-gram metrics by recognizing high-quality responses that differ from the reference.”
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models’ perceived and actual capabilities [36].”
BackgroundToken-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2024) · arXiv · Cited by 31
“Additionally, it significantly impacts reasoning accuracy, resulting in over a 20% relative accuracy reduction on benchmarks such as GSM8K (Cobbe et al., 2021).”
Reliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3
“…many mathematical tasks [27], and the simple and repetitive pattern of reasoning required to solve the problems of the current reasoning benchmarks [11, 32, 28, 10, 19], we present a new dataset that (i) is certainly outside of current models’ training sets, and (ii) each problem necessitates a…”
InfluentialShinwari, H. U. K., & Usama, M. (2025) · arXiv · doi:10.48550/arxiv.2506.18267 · Cited by 3
“To assess the efficacy of ARD-LoRA, we conduct comprehensive experiments evaluating: 1) performance and parameter efficiency on language tasks; 2) adaptation dynamics in vision-language models; and 3) computational and memory overhead.”
ReTA: Recursively Thinking Ahead to Improve the Strategic Reasoning of Large Language Models
Duan, J., Wang, S., Diffenderfer, J., Sun, L., Chen, T., Kailkhura, B., & Xu, K. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.123 · Cited by 7
Faithful Self-Refinement in Mathematical Reasoning via Progressive Back-Translation
Liao, H., Zhu, Z., Hu, S., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890054 · Cited by 3
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
Selective fine-tuning for large language models via matrix nuclear norm
Xia, T., Li, Y., Wu, Y., & Chang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104259 · Cited by 3
Evaluating LLMs on Kazakhstan's mathematics exam for university admission
Kadyrov, S., Abdrasilov, B. S., Sabyrov, A., Baizhanov, N., Makhmutova, A., & Kyllonen, P. C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1642570 · Cited by 3
Meadows, J., & Freitas, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00594 · Cited by 30
Dynamic Voting for Efficient Reasoning in Large Language Models
Xue, M., Liu, D., Lei, W., Ren, X., Yang, B., Xie, J., Zhang, Y., Peng, D., & Lv, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.203 · Cited by 10
It Ain’t Over: A Multi-aspect Diverse Math Word Problem Dataset
Kim, J., Kim, Y., Baek, I., Bak, J., & Lee, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.927 · Cited by 7
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
INFORM : Information eNtropy based multi-step reasoning FOR large language Models
Zhou, C., You, W., Li, J., Ye, J., Chen, K., & Zhang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.216 · Cited by 4
Achievements and Challenges in Japanese Question Answering: Insights from Quiz Competition Results
Ariyama, T., Suzuki, J., Suzuki, M., Tanaka, R., Akama, R., & Nishida, K. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.47 · Cited by 4
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
Li, Y., Wang, H., & Zhang, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.4 · Cited by 11
Joaquin, A. S., Wang, B., Liu, Z., Müller, P., Asher, N. M., Lim, B. Y., & Chen, N. F. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.604 · Cited by 9
Enhance Large Language Models for Scientific Paper Question Answering
Chen, Z., Jiang, Y., Yang, Y. T., Jiang, X., & Zhang, Q. (2024) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml62565.2024.10779743 · Cited by 3
Knowledge Graph as Pre-Training Corpus for Structural Reasoning via Multi-Hop Linearization
Kim, W. K., Jung, H., & Kim, W. (2024) · IEEE Access · doi:10.1109/access.2024.3523579 · Cited by 3
Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter
Chen, J., Wang, B., Jiang, Z., & Nakashima, Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34527 · Cited by 3
Transformer-Based Approach for Solving Mathematical Problems Using Automatic Speech Recognition
Grgurević, A., & Babac, M. B. (2025) · IEEE Access · doi:10.1109/access.2025.3564121 · Cited by 5
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3
Overcoming classic challenges for artificial neural networks by providing incentives and practice
Irie, K., & Lake, B. M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01121-8 · Cited by 10
DUnE: Dataset for Unified Editing
Akyürek, A. F., Pan, E., Kuwanto, G., & Wijaya, D. T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.114 · Cited by 14
Tian, Q., Zhu, H., Wang, L., Li, Y., & Lan, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.114 · Cited by 4
Robustness of Named-Entity Replacements for In-Context Learning
Goodarzi, S., Kagita, N., Minn, D., Wang, S., Dessì, R., Toshniwal, S., Williams, A., Lanchantin, J., & Sinha, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.728 · Cited by 5
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way
Lv, K., Zhang, S., Gu, T., Xing, S., Hong, J., Chen, K., Liu, X., Yang, Y., Guo, H., Liu, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.48 · Cited by 4
Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
Dutta, S., Pandey, I., Singh, J., Manchanda, S., Chakrabarti, S., & Chakraborty, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29750 · Cited by 4
TAPO: Task-Referenced Adaptation for Prompt Optimization
Luo, W., Wang, W.-P., Li, X., Zhou, W., Jia, P., & Zhao, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888677 · Cited by 3
Zhu, Y., Huang, Z., Dou, Z., & Wen, J.-R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34813 · Cited by 13
Learning Multi-Step Reasoning by Solving Arithmetic Tasks
Wang, T., & Lü, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.106 · Cited by 4
LLMs can Design Sustainable Concrete – a Systematic BenchmarkPreprint
Völker, C., Rug, T., Jablonka, K. M., & Kruschwitz, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3913272/v1 · Cited by 5
Gaps: Geometry-Aware Problem SolverPreprint
Zhang, J., Jiang, Y., & Moshfeghi, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4875118 · Cited by 4
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.21187 · Cited by 3
Hendrycks, D., Song, D., Szegedy, C., Lee, H., Gal, Y., Brynjolfsson, E., Li, S., Zou, A., Levine, L., Han, B., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.18212 · Cited by 3
Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment
Chang, E. Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.427 · Cited by 2
“To validate that the Scaling Paradox observed at L3 is not an artifact of the causal domain alone, we conducted a supporting stress test using CAP-GSM8K, a variant of the GSM8K math benchmark (Cobbe et al., 2021) augmented with adversarial pressure prompts analogous to our L2 self-doubt protocol.”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 517
“…LLMs, also referred to as reasoning-capable models or Large Reasoning Models (LRMs) [98] such as OpenAI o1 [65] and DeepSeek-R1 [33], has significantly improved performance in System-2 reasoning domains [47], particularly in challenging mathematics [18,37] and programming tasks [7,19].”
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2
“Mathematical reasoning datasets like GSM8K and MATH (Cobbe et al., 2021; Hendrycks et al., 2021) assess models’ abilities to solve grade-school and competition-level math problems, while Shi et al. (2022) explore multilingual chain-of-thought reasoning.”
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
“…9 datasets spanning knowledge (MMLU (Hendrycks et al., 2021), MMLU-pro (Wang et al., 2024e), Hellaswag (Zellers et al., 2019)), reasoning (GSM8k (Cobbe et al., 2021), Knowledge Crosswords (Ding et al., 2024), NLGraph (Wang et al., 2024a; Zhang et al., 2024a)), and safety (TruthfulQA (Lin et…”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · arXiv · Cited by 2
“…out-of-domain evaluation on four benchmarks in the paper, including LongBench (Bai et al., 2023), Zero-SCROLLS (Shaham et al., 2023), GSM8K (Cobbe et al., 2021), and Big Bench Hard (BBH) (bench authors, 2023), which cover multiple tasks from document QA to math problems and in-context…”
MethodInfluentialThink before you speak: Training Language Models With Pause Tokens
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., & Nagarajan, V. (2023) · arXiv · doi:10.48550/arxiv.2310.02226 · Cited by 2
“We consider nine varied downstream tasks: (a) reasoning (GSM8k (Cobbe et al., 2021)), (b) ex-tractive question answering (SQuAD (Rajpurkar et al., 2016), CoQA (Reddy et al., 2019)), (c) general understanding (CommonSenseQA (Talmor et al., 2019), PhysicalIQA (Bisk et al., 2020)), (d) (Berant et al.,…”
BackgroundQuantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2
“In Tables 3 and 4, we report various metrics on the discrete performance measure for GSM8k, Hellaswag, and ARC-C.”
InfluentialDemystifying Multi-Agent Debate: The Role of Confidence and Diversity
Zhu, X., Zhang, C., Chi, Y., Stafford, T., Collier, N., & Vlachos, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1694 · Cited by 28
Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback
Potamitis, N., & Arora, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12951 · Cited by 2
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0005 · Cited by 2
Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 5
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 3
CoT-Tree: An Efficient Prompting Strategy for Large Language Model Classification Reasoning
Wang, Y., Hu, J., & Luo, Z. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825547 · Cited by 2
Step-by-Step Correction of LLM-based Math Word Problems Solutions
Li, Y., Ubaidali, D. M., Wang, L., & Zhang, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889273 · Cited by 3
Beyond KV caching: Shared attention for efficient LLMs
Liao, B., & Vargas, D. V. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130587 · Cited by 2
Systematic survey of various prompt optimization methods and their classifications
Jain, A. M., & Jindal, M. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00085 · Cited by 3
Fajčík, M., Dočekal, M., Doležal, J., Ondrej, K., Beneš, K., Kapsa, J., Smrž, P., Polok, A., Hradiš, M., Nevěřilová, Z., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.32 · Cited by 14
GrapheneChat: A Large Language Model for Enhancing Graphene Research
Yang, D., Lu, C., Feng, Y., Mao, W., Wang, K., Sun, L., Wang, Y., Hu, B., Tan, J., Liu, Z., et al. (2026) · ACS Nano · doi:10.1021/acsnano.5c21335 · Cited by 2
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. D., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 19
Scalable and Explainable Automated Scoring for Open-Ended Constructed Response Math Word Problems
Hellman, S., Andrade, A., & Habermehl, K. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.12 · Cited by 3
Causal Abstraction for Chain-of-Thought Reasoning in Arithmetic Word Problems
Tan, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.12 · Cited by 7
DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
Han, C., Du, X., Che, Z., Lian, Y., Li, X., Gao, M. Y., & Wang, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.501 · Cited by 6
Gómez, F. J. D., & Galende, J. N. (2024) · Revista Tecnología Ciencia y Educación · doi:10.51302/tce.2024.19081 · Cited by 5
Gupta, P., Jain, N., & Bhat, A. (2024) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic60222.2024.10575336 · Cited by 2
Metadata
- Authors
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J.
- ORCID
- Kosaraju, Vineet
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.14168
- OpenAlex
- W3210904070
- Cited by
- 575
- References
- 28
- Citations by year
- 2026: 30 · 2025: 137 · 2024: 156 · 2023: 216 · 2022: 34 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Education › Mathematics Education and Pedagogy
- Keywords
- mathematical word problems, large language models, solution verification, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
