The DOI matches its DataCite record; 29 of the 29 references it cites were found in a registry or an index.
Training Verifiers to Solve Math Word Problems
Open accessAbstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line of research utilizes majority voting to choose candidates based on their execution performance [89, 90].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [90]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Cobbe et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [14]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within tasks, feedback refines solutions or guides actions [1, 2].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GSM8K (Grade School Math 8K) is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Each problem is designed to be solvable using only basic arithmetic operations (+, -, ×, ÷) with a difficulty level appropriate for grade school students.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Knowledge-oriented benchmarks (e.g., MMLU [325] and C-Eval [335]) assess models' world knowledge retention, while reasoning-oriented benchmarks (e.g., GSM8K [318], BBH [336], and MATH [317]) evaluate their ability to solve complex reasoning tasks.”
From A Survey of Large Language Models (2026) · cites this paper as [318]
The cited source says what the claim attributes to it.
In the citing paper
“Best-ofN (Cobbe et al., 2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential.”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Notably, ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 → 31.1% on GSM8K (Maj@3) (Cobbe et al., 2021) with Llama3 1B (Dubey et al., 2024)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM8K (Cobbe et al., 2021) provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many benchmarks like MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al., 2021) and GSM-Hard (Gao et al., 2023) math tasks, respectively, after transferring across domains.”
From Automated Design of Agentic Systems (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Each GSM8K is paired with a numerical reference answer.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Given these criteria, the mathematical reasoning benchmark datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) foot_0 emerge as fitting choices for our test bed, which allow us to study data leakage in depth but also offer a relevant and challenging environment.”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“(i) Math Word Problems test reasoning on math problems similar to GSM8k (Cobbe et al., 2021) but with varying levels of relational complexity, particularly in their degree of variable interdependence, which we define as the number of other variables on which a single variable's value depends.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This category (21 problems) is inspired by the grade-school math problems in datasets such as GSM8k (Cobbe et al., 2021), but introduces an additional dimension of variable interdependence (i.e., higher relational complexity between the variables).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM1k is modelled after the GSM8k dataset (Cobbe et al. [2021]), released by OpenAI in 2021, which consists of 8.5k grade school math problems.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [18]
Experimental — accuracy not yet measuredIn the citing paper
“Multi-step reasoning in LLMs typically involves executing sequential logical operations-typically requiring 3-10 intermediate steps-to arrive at solutions that cannot be derived through single-step inference (Cobbe et al., 2021; Yang et al., 2018; Suzgun et al., 2022).”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Cobbe et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Specifically, we apply residual dropout (Vaswani et al., 2017) along the residual paths of each layer in the network.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
BackgroundSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”
BackgroundSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 822 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Deep Learning for Symbolic MathematicsPreprint
Lample, G., & Charton, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01412 · Cited by 405 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
Background
Cited by
Sorted by citations
Items 701–791 of 791
Long-term trends in average reading proficiency of 15-year-olds
(2023) · Educational research and innovation · doi:10.1787/01ebf592-en
Average majority size in rating literacy questions in 2016 and 2021
(2023) · Educational research and innovation · doi:10.1787/1a0db47c-en
Share of numeracy questions that receive three or more uncertain ratings in 2016 and 2021
(2023) · Educational research and innovation · doi:10.1787/c9c87fc3-en
Changes in AI capabilities in literacy and numeracy between 2016 and 2021
OECD (2023) · Educational research and innovation · doi:10.1787/4301a378-en
Setting the stage: Approaches to assessing AI’s impact
OECD (2023) · Educational research and innovation · doi:10.1787/8ee0810f-en
AI literacy performance in 2016 and 2021 according to experts who participated in both assessments
(2023) · Educational research and innovation · doi:10.1787/75067cb8-en
Literacy performance of AI and adults of different proficiency
(2023) · Educational research and innovation · doi:10.1787/329dd8f7-en
AI literacy performance by expert
(2023) · Educational research and innovation · doi:10.1787/525be76d-en
AI numeracy performance according to different computation methods
(2023) · Educational research and innovation · doi:10.1787/14491e90-en
Long-term trends in average mathematics proficiency of 15-year-olds
(2023) · Educational research and innovation · doi:10.1787/6fb38463-en
AI numeracy performance in 2016 and 2021, by question difficulty
(2023) · Educational research and innovation · doi:10.1787/f1b338d4-en
Implications of evolving AI capabilities for employment and education
OECD (2023) · Educational research and innovation · doi:10.1787/ed919621-en · Cited by 1
Evolution of human skills versus AI capabilities
OECD (2023) · Educational research and innovation · doi:10.1787/d077ad2f-en
Experts’ assessments of AI capabilities in literacy and numeracy
OECD (2023) · Educational research and innovation · doi:10.1787/134fa8aa-en
Advancing low resource information extraction and dialogue system using data efficient methods
Ding, B. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/179560
MILE: Memory-Interactive Learning Engine for Neuro-Symbolic Solutions to Mathematical Problems
Wu, Y., & Nakayama, H. (2024) · IEEE Access · doi:10.1109/access.2024.3461150 · Cited by 1
Overview of the NLPCC 2024 Shared Task 10: Regulating Large Language Models
Wang, C., Xu, Z., Wang, M., Chen, X., Deng, S., & Zhang, N. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_22
Detoxifying Large Language Models via Kahneman-Tversky Optimization
Li, Q., Du, W., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_36 · Cited by 1
Mathematical Reasoning via Multi-step Self Questioning and Answering for Small Language Models
Chen, K., Wang, J., & Zhang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9440-9_7
Causal Reasoning of Stored Grain Pest Events Enhanced by Large Language Models
Xiao, L., Zhao, J., & Xu, Y. (2024) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase63199.2024.10762344
StreamAdapter: Efficient Test Time Adaptation from Contextual StreamsPreprint
Muhtar, D., Shen, Y., Yang, Y., Liu, X., Lu, Y., Liu, J., Zhan, Y., Sun, H., Deng, W., Sun, F., et al. (2024) · Qeios · doi:10.32388/g5h95j · Cited by 10
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
Bast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134
Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language ModelsPreprint
Gao, K., Xia, S., Xu, K., Torr, P. H. S., & Gu, J. (2024) · Qeios · doi:10.32388/758n37
Research on Development of LLMs and Manual Comparison of Applications
Liu, X., Li, J., Bai, T., Gao, J., Zhang, P., & Zhao, X. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10808821
SMOOTHIE: Label Free Language Model RoutingPreprint
Guha, N., Chen, M. F., Chow, T., Khare, I. S., & Ré, C. (2025) · Qeios · doi:10.32388/f319w4
Self-citationFine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and ChallengesPreprint
Hisham, M. (2025) · Qeios · doi:10.32388/z95x8o
TopoRefine: Iterative Refinement with Reasoning Topology as High-Level Feedback
Liao, H., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10888346
GPT-C: Generative PrompT Compression
Liu, L., Wang, R., Jing, L., Lv, F., & Zhu, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888801
Beyond Training Data: Zero-Shot Mathematical Problem Solving with Large Language Models
Parmar, S. H., & Goriya, M. (2024) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei64305.2024.10912380
Exploring performance and consistency in mathematical reasoning across problem types
Hwang, Y., Lim, J., Lee, Y., & Choi, H. (2025) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp64353.2025.00031
Researcher, S. V. C.-O. A., & Arslan, A. K. (2025) · Engineering and Technology Journal · doi:10.47191/etj/v10i03.31
Wu, O. T., Chan, F. K. S., & Law, Y. N. (2024) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait64506.2024.10963264
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4
Mitigating Hallucinations in LLMs through Context-Aware Decoding and Layer Comparison
Yu, S., Kim, G., & Kang, S. (2025) · Journal of Digital Contents Society · doi:10.9728/dcs.2025.26.4.995
Wang, Z., Zhang, L., Zheng, F., Ma, F., & Shan, L. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194026500658
Neural Network Methods for Selecting and Generating Synthetic Variations of Combinatorial Problems
Nikolaiev, A., & Anisimov, A. (2025) · Cybernetics and Systems Analysis · doi:10.1007/s10559-025-00774-y
Large Language Model Training and Reinforcement LearningPreprint
Alonso, M. N. I. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5194064
Benchmarking Large Language Models on Network Optimization
Cui, E. (2025) · Operations Research and Applications An International Journal · doi:10.5121/oraj.2025.12201
Advancing Large Language Models in Code Generation: Usaco Benchmark and Bug Mitigation Insights
Trentini, J., Liu, V., Peng, Y., & Zong, Z. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00057 · Cited by 1
A Meta-Analysis of Cloud-Based Llms in Implicit Multi-Step Reasoning
You, J., Gao, Y., Xu, Z., & Zhao, L. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035021
Su, F.-Y., Ngo, G.-H., Phan, B., & Chiang, J.-H. (2025) · Database · doi:10.1093/database/baaf025 · Cited by 1
ADAPTATION OF TEXT GENERATION STYLE TO A SPECIFIC AUDIENCE OR CONTENT
Zhangbyrbay, Z., Akhmetov, I., Pak, A., Jaxylykova, A., & Komada, P. (2025) · Herald of Kazakh-British technical university · doi:10.55452/1998-6688-2025-22-2-141-154
Automated Construction of High-quality Evaluation Datasets Based on LLMs
Kang, L., Han, R., Wang, M., Yang, Z., Gao, N., & Zhang, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_31
Logical Reasoning Capabilities of Large Language Models: A Comparative Evaluation on GPQA Dataset
Wan, B. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.bj25665
Singh, A., Srivastav, J., Banerjee, J., Jindal, S., & Mohapatra, A. K. (2025) · International Conference on Networks and Cryptology (NETCRYPT) · doi:10.1109/netcrypt65877.2025.11102417
"Give a Positive Review Only": Can Hidden PDF Prompts Sway LLMs?Preprint
Guo, X., Huang, Z., Song, Z., & Zhang, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5368453
Evaluating Human-Like Qualities in Language Models
Liu, R. (2025) · Transactions on Computer Science and Intelligent Systems Research · doi:10.62051/yygprz73
Tahir, H., Faisal, F., Alnajjar, F., Taj, M. I., Gordon, L., Khan, A. M., Lwin, M., & Mubin, O. (2025) · International Conference on Activity and Behavior Computing · doi:10.1109/abc64332.2025.11118419 · Cited by 1
Loureiro, A. D. S., Valverde-Rebaza, J., Noguez, J., Escarcega, D., & Marcacini, R. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_12
TALE: Token-Adaptive Low-Rank KVCache Approximation with Reconstruction Elimination
Lee, J., Hwang, S.-W., Qiao, A., Campos, D., Yao, Z., & He, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.39
Unified Cognitive Consciousness Theory: Formalization, Validation, and Outlook
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749438
Understanding ensemble-based component architectures by LLMs
Töpfer, M., Bureš, T., Hnětynka, P., & Plášil, F. (2025) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-025-00835-9
Conformal Prediction in the Age of Multimodal Foundation Models: A Survey
Stavan, C., Tilaganji, K., Krishnan, S. M., Kancheti, S. S., & Balasubramanian, V. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15120-9_13
CareEval: Evaluating Large Language Models for Decision-Making in Physical Robot Caregiving
Liu, Z., Dimitropoulou, K. A., Cheung, C., & Bhattacharjee, T. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794354
Zhang, C., Shen, L., & Li, Y. (2026) · Complex & Intelligent Systems · doi:10.1007/s40747-026-02290-8
StigmergyRouter: A Fault-Aware Adaptive Routing Demo for Multi-Agent AI Systems
Du, J., Hang, Z., & Huang, K. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813191
The meek shall inherit the Earth: personality, empathy, and the human shaping of LLMs
Mangalam, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01215-y
Cuclea, L.-N., Badea, S.-C., & Dumitran, A.-M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_5
Matsuoka, D., Kawahara, S., Murakami, K., Matsumoto, R., Ito, R., Sugimoto, S., Sugiyama, D., Hara, M., Hayashida, M., Nguyen, K., et al. (2026) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2025jh001205
Assessing Multimodal Large Language Models Using High-School Mathematics Exams with Visual Elements
Tsang, H. P. Y., & Yu, P. L. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29178-3_13
Internal Reasoning and External Control
Chang, E. Y. (2026) · ACM eBooks · doi:10.1145/3822380.3822385
Trivium: Temporal Regret as a First-Class Objective for Causal-Memory Controllers
Chang, E. Y. (2026) · ACM eBooks · doi:10.1145/3822380.3822391
Chang, E. Y. (2026) · ACM eBooks · doi:10.1145/3822380.3822386
iBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1
Enhancing Mathematical Reasoning Education with Data-Driven Fine-Tuning of Large Language Models
Pulikkottil, P., Liu, D., Akram, J., Nawaz, M., & Anaissi, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32636-2_18
Graph-Aware Reinforcement Learning for Reusable Prompt Compression in Black-Box LLMsPreprint
Eskandarpour, M., Haddadnejad, P., & Jannati, M. J. (2026) · Research Square · doi:10.21203/rs.3.rs-10458568/v1
STARS: extending interactive task learning with large language models
Kirk, J. R., Wray, R. E., & Laird, J. E. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1848830
Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability
Rani, M., Mishra, B. K., & Thakker, D. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10794-4 · Cited by 2
Jin, X., Du, Y., & Han, H. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08909-4
Khera, B., Alamian, R., Scherz, P. A., & Goetz, S. M. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09543-4 · Cited by 1
The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
Jin, Y., Wei, G.-Y., & Brooks, D. (2025) · arXiv · doi:10.48550/arxiv.2505.14733
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · arXiv · doi:10.48550/arxiv.2405.17893
“Dataset We used the GSM8K-Prolog described in Section 3.1.”
BackgroundMethodInfluentialThe ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning
Vahdati, S., Aioanei, A., Suresh, H., & Lehmann, J. (2026) · arXiv · doi:10.48550/arxiv.2603.13372
Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
Tie, G., Yuan, Z., Zhao, Z., Hu, C., Gu, T., Zhang, R., Zhang, S., Wu, J., Tu, X., Jin, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.16062 · Cited by 16
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
Cui, T., Zeng, P., Xu, W., Xiao, H., Xue, S., Gao, Q., Ouyang, W., Heng, P. A., Yin, Z., Wu, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.13912
“Within tasks, feedback refines solutions or guides actions [1, 2].”
Phan, L. (2025) · doi:10.21203/rs.3.rs-6615297/v1 · Cited by 65
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Ye, R., Tang, S., Ge, R., Du, Y., Yin, Z., Chen, S., & Shao, J. (2025) · arXiv · doi:10.48550/arxiv.2503.03686
Turning up the heat: Min-p sampling for creative and coherent llm outputs
Nguyen, M. N., Baker, A., Neo, C., Roush, A., Kirsch, A., & Shwartz-Ziv, R. (2024) · arXiv · doi:10.48550/arxiv.2407.01082 · Cited by 1
A General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citationSelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
Miao, N., Teh, Y. W., & Rainforth, T. (2023) · arXiv · doi:10.48550/arxiv.2308.00436 · Cited by 18
“Secondly, GSM8K only has arithmetic problems, while MathQA and MATH contain more diverse problems in geometry, physics, probability, or algebra.”
Influential
Metadata
- Authors
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J.
- ORCID
- Kosaraju, Vineet
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.14168
- OpenAlex
- W3210904070
- Cited by
- 575
- References
- 28
- Citations by year
- 2026: 30 · 2025: 137 · 2024: 156 · 2023: 216 · 2022: 34 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Education › Mathematics Education and Pedagogy
- Keywords
- mathematical word problems, large language models, solution verification, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
