The DOI matches its DataCite record; 29 of the 29 references it cites were found in a registry or an index.
Training Verifiers to Solve Math Word Problems
Open accessAbstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line of research utilizes majority voting to choose candidates based on their execution performance [89, 90].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [90]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Cobbe et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [14]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within tasks, feedback refines solutions or guides actions [1, 2].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GSM8K (Grade School Math 8K) is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Cobbe et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Each problem is designed to be solvable using only basic arithmetic operations (+, -, ×, ÷) with a difficulty level appropriate for grade school students.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Knowledge-oriented benchmarks (e.g., MMLU [325] and C-Eval [335]) assess models' world knowledge retention, while reasoning-oriented benchmarks (e.g., GSM8K [318], BBH [336], and MATH [317]) evaluate their ability to solve complex reasoning tasks.”
From A Survey of Large Language Models (2026) · cites this paper as [318]
The cited source says what the claim attributes to it.
In the citing paper
“Best-ofN (Cobbe et al., 2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential.”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Notably, ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 → 31.1% on GSM8K (Maj@3) (Cobbe et al., 2021) with Llama3 1B (Dubey et al., 2024)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM8K (Cobbe et al., 2021) provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many benchmarks like MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al., 2021) and GSM-Hard (Gao et al., 2023) math tasks, respectively, after transferring across domains.”
From Automated Design of Agentic Systems (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Each GSM8K is paired with a numerical reference answer.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Given these criteria, the mathematical reasoning benchmark datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) foot_0 emerge as fitting choices for our test bed, which allow us to study data leakage in depth but also offer a relevant and challenging environment.”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“(i) Math Word Problems test reasoning on math problems similar to GSM8k (Cobbe et al., 2021) but with varying levels of relational complexity, particularly in their degree of variable interdependence, which we define as the number of other variables on which a single variable's value depends.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This category (21 problems) is inspired by the grade-school math problems in datasets such as GSM8k (Cobbe et al., 2021), but introduces an additional dimension of variable interdependence (i.e., higher relational complexity between the variables).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“GSM1k is modelled after the GSM8k dataset (Cobbe et al. [2021]), released by OpenAI in 2021, which consists of 8.5k grade school math problems.”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Cobbe et al. [2021]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching. arXiv. (2025) · cites this paper as [18]
Experimental — accuracy not yet measuredIn the citing paper
“Multi-step reasoning in LLMs typically involves executing sequential logical operations-typically requiring 3-10 intermediate steps-to arrive at solutions that cannot be derived through single-step inference (Cobbe et al., 2021; Yang et al., 2018; Suzgun et al., 2022).”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Cobbe et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Specifically, we apply residual dropout (Vaswani et al., 2017) along the residual paths of each layer in the network.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
BackgroundSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”
BackgroundSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 822 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Deep Learning for Symbolic MathematicsPreprint
Lample, G., & Charton, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01412 · Cited by 405 · cited 1×
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
Background
Cited by
Sorted by citations
Items 601–700 of 791
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
Bui, N., Nguyen, H. T., Cohan, A., & Ying, R. (2026) · arXiv · doi:10.48550/arxiv.2605.09649
“For a pure text reasoning, we evaluate on AIME24 [Art of Problem Solving, 2024], GSM8K [Cobbe et al., 2021], and MATH-10 500 [Hendrycks et al., 2021], similar to [Bui et al., 2025].”
PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Woo, S., Kim, H., Shim, S., Jo, M., Jeong, H., Lee, J., Kim, J., Lee, S., Ryu, J., Park, B., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12029
“To examine whether the serving behavior observed in Section 4.3 depends on a specific backbone model, we repeat the same experiments using Qwen3-14B in place of LLaMA3.”
InfluentialA Unified Approach to Routing and Cascading for LLMs
Dekoninck, J., Baader, M., & Vechev, M. (2024) · arXiv · doi:10.48550/arxiv.2410.10347
“For open-form reasoning tasks, we use MMLU-Pro and GSM8k (Cobbe et al., 2021).”
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“This claim is evaluated using three proxies for language performance: HellaSwag [106], GSM8K [12], and (1 − bitsperbyte ) .”
InfluentialTesting Frontier Large Language Models' Physics Literacy in Parallel Physical Worlds
Zhang, D. (2026) · arXiv · doi:10.48550/arxiv.2607.00276
“Wei et al. (2022b) introduced Chain-of-Thought (CoT) prompting in 2022: supplying a few “question, intermediate steps, answer” examples in the prompt raised PaLM-540B’s accuracy on GSM8K (Cobbe et al., 2021) from 17.9% to 56.9%.”
InfluentialReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Lee, H., Oh, S., Kim, J., Shin, J., & Tack, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14565 · Cited by 26
“We evaluate ReVISE and baselines on GSM8K (Cobbe et al., 2021) and MATH-500 (Hendrycks et al., 2021), a widely used evaluation benchmark subset of MATH.”
InfluentialMulti-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Motger, Q., Oriol, M., Marco, J., & Franch, X. (2026) · arXiv · doi:10.48550/arxiv.2607.26212 · Cited by 5
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Ferreira, L., Liu, G., & Zheng, K. (2026) · arXiv · doi:10.48550/arxiv.2609.35875
“3 pp GSM8K, +6 .”
InfluentialFrom Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
Wang, H., Liang, W., Fu, Z., Zheng, N., Zhang, Y., Tong, Y., Zhu, T., Jiang, H., Li, C., Wu, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.23196
“…referred to as Few-shot CoT, has become the de facto standard for enhancing reasoning during evaluation across numerous benchmarks, from mathematical problem-solving to commonsense reasoning (Cobbe et al., 2021; Hendrycks et al., 2021; Con-tributors, 2023; Huang et al., 2023; Wang et al., 2024).”
Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025) · arXiv · doi:10.48550/arxiv.2503.16402
“Moreover, some existing benchmark datasets, such as MMLU (Hendrycks et al., 2020) and GSM8K (Cobbe et al., 2021), are already of high quality and accurately reflect real-world question distributions within their respective domains.”
“This could hinder adaptation to similar queries with minor differences ( e.g., numeric values or variable names in mathematical reasoning [18], coding tasks [26]), a common challenge in agentic AI.”
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04510 · Cited by 18
“…against baselines on 12 datasets spanning 4 categories: 1) knowledge: MMLU-pro [94], Knowledge Crosswords [19], and COM2 [23]; 2) reasoning: GSM8k [13], NLGraph [89], and Normad [76]; 3) agent: GAIA-text [67], the knowledge graph and lateral thinking puzzle subtasks of AgentBench [61]; 4)…”
Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Ma, X., Yeoh, W., Zhang, N., & Vorobeychik, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.15143
“We split the original GSM8K training set into train and validation subsets using a 0 .”
InfluentialThe Limits of Query Monitoring against Distributed Model Extraction
Khilar, S. C. (2026) · arXiv
“Capability is accuracy on GSM8K test problems (Cobbe et al., 2021).”
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Desai, R., Hu, J., Cabezas, J., Harsola, N., Shukla, P., Chaim, R. B., Assadi, A. E., Kamath, O. M., Faldu, F., Hebbar, P., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.07682
“Large language models have progressed rapidly from grade-school math [19] to competitive programming, patch generation [36], and multi-domain agentic tasks spanning terminal use [44], freelance software engineering [47], and library-scale generation [84].”
Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents
Zhang, K., Xiong, Z., Zhong, M., Jiang, Z., Yuan, Z., Li, Z., & Lin, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.00136
“Datasets Using the method in Section 2.1, we augment GSM8K (Cobbe et al., 2021) and HotpotQA (Yang et al., 2018) with GPT-4o-mini-generated distractors, yielding GSM8K-Sem-Distractor and HotpotQA-Sem-Distractor.”
InfluentialStop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Zhang, H., Cui, Z., Chen, J., Wang, X., Zhang, Q., Wang, Z., Wu, D., & Hu, S. (2025) · arXiv · doi:10.48550/arxiv.2502.08788 · Cited by 84
OmniRouter: Budget and Performance Controllable Multi-LLM Routing
Mei, K., Xu, W., Guo, M., Lin, S., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.20576 · Cited by 2
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Zhang, H., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.226 · Cited by 3
Gao, X., Xie, W., Lixin, Fang, X., Hang, C., Li, C., Lin, Y., & Xu, X. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i25.39266
Li, H., Feng, L., & Wang, D. (2026) · Electronics · doi:10.3390/electronics15102011
Critic-Guided Heterogeneous Multi-Agent Reasoning for Reliable Mathematical Problem Solving
Sharif, M. T., & Rehman, A. (2026) · arXiv · doi:10.48550/arxiv.2606.05704
ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Zhong, Z., Raghunathan, A., & Carlini, N. (2025) · arXiv · doi:10.48550/arxiv.2510.20270 · Cited by 54
Shen, X., Liu, Y., Dai, Y., Wang, Y., Miao, R., Tan, Y., Pan, S., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2505.23352 · Cited by 6
Tsui, K. (2025) · arXiv · doi:10.48550/arxiv.2507.02778 · Cited by 29
CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization
Shah, D., Badhe, S., Kathrotia, N., & Tiwari, P. (2026) · arXiv
Locality-aware Fair Scheduling in LLM Serving
Cao, S., Wang, Y., Mao, Z., Hsu, P.-L., Yin, L., Xia, T., Li, D., Liu, S., Zhang, Y., Zhou, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.14312
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv · Cited by 1
Adaptive hybrid speculative decoding for accelerating large language model inference
Yong, Y., Yang, T. T., Gao, S. S., & Yin, Y. H. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133760
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
Multi-agent Architecture Search via Agentic Supernet
Zhang, G., Niu, L., Fang, J., Wang, K., Bai, L., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2502.04180
MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
Ke, Z., Xu, A., Ming, Y., Nguyen, X.-P., Chin, R., Xiong, C., & Joty, S. (2025) · arXiv · doi:10.48550/arxiv.2505.14996
“This limitation is less apparent on simple tasks such as GSM8K and HumanEval, which are commonly used for MAS evaluation [17, 55, 58] and where most methods already perform well.”
EvoFlow: Evolving Diverse Agentic Workflows On The Fly
Zhang, G., Chen, K., Wan, G., Chang, H., Cheng, H., Wang, K., Hu, S., & Bai, L. (2025) · arXiv · doi:10.48550/arxiv.2502.07373
“…on six public benchmarks covering four domains: (1) math reasoning , GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), and MultiArith (Roy & Roth, 2016); (2) code generation , HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021)); (3) embodied , ALFWorld (Shridhar et al., 2021).”
ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
Wang, Y., Yang, L., Li, G., Wang, M., & Aragam, B. (2025) · arXiv · doi:10.48550/arxiv.2502.04306
Looped Transformers for Length Generalization
Fan, Y., Du, Y., Ramchandran, K., & Lee, K. (2024) · arXiv · doi:10.48550/arxiv.2409.15647
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
Lab, N., :, Yang, C., Huang, C., Lan, F., Chen, H., Zhou, H., Song, H., Cao, J., Zhu, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.22083
“We select a diverse set of benchmarks, including reasoning-oriented metrics (GSM8K [6], BBH [24], and MBPP [1]) and knowledge-oriented metrics (MMLU-Pro [27], SuperGPQA [9], and GPQA [21]). with a multi-stage RL recipe designed to improve response quality, reasoning efficiency, and long-horizon…”
Paech, S., Roush, A., Goldfeder, J., & Shwartz-Ziv, R. (2025) · arXiv · doi:10.48550/arxiv.2510.15061
“[Cobbe et al., 2021].”
Decentralized Multi-Agent Systems with Shared Context
Mao, Y., Gu, J., Chauhan, A., Zhang, Q., Kang, H., & Mirhoseini, A. (2026) · arXiv · doi:10.48550/arxiv.2606.10662
“If a sub-agent receives insufficient context, it return control to the main agent, triggering additional retrieval or another delegation round.”
RoPoLL: Robust Panel of LLM Judges
Acharya, A., Pan, K. W., & Verkhovsky, B. (2026) · arXiv · doi:10.48550/arxiv.2606.30931
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness; the analogue of Proposition 2 for vote-based aggregation on { 0 , 1 } outputs is the standard α < 1 / 2 Byzantine threshold.”
EvolKV: Evolutionary KV Cache Compression for LLM Inference
Yu, B., & Chai, Y. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.88
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0008
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0003
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0007
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0002
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0004
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.002.0003
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.002.0004
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.002.0006
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0005
Summerfield, C. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780192843883.003.0001
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0009
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0001
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0004
Mapping the Ethics of Algorithms
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0007
How to Deliver a Good AI Society
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0010
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0007
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0008
List of Most Common Abbreviations and Acronyms
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0011
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0011
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0010
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0009
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0005
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0013
Research on "MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting"
Inaba, T. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.1100
Comparing Power Models for circuits design with Mathematical Language Processing
Desormiere, A., Gzara, L., Bigeon, J., & Nguyen-Thê, L. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.05.097
Text VQA with Token Classification of Recognized Text and Rule-Based Numerical Reasoning
Сурков, В. О., & Evseev, D. A. (2023) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2023-22-486-496
Srivastava, N., & Kim, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790013
Zdebskyi, P., & Berko, A. (2024) · Scientific Works of Vinnytsia National Technical University · doi:10.31649/2307-5376-2024-1-1-6 · Cited by 1
CRSP: Emulating Human Cooperative Reasoning for Intelligible Story Point Estimation
Han, R., Han, W., Han, Z., Tian, Y., Chen, L., & Han, R. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644417
AI literacy performance in 2016 and 2021, counting Maybe as 50%-Yes
(2023) · Educational research and innovation · doi:10.1787/088b753a-en
(2023) · Educational research and innovation · doi:10.1787/36fe3671-en
AI numeracy performance by questions and difficulty levels
(2023) · Educational research and innovation · doi:10.1787/07646886-en
AI literacy performance using questions with high certainty
(2023) · Educational research and innovation · doi:10.1787/772b64b3-en
Proportion of workers with a well-balanced skill set
(2023) · Educational research and innovation · doi:10.1787/04cafd18-en
AI numeracy performance by expert group
(2023) · Educational research and innovation · doi:10.1787/247d9b1b-en
Numeracy proficiency levels of the working population, ALL and PIAAC
(2023) · Educational research and innovation · doi:10.1787/7c1ddcdc-en
Average expert ratings in literacy in 2016 and 2021
(2023) · Educational research and innovation · doi:10.1787/90fb6678-en
Numeracy performance of AI and adults of different proficiency
(2023) · Educational research and innovation · doi:10.1787/cd40ce20-en
OECD (2023) · Educational research and innovation · doi:10.1787/8dc29260-it
OECD (2023) · Educational research and innovation · doi:10.1787/85f9c2d2-de
AI numeracy performance in 2016 and 2021 according to experts who participated in both assessments
(2023) · Educational research and innovation · doi:10.1787/d1ca7693-en
Daily use of literacy and numeracy practices at work
(2023) · Educational research and innovation · doi:10.1787/5a20875f-en
Daily use of literacy and numeracy at work together with other skills
(2023) · Educational research and innovation · doi:10.1787/3195a267-en
OECD (2023) · Educational research and innovation · doi:10.1787/4d420b4c-en
Literacy performance of AI and average adults
(2023) · Educational research and innovation · doi:10.1787/d53da908-en
AI numeracy performance by expert
(2023) · Educational research and innovation · doi:10.1787/c71dfab7-en
AI literacy performance according to different computation methods
(2023) · Educational research and innovation · doi:10.1787/10b05e2c-en
Methodology for assessing AI capabilities using the Survey of Adult Skills (PIAAC)
OECD (2023) · Educational research and innovation · doi:10.1787/b70b1373-en
AI literacy performance by questions and difficulty levels
(2023) · Educational research and innovation · doi:10.1787/bbb45887-en
Success rate of AI in PIAAC according to experts' assessments
(2023) · Educational research and innovation · doi:10.1787/f2850ba5-en
Share of literacy questions that receive three or more uncertain ratings in 2016 and 2021
(2023) · Educational research and innovation · doi:10.1787/6aedb2ca-en
AI numeracy performance using questions with high certainty
(2023) · Educational research and innovation · doi:10.1787/cd2e6456-en
AI literacy performance in 2016 and 2021, by question difficulty
(2023) · Educational research and innovation · doi:10.1787/a151e67b-en
Numeracy performance of AI and average adults
(2023) · Educational research and innovation · doi:10.1787/08077ac4-en
Metadata
- Authors
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J.
- ORCID
- Kosaraju, Vineet
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.14168
- OpenAlex
- W3210904070
- Cited by
- 575
- References
- 28
- Citations by year
- 2026: 30 · 2025: 137 · 2024: 156 · 2023: 216 · 2022: 34 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Education › Mathematics Education and Pedagogy
- Keywords
- mathematical word problems, large language models, solution verification, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
