Training Verifiers to Solve Math Word Problems
Open accessAbstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The diversity and complexity of tasks make it difficult to represent them in a unified yet flexible manner.
“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
From AFlow: Automating Agentic Workflow Generation (2024) · cites this paper as Cobbe et al., 2021
The Introduction’s Dataset description (page not provided) describes diverse math word problems requiring multiple steps and arithmetic operations. It does not say that differing requirements or dependencies make the tasks difficult to represent in a unified, flexible manner. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings.
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Cobbe et al., 2021
The Dataset discussion (Introduction, p. unavailable) describes GSM8K as individual grade-school math problems taking 2–8 steps, and Methods (§4, p. unavailable) describes evaluation by checking the final answer. This supports GSM8K as an example of bounded tasks with a fixed correctness criterion. The paper does not establish that the majority of evaluation protocols simplify assumptions or use static, pre-defined rules; it also says GSM8K deliberately avoids repeated problem templates. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAnother line of research utilizes majority voting to choose candidates based on their execution performance.
“Another line of research utilizes majority voting to choose candidates based on their execution performance [89, 90].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [90]
Section 5.1 (page not specified) describes majority voting among top verifier-ranked solutions, with votes determined by their final answers. It does not describe choosing candidates based on execution performance; that part of the claim is unsupported. (LLM, conf=0.85)
Experimental — accuracy not yet measuredCobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.
“Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023b) train a verifier or a critique model on a high-quality dataset to verify or refine LLM outputs, which can be used to provide feedback for correcting prediction errors.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Cobbe et al. (2021)
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable, 1 not checked. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredMany established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated, leading to discrepancies between the models' perceived and actual capabilities.
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [14]
Section 3.1 (page unavailable) says early math word-problem datasets are too small to test the limits of modern language models, supporting the narrower point that some older benchmarks have not kept pace. The paper does not say that many established benchmarks have quickly become saturated. The sentence attributes the separate claim about perceived versus actual capabilities to citation [36], not to this source. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Within tasks, feedback refines solutions or guides actions.
“Within tasks, feedback refines solutions or guides actions [1, 2].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
Section 4.2 (Verification; page unavailable) says the verifier scores candidate math solutions and the system returns the highest-ranked one. This supports using an evaluative signal to improve the solution selected for a task, but the paper does not describe feedback refining an existing solution or guiding actions as they are taken. (LLM, conf=0.82)
Experimental — accuracy not yet measuredMajority voting in mathematical reasoning is a related but coarser ensemble technique on binary correctness.
“Majority voting in mathematical reasoning (Cobbe et al., 2021) is a related but coarser ensemble technique on binary correctness”
From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Cobbe et al., 2021
Section 5.1 (page unavailable) describes majority voting among top verifier-ranked math solutions, supporting the identification of a voting technique. But the votes are for final answers, not binary correct/incorrect labels; the claim's description of what is voted on is unsupported. (LLM, conf=0.86)
Experimental — accuracy not yet measuredGSM8K is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.
“GSM8K (Grade School Math 8K) is a dataset comprising 8,792 high-quality grade school-level math word problems, including 7,473 for training and 1,319 for testing.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Cobbe et al., 2021
The Introduction’s Dataset section describes GSM8K as high-quality grade-school math problems and gives an approximate split of 7.5K training and 1K test problems. Our copy does not give the claimed exact counts of 8,792 total, 7,473 training, and 1,319 test problems. (LLM, conf=0.86)
Experimental — accuracy not yet measuredKnowledge-oriented benchmarks assess models' world knowledge retention, while reasoning-oriented benchmarks evaluate their ability to solve complex reasoning tasks.
“Knowledge-oriented benchmarks (e.g., MMLU [325] and C-Eval [335]) assess models' world knowledge retention, while reasoning-oriented benchmarks (e.g., GSM8K [318], BBH [336], and MATH [317]) evaluate their ability to solve complex reasoning tasks.”
From A Survey of Large Language Models (2026) · cites this paper as [318]
Best-of-N and majority voting may overlook a model's full reasoning potential.
“Best-ofN (Cobbe et al., 2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential.”
From Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025) · cites this paper as Cobbe et al., 2021
Compared with MultiArith and GSM8K, the Counter-Intuitive AR dataset presents two distinct challenges: resistance to intuition and multi-step reasoning.
“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges:”
From Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2024) · cites this paper as Cobbe et al., 2021
ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 to 31.1% on GSM8K (Maj@3) with Llama3 1B.
“Notably, ReVISE enhances reasoning performance beyond prior methods, improving accuracy from 27.1 → 31.1% on GSM8K (Maj@3) (Cobbe et al., 2021) with Llama3 1B (Dubey et al., 2024)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Cobbe et al., 2021
For math reasoning on GSM8k, using continuous thoughts is beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.
“For math reasoning (GSM8k, Cobbe et al., 2021), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains.”
From Training Large Language Models to Reason in a Continuous Latent Space (2024) · cites this paper as Cobbe et al., 2021
Early studies have shown that LLMs exhibit basic reasoning capabilities in mathematics, coding, and general reasoning tasks.
“Early studies have shown that large language models (LLMs) exhibit basic reasoning capabilities in mathematics (Cobbe et al., 2021), coding (Chen et al., 2021), and general reasoning tasks.”
From Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning (2025) · cites this paper as Cobbe et al., 2021
GSM8K provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.
“GSM8K (Cobbe et al., 2021) provides 8,500 grade-school mathematics word problems requiring multi-step reasoning, with solutions averaging 2-8 reasoning steps.”
From Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression (2026) · cites this paper as Cobbe et al., 2021
Many benchmarks like MATH and GSM8K have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.
“Many benchmarks like MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have fixed sets of ground-truth answers, which allow for clear and objective measurements of model performance.”
From ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (2025) · cites this paper as Cobbe et al., 2021
Meta Agent Search improves accuracy over baselines by 25.9% on GSM8K after transferring across domains.
“they improve accuracy over baselines by 25.9% and 13.2% on GSM8K (Cobbe et al., 2021) and GSM-Hard (Gao et al., 2023) math tasks, respectively, after transferring across domains.”
From Automated Design of Agentic Systems (2024) · cites this paper as Cobbe et al., 2021
Each GSM8K is paired with a numerical reference answer.
“Each GSM8K is paired with a numerical reference answer.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [14]
When using model-generated chains of thought, the intermediate steps may not always be correct.
“When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a).”
From The Llama 3 Herd of Models (2024) · cites this paper as Cobbe et al., 2021
Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning.
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [1]
GSM8K and MATH are fitting choices for the testbed because they include both training and test sets, improving performance is challenging with limited effective datasets, and they are of widespread interest.
“Given these criteria, the mathematical reasoning benchmark datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) foot_0 emerge as fitting choices for our test bed, which allow us to study data leakage in depth but also offer a relevant and challenging environment.”
From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Cobbe et al., 2021
In contrast to LAMBADA and GSM8K, which test skills not directly applicable to software engineering, the SWAA set represents actions needed in both software engineering work and the longer tasks studied.
“In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [45]
Math Word Problems in NLR test reasoning on math problems similar to GSM8k but with varying levels of relational complexity, particularly degree of variable interdependence.
“(i) Math Word Problems test reasoning on math problems similar to GSM8k (Cobbe et al., 2021) but with varying levels of relational complexity, particularly in their degree of variable interdependence, which we define as the number of other variables on which a single variable's value depends.”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
The Math Word Problems category (21 problems) is inspired by grade-school math problems in datasets such as GSM8k but introduces an additional dimension of variable interdependence.
“This category (21 problems) is inspired by the grade-school math problems in datasets such as GSM8k (Cobbe et al., 2021), but introduces an additional dimension of variable interdependence (i.e., higher relational complexity between the variables).”
From Reliable Reasoning Beyond Natural Language (2024) · cites this paper as Cobbe et al., 2021
Both context and semantic caching lack flexibility for handling slight variations in input.
“Both context and semantic caching lack flexibility for handling slight variations in input.”
From Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents (2025) · cites this paper as [18]
The Introduction discusses errors in mathematical reasoning, and §4.3 discusses token-level verifiers. Neither addresses context caching, semantic caching, or their handling of slight input variations; searches for those terms found no relevant passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Specifically, we apply residual dropout (Vaswani et al., 2017) along the residual paths of each layer in the network.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“In recent years, large language models have demonstrated impressive skills across many diverse tasks (Wang et al., 2019; Brown et al., 2020).”
BackgroundSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1× · see citing works
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”
BackgroundSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Solving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1× · see citing works
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundDeep Neural Solver for Math Word Problems
Wang, Y., Liu, X., & Shi, S. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1088 · Cited by 404 · cited 1× · see citing works
“Previous work has attempted to solve classic math word problem benchmarks with recurrent seq2seq models (Sutskever et al., 2014) and closely related variants (Wang et al., 2017; Huang et al., 2018).”
BackgroundLearning to Automatically Solve Algebra Word Problems
Kushman, N., Zettlemoyer, L., Barzilay, R., & Artzi, Y. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1026 · Cited by 400 · cited 1× · see citing works
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 1× · see citing works
“AQuA-RAT (Ling et al., 2017) contains 100K problems, but this dataset unfortunately suffers from both a high degree of problem templatiza-tion and poor quality control of the natural language solutions.”
BackgroundMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 4× · see citing works
“However, even the largest models falter when required to perform multi-step mathematical reasoning (Hendrycks et al., 2021).”
BackgroundMethodInfluentialCommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 296 · cited 1× · see citing works
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275 · cited 1×
“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”
BackgroundDeep Learning for Symbolic MathematicsPreprint
Lample, G., & Charton, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01412 · Cited by 227 · cited 1× · see citing works
“Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
BackgroundLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 209 · see citing works
A Goal-Driven Tree-Structured Neural Model for Math Word Problems
Xie, Z., & Sun, S. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/736 · Cited by 203 · cited 1× · see citing works
“More recent work has improved performance by designing specialized encoder-decoder architectures (Amini et al., 2019; Chiang and Chen, 2018; Xie and Sun, 2019; Chen et al., 2020; Li et al., 2020), with the strongest results often relying on large pretrained encoders from the BERT family (Chen et…”
MethodHow well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation
Huang, D.-Q., Shi, S., Lin, C.-Y., Yin, J., & Ma, W.-Y. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p16-1084 · Cited by 158 · cited 1× · see citing works
“Dolphin18K (Huang et al., 2016) is a larger dataset containing 18K problems, but solutions are provided only in the form of equations or fi-nal answers.”
BackgroundMathQA: Towards Interpretable Math Word Problem Solving with Operation-Based FormalismsPreprint
Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13319 · Cited by 137 · cited 2× · see citing works
“More recent work has improved performance by designing specialized encoder-decoder architectures (Amini et al., 2019; Chiang and Chen, 2018; Xie and Sun, 2019; Chen et al., 2020; Li et al., 2020), with the strongest results often relying on large pretrained encoders from the BERT family (Chen et…”
BackgroundMethodA Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Miao, S.-Y., Liang, C.-C., & Su, K.-Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.92 · Cited by 135 · cited 2× · see citing works
“The recently developed ASDiv dataset (Miao et al., 2021), which contains 2.3K math word problems, addresses common flaws in prior datasets by ensuring problems have both high diversity and high quality.”
BackgroundMethodMathBERT: A Pre-Trained Model for Mathematical Formula UnderstandingPreprint
Peng, S., Yuan, K., Gao, L., & Tang, Z. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00377 · Cited by 93 · cited 1× · see citing works
“Similarly, Shen et al. (2021b) propose a pretrained a corpus of pre-K to college level curricula extracted from the internet, and Peng et al. (2021) select among many model completions.”
BackgroundGenerate & Rank: A Multi-task Framework for Math Word Problems
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., & Liu, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.195 · Cited by 88 · see citing works
Neural Math Word Problem Solver with Reinforcement Learning
Huang, D., Liu, J., Lin, C.-Y., & Yin, J. P. (2018) · International Conference on Computational Linguistics · Cited by 71 · see citing works
Li, S., Wu, L., Feng, S., Xu, F., Xu, F., & Zhong, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.255 · Cited by 71 · see citing works
Chen, X., Chen, L., Yu, A. W., Zhou, D., Song, D. X., & Le, Q. V. (2020) · International Conference on Learning Representations · Cited by 64 · see citing works
Collaborative Storytelling with Large-scale Neural Language Models
Nichols, E., Gao, L., & Gomez, R. (2020) · Motion, Interaction and Games · doi:10.1145/3424636.3426903 · Cited by 50 · cited 1× · see citing works
“Nichols et al. (2020) proposed a sample-and-rank approach to improve the collaborative storytelling ability of large language models, with the training signal coming from the preferences of human workers.”
BackgroundMathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47 · cited 1× · see citing works
“Similarly, Shen et al. (2021b) propose a pretrained a corpus of pre-K to college level curricula extracted from the internet, and Peng et al. (2021) select among many model completions.”
BackgroundKim, B., Ki, K. S., Lee, D., & Gweon, G. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-main.308 · Cited by 45 · cited 1× · see citing works
“…specialized encoder-decoder architectures (Amini et al., 2019; Chiang and Chen, 2018; Xie and Sun, 2019; Chen et al., 2020; Li et al., 2020), with the strongest results often relying on large pretrained encoders from the BERT family (Chen et al., 2019; Kim et al., 2020; Liang et al., 2021).”
BackgroundApe210K: A Large-Scale and Template-Rich Dataset of Math Word ProblemsPreprint
Zhao, W., Shang, M., Liu, Y., Wang, L., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11506 · Cited by 32 · cited 1× · see citing works
“Ape210K (Zhao et al., 2020) is the largest publicly available dataset, consisting of 210K Chinese elementary school-level math problems.”
BackgroundChen, K., Huang, Q., Palangi, H., Smolensky, P., Forbus, K. D., & Gao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02339 · Cited by 20 · see citing works
Semantically-Aligned Equation Generation for Solving and Reasoning Math Word ProblemsPreprint
Chiang, T.-R., & Chen, Y.-N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00720 · Cited by 10 · cited 1× · see citing works
“More recent work has improved performance by designing specialized encoder-decoder architectures (Amini et al., 2019; Chiang and Chen, 2018; Xie and Sun, 2019; Chen et al., 2020; Li et al., 2020), with the strongest results often relying on large pretrained encoders from the BERT family (Chen et…”
Method
Flagged first, then by citations
Cited by
Sorted by citations
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“An example of chain-of-thought prompting for the GSM8K benchmark (grade-school math problems) is given below in Figure 8.”
BackgroundMethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
“● Models have the ability to generate neutral language intermediate steps by training from scratch (Ling et al., 2017) or finetuning a pretrained model (Cobbe et al., 2021).”
BackgroundMethodInfluentialHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
“Self-consistency is far simpler than prior approaches that either train an additional verifier (Cobbe et al., 2021) or train a re-ranker given additional human annotations to improve generation quality (Thoppilan et al.”
BackgroundMethodInfluentialCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2019 · Cited by 755 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · see citing works
“To measure math-reasoning capabilities of the proposed method, we report results on the GSM8K benchmark Cobbe et al. (2021), which is comprised of a set of middle-school math word problems.”
MethodResultThe Ethics of Artificial Intelligence
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 608 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
“LLMs are known to have limited math and reasoning capability [9], which results in its failure of grading such questions because they do not know the correct answers.”
BackgroundMethodA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · see citing works
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344 · see citing works
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“Generating and combining many CoTs bridges the gap with human-level performances CoTs can be combined and/or filtered using human or automated feedback (Wang et al., 2022; Cobbe et al., 2021).”
BackgroundEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 274 · see citing works
A Survey on Model Compression for Large Language Models
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00704 · Cited by 273 · see citing works
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“…for a wide range of downstream tasks: large language models can enable not only fluent text generation [1, 2, 3] but emergent problem-solving [4, 5, 6] and creative generation of prose [7, 2] and code [8], while vision-language models enable open-vocabulary visual recognition [9, 10, 11] and…”
BackgroundThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
Zhou, H., Hu, C., Yuan, Y., Cui, Y.-F., Jin, Y.-L., Chen, C., Wu, H.-L., Yuan, D., Jiang, L., Wu, D., et al. (2024) · IEEE Communications Surveys and Tutorials · doi:10.1109/comst.2024.3465447 · Cited by 368 · see citing works
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E., Bing, L., Xu, X., Poria, S., & Lee, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.319 · Cited by 349 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
“We use the Grade School Math 8k (GSM-8k) dataset (Cobbe et al., 2021) for evaluating SELF-REFINE on math reasoning.”
BackgroundMethodInfluentialCasheekar, A. M., Lahiri, A., Rath, K., Prabhakar, K. S., & Srinivasan, K. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100632 · Cited by 236 · see citing works
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., & Gadepally, V. (2023) · 2023 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec58863.2023.10363447 · Cited by 225 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · see citing works
“[87] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.”
BackgroundLarge Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230 · see citing works
Generative AI in Higher Education: The ChatGPT Effect
Chan, C. K. Y., & Colloton, T. (2024) · Routledge · doi:10.4324/9781003459026 · Cited by 203 · see citing works
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123433119 · Cited by 187 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
A Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 334 · see citing works
Large language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161 · see citing works
Hagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 192 · see citing works
Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D. (2024) · Proceedings of the 17th ACM International Conference on Web Search and Data Mining · doi:10.1145/3616855.3635752 · Cited by 155 · see citing works
PAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · see citing works
Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., et al. (2023) · doi:10.2139/ssrn.4583531 · Cited by 154 · see citing works
MathPrompter: Mathematical Reasoning using Large Language Models
Imani, S., Du, L., & Shrivastava, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.4 · Cited by 212 · see citing works
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · see citing works
Understanding Large-Language Model (LLM)-powered Human-Robot Interaction
Kim, C. Y., Lee, C. P., & Mutlu, B. (2024) · Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3610977.3634966 · Cited by 162 · see citing works
Large Language Models Are Reasoning Teachers
Ho, N., Schmid, L., & Yun, S.-Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.830 · Cited by 238 · see citing works
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Shtedritski, A., Rupprecht, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01101 · Cited by 142 · see citing works
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.825 · Cited by 168 · see citing works
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 139 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“, how is large language model prompting affected by irrelevant context, and what strategies can be used to improve performance? To measure distractibility, we construct the GSM-IC dataset, a grade-school math problem dataset derived from GSM8K (Cobbe et al., 2021) and introduce two different metrics.”
BackgroundMethodInfluentialKuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142 · see citing works
Towards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 143 · see citing works
Faithful Chain-of-Thought Reasoning
Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., & Callison-Burch, C. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.20 · Cited by 201 · see citing works
Espejel, J. L., Ettifouri, E. H., Alassan, M. S. Y., Chouham, E. M., & Dahhane, W. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100032 · Cited by 131 · see citing works
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119 · see citing works
Language Models of Code are Few-Shot Commonsense Learners
Madaan, A., Zhou, S., Alon, U., Yang, Y., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.90 · Cited by 131 · see citing works
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120 · see citing works
Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency Department
Huang, J., Neill, L., Wittbrodt, M., Melnick, D., Klug, M., Thompson, M., Bailitz, J., Loftus, T., Malik, S., Phull, A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36100 · Cited by 113 · see citing works
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
How understanding large language models can inform the use of ChatGPT in physics education
Polverini, G., & Gregorcic, B. (2023) · European Journal of Physics · doi:10.1088/1361-6404/ad1420 · Cited by 131 · see citing works
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108 · see citing works
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106 · see citing works
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 112 · see citing works
Tool learning with large language models: a survey
Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., & Wen, J.-R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40678-2 · Cited by 148 · see citing works
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · see citing works
“Math datasets including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021) and CMath (Wei et al., 2023).”
BackgroundMethodMaieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 117 · see citing works
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · see citing works
“Using the GSM8K dataset (Cobbe et al., 2021), the models must correctly solve grade school mathematical reasoning tasks.”
Teaching Small Language Models to Reason
Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., & Severyn, A. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.151 · Cited by 127 · see citing works
Optimizing generative AI by backpropagating language model feedback
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Lu, P., Huang, Z., Guestrin, C., & Zou, J. (2025) · Nature · doi:10.1038/s41586-025-08661-4 · Cited by 85 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers
Zhu, Q., Zhang, X., & Luo, J. (2023) · Journal of Mechanical Design · doi:10.1115/1.4056598 · Cited by 89 · see citing works
Zhang, X., Li, S., Hauer, B. D., Shi, N. Z., & Kondrak, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.491 · Cited by 83 · see citing works
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Xiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 80 · see citing works
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024) · arXiv · doi:10.48550/arxiv.2410.12405 · Cited by 78
“These specific tasks include, but are not limited to, reading comprehension (Sakaguchi et al., 2021; Mostafazadeh et al., 2017), mathematical problem solving (Cobbe et al., 2021; Hendrycks et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).”
BackgroundRLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., & Hang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103809 · Cited by 81 · see citing works
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 99 · see citing works
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 132 · see citing works
Large Language Model based Multi-Agents: A Survey of Progress and Challenges
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.01680 · Cited by 70 · see citing works
Lanzi, P. L., & Loiacono, D. (2023) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3583131.3590351 · Cited by 69 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
A Review on the Use of Large Language Models as Virtual Tutors
García-Méndez, S., de Arriba-Pérez, F., & Somoza-López, M. D. C. (2024) · Science & Education · doi:10.1007/s11191-024-00530-2 · Cited by 78 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
- Authors
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J.
- ORCID
- Kosaraju, Vineet
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.14168
- OpenAlex
- W3210904070
- Cited by
- 575
- References
- 28
- Citations by year
- 2026: 30 · 2025: 137 · 2024: 156 · 2023: 216 · 2022: 34 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Mathematics › Applied Mathematics › Mathematics Education and Pedagogy
- Keywords
- mathematical word problems, large language models, solution verification, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
