Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“Math Word Problems in NLP Math word problems have long been a focus of interest in the NLP community (Roy and Roth, 2015; Kushman et al., 2014; Huang et al., 2017; Amini et al., 2019; Xie and Sun, 2019; Drori et al., 2022), with research primarily aiming to improve computational models’ ability to…”

“Specifically, for the math reasoning problem, we use Multiarith (Roy and Roth, 2015) for evaluation.”

“For numerical and mathematical reasoning, we follow CoT (Wei et al., 2022) and include GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), ASDiv (Miao et al., 2020), and AQuA (Ling et al., 2017).”

“…10 reasoning datasets, including: (1) 6 arithmetic datasets: SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), Mul-tiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AQUA-RAT (Ling et al., 2017), and SVAMP (Patel et al., 2021); (2) 2 commonsense reasoning datasets:…”

“Addressing whether the observed performance drop in LLMs is primar-135 Koncel-Kedziorski et al., 2015; Roy and Roth, 2016; Liang et al., 2016; Ling et al., 2017).”

“The lack of a positional bias supports the claim that LLMs can use information equally well across different positions under multiple homogeneous problems.”

“…al., 2021b) numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015b; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021b; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive…”

“Some solvers (such as Roy & Roth, 2015) had a joint optimization objective to identify relevant numbers and populating the expression tree.”

“Traditional MWP datasets like AddSub (Hosseini et al., 2014) and MultiArith (Roy and Roth, 2015) are integrated into a MWP repository.”

“…2023; Isik et al., 2024) have shown that further fine-tuning is still crucial for tasks involving complex reasoning such as arithmetic reasoning (Roy and Roth, 2016; Cobbe et al., 2021) and commonsense reasoning (Mihaylov et al., 2018; Clark et al., 2019), etc. Fine-tuning LLMs requires…”

“We evaluate on five benchmark datasets involving arithmetic and commonsense CoT reasoning: GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), MathQA (Amini et al., 2019) and CSQA (Talmor et al., 2019).”

“For the two arithmetic reasoning tasks (MultiArith and AddSub), the performance of PoT fluctuates wildly after removing the external interpreter.”

“Statistical machine learning methods were developed to map math word problems to specific equation templates (Kushman et al., 2014; Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2017).”

“…on two categories of QA tasks: (1) Fixed QA , where the answer set or format is constrained, including GSM8K and Mul-tiArith (Cobbe et al., 2021; Roy and Roth, 2015) for multi-step arithmetic reasoning, and Sports understanding (Suzgun et al., 2022) from Big-Bench-Hard for binary reasoning over…”

“These findings MultiArith SVAMP SingleEq Spearman rank -0.09 (0.6) 0.01 (1.0) 0.01 (0.9) -0.25 (0.1) -0.07 (0.7) -0.12 (0.5) (a) Mathematical reasoning tasks Test Type Dolly Self SNI UNI Vicuna Spearman rank 0.3 (0.2) -0.06 (0.8) -0.06 (0.8) -0.13 (0.6) 0.07 (0.8) (b) Instruction following tasks Table 5: Spearman rank…”

“We evaluate our approach across various mathematical reasoning datasets and demonstrate that SIKeD improves the ability of smaller models to handle complex reasoning, achieving significant performance gains.”

“…we use include the mathematical reasoning dataset SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2016), the SingleEQ (Koncel-Kedziorski et al., 2015), GSM8K (Cobbe et al., 2021), and the commonsense inference dataset ARC (Clark et…”

“We conduct experiments with text-davinci-002 on the GSM8k, AQuA, and MultiArith datasets.”

“Shanahan et al. (2023) illustrate LLM-based dialogue agents are not conscious beings with personal motives or a sense of self-preservation; their display of such traits is simply an act of role-play.”

“Early studies relying on hand-crafted features [14,19,20] and predefined patterns [21] have limitations in generalization.”

“We use GSM8k (Cobbe et al., 2021) as seed data to collect trajectories and test the performance on four math testing sets, including GSM8k, ASDiv (Miao et al., 2020), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015).”

“Datasets We evaluate our approach across seven benchmarks, covering three categories of reasoning tasks: (i) Arithmetic Reasoning: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), Addsub (Hosseini et al., 2014), AQuA (Ling et al., 2017) (ii) Commonsense Reasoning: Com-monsenseQA (Talmor et al., 2019) and S…”

“Even on the simpler MultiArith benchmark, where the uncompressed Base Model approaches saturation, TokenSkip’s accuracy notably degrades at γ = 0 .”

“These datasets have been meticulously curated from various online sources [74, 75] The “Ape210k” dataset is a highly robust and comprehensive corpus of mathematical problem-solving, boast-ing 210,488-word problems and an impressive 56,000 unique equation templates.”

“…the methods including, GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), AQUA-RAT (Ling et al., 2017), SingleOp and MultiArith subsets from (Roy and Roth, 2016) , which can measure the reasoning ability of LLM. Additionally, we examine the performance of S 3 Prompt on the high school…”

“Multi-Arith is a dataset of 180 test samples published with the algorithmic solver for mathematical word problems.”

“, 2016), CommonCore (Roy and Roth, 2015) and Illinois (Roy and Roth, 2016).”

“First, because we chose experimental stimuli from existing datasets (Roy & Roth, 2015; Roy et al., 2015), we could not perfectly balance the number of letters and digit sizes.”

“Meanwhile, in the generation [7, 13] method, a model learns to generate the correct equation template directly using tokens as a unit.”

“Benchmark Selection We evaluate our method on a diverse set of tasks spanning multiple domains, including natural language understanding with GLUE (Wang et al., 2018), natural language generation with E2E (Novikova et al., 2017), mathematical reasoning with GSM8k (Cobbe et al., 2021), SingleEq (Koncel-Kedziorski et al…”

“…(Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDiv (Miao et al., 2020), AQuA (Ling et al., 2017), AddSub (Hosseini et al., 2014), Multi-Arith (Roy and Roth, 2015), SingelEQ (Roy et al., 2015), three commonsense reasoning benchmarks (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021),…”

“…in early (rule-based) research (Briars and Larkin, 1984; Fletcher, 1985; Bakman, 2007; Yuhui et al., 2010) to solving linear equations in later statistical-NLP approaches (Hosseini et al., 2014; Kushman et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Upadhyay and Chang, 2015).”

“…2022; Ning et al., 2024), generating multiple candidate outputs with verification (Gou et al., 2024; Wang et al., 2023; Zhang et al., 2024b;a), and using non-linear reasoning structures to organize complex inference paths (Yao et al., 2023; Roy & Roth, 2015; Besta et al., 2024; Wei et al., 2022).”

“%) and MultiArith.”

“…the performance of T OPO DIM on three types of reasoning datasets: ❶ General Reasoning: MMLU-Pro (Wang et al., 2024b); ❷ Mathematics: MultiArith (Roy and Roth, 2016), GSM8K (Cobbe et al., 2021) , AIME (2023 − 2025) ; ❸ Coding: HumanEval (Hendrycks et al., 2020); LiveCodeBench (202305 − 202403)…”

“The mathematical reasoning category comprises GSM8K[15] for grade school math problems,MultiArith[44]for arithmeticwordproblems,SVAMP[38] for structurally diverse math questions, and AQuA[33] for algebraic reasoning.”

“Table 1: MultiArith prompts found by comparing traditional global optimization approach against our proposed local optimization.”

“We evaluate our method on ten reasoning datasets, including six arithmetic datasets ( i.e., MultiArith (Roy and Roth, 2016), GSM8K (Cobbe et al., 2021), SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), and SVAMP (Patel et al., 2021)), two…”

“Besides, LLMs can identify meaningful information within noisy environments [29, 4].”

“Roy and Roth (2015) proposed Expression Tree ( S 03 ) as the first bridge state.”

“This sometimes poses a challenge on reasoning problems when the base language model generates compound expressions requiring more than one operation to evaluate, such as 3 + 5 + 7 = .”

“Later, researchers pointed out that the output expression can be better described as a tree structure, e.g. expression tree [72], equation tree [25], so the Seq2Tree model was proposed.”

“In terms of datasets, for mathematics, we select easy mathematical datasets such as GSM8K (Cobbe et al., 2021) and MultiArith (Roy and Roth, 2015), and hard mathematical dataset MATH (Hendrycks et al., 2021b)—of which we opt for levels 1, 2, and 3, excluding levels 4 and 5 due to their extreme complexity.”

“However, these models predominantly focus on simpler mathematical prob-1 lems (Cobbe et al., 2021; Ling et al., 2017; Roy & Roth, 2016; Patel et al., 2021).”

“The source of complexity in complex QA tasks can be due to a complex question structure [27, 49], the need to perform comparisons or aggregations [19, 29], mathematical operations [42], or having to extract from complex objects like tables and text [7].”

“The subset AddSub features relatively easier numerical reasoning, whereas MultiArith necessitates multi-step numerical calculations.”

“The MWP task includes number word problems [22], logic puzzle problems [23], arithmetic word problems [24], [25], algebra word problems [26]–[28], and geometry word problems [29], and have drawn interests from lots of researchers.”

“The MAWPS dataset combines four datasets that were published previously by [5], [7], [8], and [9].”

“The features are identical to the ones that Roy and Roth (2015) describe.”

“Hosseini et al. [5], Kedziorski et al. [6] and Roy & Roth [7]-[9] have been working on graph based approaches by considering how quantities and units could be expressed as states/trees to generate arithmetic equations.”

“Initial statistical based approaches including the works of Hosseini et al. [3] and Roy and Roth [4] mainly focus on simple arithmetic math word problems.”

“…answering (Xu et al., 2022; Hopkins et al., 2019), including reading comprehension (Dua et al., 2019) and math word problems (Hosseini et al., 2014; Roy and Roth, 2015; Koncel-Kedziorski et al., 2016; Huang et al., 2016; Amini et al., 2019; Miao et al., 2020; Patel et al., 2021; Cobbe et al.,…”

“Arithmetic Reasoning To evaluate the method’s generalization ability from symbolic operations to arithmetic reasoning in addition and subtraction tasks, we use five established arithmetic reasoning datasets: AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015), MultiArith (Roy & Roth, 2016), GSM8K…”

“In the mathematical domain, we employed four widely-used datasets: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), and AQUA (Ling et al., 2017).”

“All models are trained on the GSM8K training set (Cobbe et al., 2021) and evaluated on four mathematical reasoning benchmarks, including GSM8K-NL (Cobbe et al., 2021), ASDiv-Aug (Xu et al., 2025), MultiArith (Roy & Roth, 2015) and SVAMP (Patel et al., 2021).”

“We evaluate BiRouter on six public benchmarks: MMLU (Hendrycks et al. 2021), GSM8K (Cobbe et al. 2021), MultiArith (Roy and Roth 2016), SVAMP (Pa-tel, Bhattamishra, and Goyal 2021), HumanEval (Chen et al. 2021) and MBPP (Austin et al. 2021).”

“As many models are already post-trained on mathematical reasoning data, the easier dataset shows only small performance differences, we therefore use two math datasets: MultiArith (Roy & Roth, 2015) (easy) and the combination of AIME’24’25 (math ai, 2025a;b) (hard).”

“APE is tested on several datasets like 24 instruction induction tasks from [26], a curated subset of 21 tasks from BIG-Bench, MultiArith [34] and GSM8K [35] for zero-shot chain-of-thought reasoning, Truth-fulQA [36] for evaluating truthfulness and informativeness.”

“• MultiArith (Roy and Roth, 2016) - a dataset focused on arithmetic word problems that require multiple steps to solve.”

“• MultiArith (Roy and Roth, 2015): This dataset consists of multi-step arithmetic word problems involving basic operations, such as addition followed by subtraction or subtraction followed by division.”

“MultiArith, AddSub, and SingleEq come from the Math World Problem Repository (Koncel-Kedziorski et al., 2016), while the other three are from more contemporary benchmarks.”

“Though there are quite a few public datasets for math word problem solving (Koncel-Kedziorski et al. 2016), the only dataset which comprises word problems attached with unique schemas is the SingleOp (Roy and Roth 2016) dataset with 562 word problems.”

“In Section 5.2, we evaluate on diverse reasoning tasks: mathematical reasoning (GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), Multi-Arith (Roy and Roth, 2015)), general knowledge (MMLU (Hendrycks et al., 2021)), and common-sense reasoning (Arc-challenge (Clark et al., 2018), CommonsenseQA (Talmor et al., 201…”

“…we use include the mathematical reasoning dataset SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2016), the SingleEQ (Koncel-Kedziorski et al., 2015), GSM8K (Cobbe et al., 2021), and the commonsense inference dataset ARC (Clark et…”

“Representative datasets include arithmetic word problem benchmarks based on verb categorization [18], algebraic equation parsing [20], rationale generation for algebraic reasoning [22], general arithmetic word problems [29], and robustness-oriented math word problem evaluation [28].”

“As shown in Table 4, a state-of-the-art model already achieves near-perfect performance on several widely used datasets, including HumanEval [13], MBPP [8], MultiArith [54], GSM8K [16], MATH [24], and MMLU [23].”

“• MultiArith [46, 47].”

“Following (Zhang et al., 2025b), we evaluate our approach on general reasoning us-ing MMLU (Hendrycks et al., 2021), mathematical reasoning using GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), and AQuA (Ling et al., 2017), as well as code generation using HumanEval (Chen et al…”

“Task Type Metric GSM8K(Cobbe et al., 2021) Grade-school mathematical word problems Accuracy SVAMP(Patel et al., 2021) Arithmetic word problems Accuracy MultiArith(Roy & Roth, 2015) Multi-step arithmetic word problems Accuracy”

“…a general reasoning dataset, MMLU (Hendrycks et al. 2021); (2) mathematical reasoning datasets, including GSM8k (Cobbe et al. 2021), Multi-Arith (Roy and Roth 2015), SVAMP (Patel, Bhattamishra, and Goyal 2021), AQuA (Ling et al. 2017); and (3) a code generation dataset, HumanEval (Chen et al.…”

“During the evaluation phase, we systematically benchmark our model across six diverse datasets: GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2016), AddSub (Hos-seini et al., 2014), AQuA (Ling et al., 2017), and SingleEq (Koncel-Kedziorski et al., 2015).”

“Math follows Kojima et al. (2022) and is a combination of two datasets of multi-step math questions (MultiArith (Roy & Roth, 2016) + SVAMP (Patel et al., 2021)) which require chain-of-thought reasoning for answering.”

“…Models (LLMs) show strong performance on mathematical benchmarks like GSM8K, SVAMP, MultiArith, and ASDiv (Cobbe et al., 2021; Patel et al., 2021; Roy and Roth, 2015; Miao et al., 2020), yet they frequently make systematic errors, often reap-plying familiar solution strategies even when the…”

“…benchmarks where both reasoning steps are of the same type—i.e., knowledge-based Bam-boogle (Press et al., 2023) and math-based Multi-Arith (Roy and Roth, 2015)—the compositionality gap is low or non-existent, and failure to reach the correct solution for a compositional question is mainly…”

“Early benchmarks mainly focus on fundamental arithmetic [RR15] and algebraic [LYDB17] problems.”

“The benchmarks used are categorized as follows: • Mathematical : GSM8K (Cobbe et al., 2021), Multi-Arith (Roy & Roth, 2016) • Symbolic : FOLIO (Han et al., 2022), Contex-tHub (Hua et al., 2024) • Knowledge : ARC (Clark et al., 2018), GPQA (Rein et al., 2023) • Soft Reasoning : MuSR (Sprague et al.,…”

“MultiArith (Roy and Roth, 2015) is a collection of complicated arithmetic problems designed to test machine learning models.”

“Dataset and LLMs We evaluate our method on popular mathematical benchmarks: (1) the GSM8K benchmark of math word problems (Cobbe et al., 2021); (2) MultiArith (Roy & Roth, 2016); (3) the SVAMP dataset of math word problems with varying structures (Patel et al., 2021); (4) the MAWPS benchmark (Koncel-Kedziorski et al.,…”

“Earlier work mainly focused on training reasoning models that would generate intermediate steps based on the given problem, expressed in formal language (Roy and Roth, 2015; Amini et al., 2019) and natural language (Ling et al., 2017; Cobbe et al., 2021a).”

“We performed evaluation on 7 math datasets: MultiArith (Roy & Roth, 2016), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), SingleEQ (Koncel-Kedziorski et al., 2015), SVAMP (Patel et al., 2021), and MAWPS (Koncel-Kedziorski et al., 2016).”