Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“These approaches marked the beginning of the prompting era and were quickly followed by methods to elicit algorithmic and structured reasoning (Roy and Roth, 2015; Ling et al., 2017).”

“…model to a standard autoregressive attention mechanism. difficulty mathematical reasoning datasets (GSM8k-Aug (Deng et al., 2024), GSM-Hard (Gao et al., 2023), SVAMP (Patel et al., 2021), and MultiArith (Roy & Roth, 2015)) and high-difficulty datasets (Math500 (Hendrycks et al., 2021) and AIME24).”

“For arithmetic reasoning tasks, we consider the following six arithmetic reasoning problem benchmarks: (1) Multi-Arith (Roy and Roth, 2015), (2) GSM8K (Cobbe et al., 2021), (3) AddSub (Hosseini et al., 2014), (4) AQuA (Ling et al., 2017), (5) SingleEq (Koncel-Kedziorski et al., 2015), and (6) SVAMP (Patel et al., 2021…”

“It integrates several datasets (Hosseini et al., 2014; Kush-man et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015) and thus serves as a comprehensive benchmark. increasing the number of datasets used in training with MinT boosts performance across all views.”

“Tree-based methods (Koncel-Kedziorski et al., 2015; Roy and Roth, 2016; Roy et al., 2016; Roy and Roth, 2017) utilize the inherent binary tree-like structure of expressions/equations.”

“Indeed, Kojima et al. (2022) show various irrelevant and misleading base-lines perform poorly on an arithmetic dataset (Roy and Roth, 2015).”

“Such discrete reasoning has indeed been incorporated for simpler, well-structured tasks like math word problems with a small context and few number entities like (Koncel-Kedziorski et al. 2016; Roy and Roth 2015) or QA over structured data (Knowledge Bases or Tables) (Zhong, Xiong, and Socher 2017; Liang et al.”

“Inference Tasks: For mathematical application problems, we used GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2016).”

“5 with self-consistency is capable of achieving close to 100% accuracy on certain tasks such as MultiArith (Roy and Roth, 2015), ARC (Clark et al., 2018), and CoinFlip (Wei et al., 2022).”

“The model can learn to predict new operations when appropriate in a math word problem setting (Roy and Roth, 2015) with very few training examples.”

“The evolution subset as the prompt candidates for evaluation are determined by searching the performance of math reasoning on 100 random examples.”

“For mathematical reasoning, we employ a diverse set of five benchmarks: GSM8K(Cobbe, Kosaraju, and Bavarian 2021), Mul-tiArith(Roy and Roth 2016), SVAMP(Patel, Bhattamishra, and Goyal 2021), AQuA(Ling et al. 2017), and the particularly challenging MATH-500(Hendrycks et al. 2021b) subset.”

“For example, GSM8K [5], SVAMP [38], MultiArith [4] are for mathematical reasoning, they are in the same semantic domain.”

“As shown in Figure 1, we input a mathematical problem from the MultiArith dataset (Roy and Roth, 2015) into GPT-4o mini 1 , where Zero-shot-CoT, PoT, and PS produced incorrect answers.”

“While earlier models for solving MSPs (Hosseini et al., 2014; Kushman et al., 2014; Roy and Roth, 2015) focused on extracting various features from text to learn probabilistic models, recent efforts have used pre-trained large language models (LLMs) (Yang et al., 2021; Drori et al., 2022; Lewkowycz…”

“Some solvers (such as Roy and Roth (2015)) had a joint optimisation objective to identify relevant numbers and populating the expression tree.”

“Researchers [4], [35] at Illinois University developed an MWP solver which can solve diverse arithmetic word problems.”

“For example, Roy et al. [17] proposed a tree-based method to represent and evaluate the target arithmetic expressions for solving algorithmic problems.”

“For Arithmetic Reasoning , we employ GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015).”

“The math datasets for evaluation include GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), Multi-arith (Roy and Roth, 2015), Svamp (Patel et al., 2021), and Singleeq (Koncel-Kedziorski et al., 2015).”

“We further assess MoS 2 in text generation by fine-tuning LLaMA-7B [50] ontheMath10K[17]datasetandevaluatingitontheMultiArith[43] andAddSub[14]benchmarks,followingLLM-Adapters[17].”

“Roy et al. (Roy et al., 2015; Roy and Roth, 2016) mapped unstructured text to a more easily reasoned representation space to eliminate text ambiguity, which can reason multi-step arithmetic problems.”

“…learning abilities (Brown et al., 2020; Wei et al., 2022; Min et al., 2022) to guide LLMs to generate desired task responses, marking the advent of the prompting era and surpassing the age of the intermediate steps in algorithmic and structured reasoning (Roy and Roth, 2015; Ling et al., 2017).”

“However, excessive emphasis on self-reflection learning (i.e., a larger value of λ ) typically leads to poorer performance and instability, especially for the MultiArith dataset.”

“In addition, to verify the model’s generalization capability on out-of-distribution tasks, we also test on the Multi-Arith (Roy and Roth, 2015), ASDiv (Miao et al., 2020) and SVAMP (Patel et al., 2021) datasets.”

“Solving arithmetic (Roy and Roth, 2016) or algebra (Kushman et al., 2014) word problems.”

“Arithmetic: AQUA (Ling et al., 2017), GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015b).”

“The algorithmic procedure is provided in Appendix A.2, and Roth, 2015), AQuA (Patel et al., 2021), and SVAMP (Ling et al., 2017); and code generation with HumanEval (Chen et al., 2021).”

“Problems in Contextual Reasoning category are sourced from datasets AddSub (Hosseini et al., 2014; Mishra et al., 2022), SingleOp (Roy and Roth, 2015; Mishra et al., 2022), SVAMP (Patel et al., 2021), and GSM8K (Cobbe et al., 2021), with increasing difficulty.”

“Solving arithmetic (Roy and Roth, 2016) or algebra (Kushman et al., 2014) word problems.”

“We conduct our experiments across twelve datasets categorized in 4 categories: (1) Arithmetic , including MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AddSub (Hos-seini et al., 2014), AQUA-RAT (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021) (2) Commonsense…”

“MultiArith [18] is the subset of the Mawps repository [17].”

“Mathematical Reasoning We considered five popular mathematical reasoning datasets, namely GSM8K (G8K) (Cobbe et al., 2021), MultiArith (MArith) (Roy and Roth, 2016), AQuA (Ling et al., 2017), ASDiv (Miao et al., 2021), and MATH (Hendrycks et al., 2021a).”

“We conduct comprehensive experiments on eight math word problem datasets, including AddSub (Hosseini et al. 2014), SingleOp (Roy, Vieira, and Roth 2015), MultiArith (Roy and Roth 2015), SingleEq (Koncel-Kedziorski et al. 2015), SVAMP (Patel, Bhattamishra, and Goyal 2021), GSM8K (Cobbe et al. 2021), GSM-IC2-1K (Shi et…”

“Specifically, rule-based methods rely on manually crafted schemas and pattern matching [2], while statistic-based methods leverage traditional machine learning models like SVM to select and complete pre-defined templates [27].”

“We evaluate the model trained on MetaMathQA on in-distribution MATH500 (Hendrycks et al.) and out-of-distribution multiArith (Roy & Roth, 2015) and DeepMind-Mathematics (Saxton et al., 2019).”

“We evaluate on a diverse set of mathematical benchmarks, including SVAMP (Patel et al., 2021), GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016) GSMSymbolic-p1 (Mirzadeh et al., 2024), MATH-500 (Hendrycks et al., 2021), AQUA (Ling et al., 2017), AIME (2024), OlympiadBench (He et al., 2024) datasets.”

“For arithmetic reasoning tasks, we also use several professional datasets, including MultiArith (Roy and Roth, 2016), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021) different arithmetic reasoning-related problems.”

“This problem has received significant attention in the recent literature (Lu et al., 2022), and specific datasets (Cobbe et al., 2021; Roy and Roth, 2015; Patel et al., 2021) have been proposed as a benchmark for this task.”

“multiplication, and division word problems. 1) AI2 [3]. AI2 is a collection of 395 addition and subtraction problems, containing numeric values, where some may not be relevant to the question. 2) CC [20]. The Common Core dataset contains 600 2- step questions. The Cognitive Computation Group at the University of Penns…”

“To evaluate the effectiveness of our approach, we conduct experiments using seven mathematical and reasoning datasets: AddSub (Hosseini et al., 2014), SingleEQ (Koncel-Kedziorski et al., 2015), Multi-Arith (Roy and Roth, 2016), GSM8k (Cobbe et al., 2021), ASDiv (Miao et al., 2021), SVAMP (Patel et al., 2021), and MATH…”

“Across all models except for Zephyr, the impact of noise was more pronounced in the GSM8K dataset than in MultiArith, with a larger drop in accuracy as the noise levels increased (Figure 3).”

“CC (Roy and Roth, 2015).”

“[27–29] proposed various methods to understand a word problem and developed algorithms to form expression tree (representing a word problem) with relevant quantities.”

“A common property of these datasets is that they have been curated such that any given question can be solved by a limited-depth operator tree (AI2, Illinois, Commoncore) or a limited set of equation templates (Alg514 and DRAW).”

“We choose this particular arithmetic reasoning dataset as it contains real-world math problems paired with diverse natural language texts as opposed to synthetically generated problems (Roy & Roth, 2015).”

“We followed Hu et al. [28] and evaluated S 2 FT on seven math reasoning tasks, including MultiArith [55], GSM8K [14], AddSub [25], AQuA [37], SingleEq [31], SVAMP [50] and MAWPS [32].”

“We consider GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016) datasets and Mixtral-8x7B, Llama-2 70B language models.”

“Building upon prior efforts in mathematical problem dataset curation, we leverages existing MWP datasets (Koncel-Kedziorski et al., 2016; Roy and Roth, 2016, 2018; Kushman et al., 2014).”

“The second component of our system is the arithmetic word problem solver developed in our previous work (Roy and Roth, 2015).”

“…are commonly used in evaluating CoT reasoning ability (Zheng et al., 2023; Wang et al., 2023): (i) AddSub (Hosseini et al., 2014), (ii) Multi-Arith (Roy and Roth, 2015), (iii) SingleEQ (Kon-cel-Kedziorski et al., 2015), (iv) SVAMP (Patel et al., 2021), (v) GSM8K (Cobbe et al., 2021), (vi) AQuA…”

“MultiArith (Roy and Roth, 2016) consists of 600 multi-step arithmetic problems extracted from the commoncoresheets website.”

“We evaluate our CoT framework on 10 reasoning datasets, including six arithmetic datasets ( i.e., MultiArith (Roy & Roth, 2016), GSM8K (Cobbe et al., 2021), SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), and SVAMP (Patel et al., 2021)), two commonsense reasoning da…”

“Most prior research has focused on elementary arithmetic problems (Roy and Roth, 2015; Koncel-Kedziorski et al., 2016; Patel et al., 2021; Cobbe et al., 2021) and algebra problems (Kushman et al., 2014; Huang et al., 2016) due to their straightforward solvability and evaluative convenience through…”

“It is impor-6817 (Hendrycks et al., 2021c) EN 5000 TabMWP (Lu et al., 2023) EN 7686 Dolphin18K (Huang et al., 2016) EN 1504 Math23K (Wang et al., 2017) ZH 1000 ASDiv (Miao et al., 2020) EN 2305 SingleOp (Roy et al., 2015) EN 562 AddSub (Hosseini et al., 2014) EN 395 MultiArith (Roy and Roth, 2015) EN 600 MMLU-Math (He…”

“…al., 2021) for diverse grade school math problems, (2) SVAMP dataset (Patel et al., 2021) for challenging math problems, and (3) MultiArith dataset (Roy & Roth, 2016) for multi-step reasoning 4 LLMs Waver in Judgements As main results, we analyze ChatGPT’s judgement consistency in arithmetic (§…”

“In the context of natural language understanding, recent studies on MWPS have been mainly conducted using machine learning techniques [2, 3].”

“Other related work explores the use of language models to solve algebra word problems (Kushman et al. 2014; Roy and Roth 2015) or to evaluate nested arithmetic expressions (Hupkes and Zuidema 2018).”

“Roy and Roth (2016) (2022b) introduced two benchmarks: NumGLUE, with 100K problems across eight tasks to assess arithmetic reasoning, and LILA (Mishra et al., 2022a), which combines 20 datasets for mathematical reasoning, categorized into 23 tasks.”

“We conduct a series of experiments on various reasoning benchmarks: arithmetic reasoning: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), AddSub (Hosseini et al., 2014), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017) and SingleOp (Wei et al., 2022). common-sense reasoning: CSQA…”

“To assess the end-to-end performance of our integrated framework, we evaluate it on six standard benchmarks spanning three domains: general reasoning ( MMLU (Hendrycks et al., 2021)), mathematical problem solving ( GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), MultiArith (Roy and Roth, 2016), and SVAMP (Patel…”

“SVAMP [15] with 1,000 samples, and MultiArith [16] with 180 samples.”

“• Arithmetic Reasoning : This benchmark consists of seven subsets: MultiArith [25], GSM8K [26], AddSub [27], AQuA [28], SingleEq [29], SVAMP [30] and MAWPS [31].”

“We conducted evaluations across seven mathematical reasoning datasets, including GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2020), MultiArith (Roy and Roth, 2015), SingleEQ (Koncel-Kedziorski et al., 2015), SingleOP (Roy et al., 2015), and AddSub (Hosseini et al., 2014).”

“The CoinFlip dataset is sourced from (Wei et al., 2022), the MultiArith dataset is acquired from (Roy and Roth, 2016), and the remaining datasets are derived from the Big-Bench dataset (bench authors, 2023).”

“Recent research (Hosseini et al., 2014; Kushman et al., 2014; Roy and Roth, 2015; Wang et al., 2017, 2018b,a) focused on designing algorithms to automatically solve arithmetic word problems.”

“Early attempts used hand-crafted features collected by experts to make a model understand a word problem (Kushman et al., 2014; Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Zhou et al., 2015; Upadhyay et al., 2016; Roy and Roth, 2017).”

“Traditional math word problem (MWP) datasets like AddSub (Hosseini et al., 2014) and MultiArith (Roy and Roth, 2016) are integrated into broader MWP repositories.”

“Including StrategyQA (Geva et al., 2021) for commonsense reasoning, Addsub (Hosseini et al., 2014), Multiarith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), SingleEq (Koncel-Kedziorski et al., 2015) and GSM8K (Cobbe et al., 2021) for arithmetic math inference and Date Understanding (Srivastava…”

“On MultiArith with Qwen2.”

“(a) Arithmetic Reasoning: MultiArith (Roy and Roth, 2015) includes elementary school arithmetic questions.”

“They produced CC_Ko and IL_Ko by translating the two English MWPS datasets, IL and CC [39].”

“To evaluate our method, we use a range of datasets focused on mathematical reasoning and multi-task problem solving, including AddSub [8], MultiArith [9], and Sin-gleEQ [10] for arithmetic reasoning, as well as SVAMP [11] and GSM8K [12] for multi-step problem solving.”

“Unlike fine-tuning task-specific models (Chiang and Chen, 2019; Ling et al., 2017; Roy and Roth, 2016; Cobbe et al., 2021), recent advances in Large Language Models (LLMs) have paved the way for new approaches that employ in-context learning (ICL) (Wei et al., 2023) to solve complex reasoning…”

“This strategy can sometimes “fool” the LLMs into producing incorrect or irrelevant answers for certain tasks, such as arithmetic (Roy and Roth, 2016) and commonsense reasoning (Talmor et al., 2018).”

“MultiArith is a multi-step arithmetic reasoning dataset (Roy and Roth, 2016).”

“The RoCoFT 3-Row method achieves the best results on Boolean Questions (BoolQ, 65.92%), MultiArith (89.45%), and S.MP (56.79%), while the RoCoFT 3-Column method excels on SIQA (69.96”

“(1) Mathematical reasoning incorporates six representative datasets, namely GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SingleEQ (Koncel-Kedziorski et al., 2016), SVAMP (Patel et al., 2021), AddSub (Hos-seini et al., 2014), and AQuA (Ling et al., 2017).”

“Encouragingly, we find that 96.5% and 92.5% of the generated ASTs for GSM8K and MultiArith, respectively, are valid and free from these errors.”

“On AddSub dataset, few-shot R 3 prompting method surpasses the SoTA results (Roy and Roth, 2016) which is obtained by training with full data ( 94 .”

“A commonly used IMR is constant expressions (Roy and Roth 2015; Koncel-Kedziorski et al. 2016).”

“…previous work (Tan et al., 2025), we primarily train and evaluate models on GSM8K-Aug (Deng et al., 2023), and additionally evaluate trained models using three out-of-domain math reasoning datasets: GSM-Hard (Gao et al., 2023), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015).”

“Then, on three grade-school level mathematical benchmarks (GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MultiArith (Roy & Roth, 2015)), MarCos obtains performance on par with or even superior to transformer-based CoT reasoning , with a 4 .”

“For out-of-distribution evaluation, we adopt GSM-HARD (Gao et al., 2023), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015) as robustness benchmarks.”