Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“The gains on SVAMP and MultiArith are particularly striking.”

“Our evaluation spans seven challenging benchmarks: MMLU [20] for general reasoning; GSM8K [21], Multi-Arith [22], SVAMP [23], AQuA [24], and MATH-500 [25] for mathematical reasoning; and HumanEval [26] for code generation.”

“2) MAWPS [16] is an online repository that unifies datasets from several previous works [155], [157]– [159].”

“The dynamic operator selection enables efficient scaling: simple arithmetic problems [15] use lightweight single-operator solutions (O I/O only), while complex multi-step problems trigger sophisticated collaboration patterns inspired by program-aided reasoning [73], [74] Code Generation: HumanEval…”

“In language-based tasks, NeSy models have successfully mapped text to symbolic forms such as equations and expression trees (Roy and Roth, 2016; Chiang and Chen, 2018; Chen et al., 2019), allowing for structured, inter-pretable reasoning beyond shallow pattern matching.”

“Roy et al. [28] presented a pioneering algorithmic approach to automatically solve complex arithmetic word problems without annotations or templates, achieving state-of-the-art performance.”

“…[8, 9] showing their superior potential by conducting comprehensive evaluation on reasoning benchmarks, including arithmetic reasoning [10–12], commonsense reasoning [13–15], logical reasoning [16–18]), symbolic reasoning [1], event relation reasoning [19, 20], augmentation [21, 22], and…”

“…2014), which is a collection of addition and subtraction problems, (2) SingleEq (Roy et al., 2015), which contains single equation problems, (3) SingleOp (Roy et al., 2015) with single operation arithmetic problems, and (4) SimulEq (Kushman et al., 2014) with multiple equation math word problems.”

“A DD S UB AQ U A GSM8K MAWPS M ULTI A RITH S INGLE E Q SWAMP A VG .”

“We conducted experiments on multiple widely used numerical reasoning datasets, includ-ingGSM8K[5],MultiArith[12],ASDiv[13],SVAMP[14],SingleOp[15],andAddSub[16], tovalidatetheeffectivenessofourapproach.”

“As explained by Roy & Roth [12] arithmetic word problems are usually directed towards elementary school students and can be solved by combining the numbers mentioned in the text with basic operations (addition, subtraction, multiplication, division).”

“(Roy and Roth, 2015).”

“The bilingual MWPs dataset is compiled from AddSub (Hosseini et al., 2014), Sin-gleOp (Roy et al., 2015) and MultiArith (Roy and Roth, 2015), containing 1557 elementary school math word problems.”

“Although, the Mathprompter technique was evaluated for MultiArith dataset[31] which is a subset of MATH dataset[32] and used 175B parameter LLM for few-shot and few-shot-CoT settings and 540B parameter LLM(PaLm 540B) for zero-shot and zero-shot-CoT settings but in order to have a fair and direct…”

“Finally, there is also a large body of work that focuses on more specialised reasoning tasks, such as numerical reasoning (Roy and Roth, 2015; Cobbe et al., 2021; Amini et al., 2019).”

“Conventional methods match answer equations with predefined rules [5, 6, 7] or simple semantic mapping models [8, 9, 10] and feature extraction in machine learning to generate answer equations [11-15].”

“There are eight type of word clasess (Faruq et al., 2021; Roy & Roth, 2015).”

“expression tree [4] , equation tree [5] , so the Seq2Tree model was proposed [6] .”

“We also show the results of considering removing brackets, where − / ÷ can not be the children node of + / ∗ , which is the compression considered in Roy and Roth (2015); and Commutative Law, which is the compression considered in Wang et al. (2018a).”

“The existing approaches [4], [5], [6], [7] is the task that takes as input a word problem in the natural language and represents it in a formal language, such as, that translate story problems to system of algebraic equations.”

“[18] used expression trees to represent a problem, which is then converted to equations.”

“Discriminative tasks encompass: factuality (TruthfulQA (Lin et al., 2022)), general knowledge and commonsense (MMLU (Hendrycks et al., 2020), ARC-Challenge (Clark et al., 2018), Com-monsenseQA (Talmor et al., 2019), HellaSwag (Zellers et al., 2019), GPQA (Rein et al., 2024 Generative tasks include: mathematical reason…”

“The field of Arithmetic Reasoning involves more interference based on text-based math problems, known benchmarks like GSM8K (Cobbe et al., 2021) and MultiArith (Roy & Roth, 2016).”

“Mathematical skills are covered by GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), SVAMP (Patel et al., 2021), TabMWP (Kang et al., 2024), MultiArith (Roy et al., 2015).”

“The reasoning performance of these SLMs is assessed on the GSM8K test dataset, as well as ASDiv, SVAMP, and MultiArith.”

“For the ID dataset, we use the MultiArith [44], which consists of Math Word Problems on arithmetic reasoning.”

“This dataset is constructed by merging several public datasets, such as AI2 [30], IL [31], and SingleEQ [32].”

“…finetuing SOTA baselines are obtained from: a : GPT-3 175B finetuned plus an additional 175B verifier (Cobbe et al., 2021), b : Relevance and LCA operation classifier (Roy and Roth, 2015), c : (Pi et al., 2022), d : DeBERTaV3-large + KEAR (Xu et al., 2021), e : UnifiedQA-FT (Khashabi et al., 2020).”

“Roy and Roth [21] solve arithmetic problems with multiple steps and operations by mapping quantities and words to candidate equation trees and selecting the highest probability tree.”

“We evaluate EvoFlow on six public benchmarks covering four domains: (1) math reasoning , GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), and MultiArith (Roy & Roth, 2016); (2) code generation , HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021)); (3) embodied , ALFWorld (Shridhar et al., 2021).”

“…benchmarks from three domains: general reasoning: MMLU (Hendrycks et al., 2021), mathematical reasoning: GSM8K (Cobbe et al., 2021), Multi-Arith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), and AQuA (Ling et al., 2017), and code generation: HumanEval (Chen et al., 2021), following the…”

“Roy and Roth (2016) is composed of a diverse set of multi-step arithmetic word problems that require understanding complex relationships between quantities and performing multiple operations to arrive at the correct numerical answer.”

“The effectiveness of the checkpoints is evaluate on the GSM8K benchmark and additional three datasets: MultiArith (Roy and Roth, 2015), AS-Div (Miao et al., 2020), and SVAMP (Patel et al., 2021).”

“To assess generalization across reasoning distributions, we additionally evaluate on three out-of-domain benchmarks: GSM-Hard (Gao et al., 2023), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2016).”

“For models trained on GSM8K-Aug-NL, we evaluate out-of-domain (OOD) generalization on three mathematical reasoning benchmarks: SVAMP (Patel et al., 2021), GSM-Hard (Gao et al., 2023), and MultiArith (Roy & Roth, 2015).”

“It is trained separately for each of the four backbones and evaluated in domain on the GSM8K test set (Cobbe et al., 2021) and out of domain on SVAMP, GSM-Hard and MultiArith, over which we report an unweighted macro-average.”

“ACED-ALL improves over direct PoT on four benchmarks and matches the saturated result on MultiArith.”

“We report the standard evaluation metric for each dataset, using accuracy for MMLU, GSM8K, MultiArith, SVAMP, and AQuA, and Pass@1 for HumanEval.”

“%) of models trained on GSM8K-Aug, on GSM8K (IID) and on GSM8K-Hard, SVAMP and MultiArith (OOD).”

“Our evaluation suite includes MultiArith (Roy & Roth, 2015) and AddSub (Hosseini et al., 2014) (multi-step arithmetic word problems with numeric answers), GSM8K (grade-school math word problems requiring multi-step reasoning), AQuA (multiple-choice algebraic word problems), SingleEq…”

“On tasks such as MultiArith, where the initial prompt is already quite effective, the gains are more modest as there is less room for optimization.”

“We investigate paraphrase-level overlap between the Orca-Math corpus (Mitra et al., 2024) used to construct our post-training data and four small arithmetic benchmarks initially considered for evaluation: AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), SingleEQ (Koncel-Kedziorski et al., 2015), and SVAM…”

“We evaluate our method on the same six benchmarks (Cobbe et al., 2021; Ling et al., 2017; Roy and Roth, 2015; Patel et al., 2021; Hendrycks et al., 2021; Chen et al., 2021) used by ARG-Designer (Li et al., 2026), summarized in Table 1.”

“Mathematical reasoning performance is evaluated on GSM8K [9], AQuA [27], MAWPS [23], AddSub [15], MultiArith [42], SingleEq [22], and SVAMP [37], while commonsense reasoning is evaluated on BoolQ [6], PIQA [4], Social IQA [44], ARC-Easy and ARC-Challenge [8], OpenBookQA [34], HellaSwag [57], and…”

“We use the pretrained model weights for both backbones and evaluate on 6 mathematical reasoning tasks: MultiArith [58], GSM8K [46], AddSub [59], AQuA [48], SingleEq [60], and SVAMP [50].”

“LLMs have demonstrated remarkable emergent capabilities in complex reasoning tasks Talmor et al. (2018); Roy & Roth (2016).”

“To this end, we consider four math reasoning datasets: AddSub [21], SingleEq [26], MultiArith [42], and SVAMP [40], which are widely used to assess the mathematical reasoning ability of LLMs.”

“The main exception is MultiArith, where IFFR rises again at 27B (Figure 4c), mirroring the non-monotonic pattern noted above and showing that scale reduces but does not eliminate reversion to the learned task prior.”

“We perform in-domain evaluation on the GSM8k test set and out-of-distribution (OOD) evaluation on SVAMP (Patel et al., 2021), MultiArith (Roy & Roth, 2015), and GSM-Hard (Gao et al., 2023).”

“[29] 300 Arithmetic Word Problems Foundational MultiArith [26] 180 Multi-step Arithmetic Foundational MATH500 [25] 500 Advanced Mathematics Hard AMC23 [27] 40 Competition Mathematics Very Hard OlympiadBench [28] 674 Olympiad-level Mathematics Extremely Hard Foundational Arithmetic.”

“GSM8K, SVAMP, MultiArith, Coin Flip, and CommonsenseQA are used as in-domain tasks for supervised fine-tuning and evaluation, whereas MATH-500, BIG-Bench Hard, Strat-egyQA, and ScienceQA are used as out-of-domain benchmarks.”

“Benchmarks: We evaluate ATOM on six benchmarks spanning three domains: General Reasoning (MMLU [10]), Mathematical Reasoning (GSM8K [6], MultiArith [25], SVAMP [22], AQuA [18]), and Code Generation (HumanEval [4]).”

“SVAMP [31] contains 1,000 word problems designed to test robustness to linguistic variation, while MultiArith [32] contains 180 compositional multi-step arithmetic problems.”

“Semantic parsing methods convert textual descriptions into executable expressions or formal logical structures, enabling equation construction and symbolic reasoning (Roy and Roth, 2015).”

“However, existing benchmarks for agent systems focus more on straightforward tasks that are designed for specific domains like arithmetic reasoning [8], tool-use [9], information retrieval in web navigation [10], etc.”

“• MultiArith : Contains 600 multi-step arithmetic problems aimed at assessing a model’s capacity for multi-step reasoning.”

“• MultiArith [2]: 600 multi-step arithmetic word problems from the Math Word Problem Repository.”

“For arithmetic reason - ing, we chose the GSM8K [5], MultiArith [25] and SVAMP [22] datasets.”

“Reasoning: SVAMP [Patel et al., 2021], ASDiv [Miao et al., 2020], and MultiArith [Roy and Roth, 2015] under a strict fit → test protocol with Qwen3-0.”

“The mathematical reasoning set includes GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), and ASDiv (Miao et al., 2020).”

“5-14B (Yang et al., 2025) on Math10K (Hu et al., 2023), and evaluate them on GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy & Roth, 2016), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), and SingleEq (Koncel-Kedziorski et al., 2015).”

“We evaluate our method across two categories of tasks: Mathematical Reasoning Tasks: We use six benchmarks, including MATH500 (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), ASDiv (Miao et al., 2020), MAWPS (Koncel-Kedziorski et al., 2016), MultiArith (Roy & Roth, 2015), and SVAMP (Patel et al., 2021), which tes…”

“…cover two primary domains: i) Mathematical Reasoning, including GSM8K [Cobbe et al. , 2021], MATH [Hendrycks et al. , 2021], and Multi-Arith [Roy and Roth, 2015], which require complex multi-step logical deduction; and ii) Code Generation, including HumanEval [Chen et al. , 2021] and MBPP…”

“…HellaSwag (Zellers et al., 2019), Wino-Grande (Sakaguchi et al., 2021), ARC-easy, ARC-challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018) and three mathematical reasoning tasks – MultiArith (Roy & Roth, 2016), SingleEq (Koncel-Kedziorski et al., 2016) and AQuA (Ling et al., 2017).”

“(Patel et al., 2021; Roy & Roth, 2015; Zhang et al., 2019).”

“We also introduce three additional out-of-domain math reasoning datasets as out-of-domain benchmarks: (1) GSM-Hard (Gao et al., 2022), a modified version of the GSM8k test split by using larger numbers to increase difficulty; (2) SVAMP (Patel et al., 2021) and (3) Multi-Arith (Roy and Roth, 2015) are two simpler grade…”

“Tasks and Datasets In mathematical reasoning, we consider the following six math prob - lem datasets following Hu et al. (2023): GSM8K (Cobbe et al., 2021), MultiArith (Roy & Roth, 2016), SVAMP (Patel et al., 2021), AddSub (Hosseini et al., 2014 ), AQuA (Ling et al., 2017 ) and SingleEq (Koncel-Kedziorski et al., 2015…”

“This simple change in prompt engineering sky-rocketed the accuracy of GPT-3 on several key benchmarks, MultiArith [59]: 17.7% → 78.7%, GSM8K [60]: 10.4% → 40.7%, and AQUA-RAT [61]: 22.4% → 33.5%.”

“E XPERIMENTS AND E VALUATION A. Experiment Setup 1) Datasets : We assess the performance of EvoRL on five publicly available benchmarks across Mathematical Reasoning; GSM8K [38], MATH [39]and MultiArith [40], and Code Generation domains; HumanEval [41] and MBPP [42].”

“MultiArith [20] consists of elementary math problems with multiple steps.”

“Specifically, rule-based methods rely on manually crafted schemas and pattern matching [31], while statistic-based methods leverage traditional machine learning models like SVM to select and complete pre-defined templates [32].”

“Overall, easier datasets with simpler tasks (GSM8K, MultiArith) are less affected by noise, which is reflected in their lower ASR values.”

“Seq2Seq models [6-8] pioneered end-to-end learning (58.3% accuracy), and the GTS architecture achieved 75.6% accuracy with graph-based decoding.”

“This idea of tree-based approaches [15,16,17,18] is to convert the derivation of the math expression into constructing an equivalent tree step by step in a bottom-up manner.”

“We evaluate CHOIR on arithmetic reasoning tasks: GSM8K (Cobbe et al., 2021), SingleEq (Koncel-Kedziorski et al., 2015), and MultiArith (Roy and Roth, 2015), and the commonsense reasoning task: CSQA (Talmor et al., 2019).”

“The included tasks are AddSub (Hos-seini et al., 2014), AQuA (Ling et al., 2017), GSM8K (Cobbe et al., 2021), MAWPS (Koncel-Kedziorski et al., 2016), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021).”

“Multi-Arith (Roy and Roth, 2015) consists of multi-step arithmetic word problems requiring compositional numerical reasoning, GSM8K (Cobbe et al., 2021), which contains high school mathematical word problems, and SVAMP (Patel et al., 2021), a dataset of mathematical problems of fourth grade level,…”

“AQuA (Ling et al., 2017) focuses on multiple-choice algebra questions, MultiArith(Roy and Roth, 2016) requires sequential arithmetic steps, and SVAMP (Patel et al., 2021) evaluates robustness through perturbed math problems.”

“For arithmetic reasoning, we utilized GSM8K (Cobbe et al., 2021) with its multi-step grade school math problems (train: 7.47K, test: 1.32K), SVAMP (Patel et al., 2021) offering varied math word problems (train: 700, test: 300), and Multi-Arith (Roy and Roth, 2015) for problems requiring multiple operations (train: 420…”

“• MultiArith [52]: A dataset composed of multi-step arithmetic word problems, requiring the model to sequence multiple calculations correctly.”

“(1) Mathematical reasoning incorporates 6 representative datasets, namely MultiArith [15], AddSub [7], AQUA [11], SingleEQ [10], SVAMP [14], GSM8K [3].”

“These datasets include ASDiv, which contains diverse math word problems [18], SVAMP, which features math word problems with varying structures [19], and MultiArith, which consists of arithmetic word problems [20].”