Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“For instance, (Cobbe et al., 2021; Patel et al., 2021; Roy & Roth, 2016) test arithmetic reasoning, (Rajpurkar, 2016; Joshi et al., 2017) test question-answering, (Hendrycks et al., 2020) tests multi-level knowledge in a diverse set of fields, and (Thakur et al., 2021) test information-retrieval.”

“We focused on arithmetic reasoning tasks, specifically AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), and SVAMP (Patel et al., 2021).”

“[110] text CI,CoT A text datasets for large language model with causal labels MultiArith [193] text QA,AR,CoT an arithmetic datasets with multiple steps and operations CSQA [233] text QA,CR,CoT A commonsense datasets answering with piror knowledge Coin Flip [254] text QA,SR,CoT A symbolic reasoning…”

“Roy [11,12] defined the valid expression: a read-once arithmetic expression as one that makes use of each quantity at most once.”

“Soal cerita dalam matematika merepresentasikan pemecahan masalah (Johnson & Clohessy, 2014) yang memerlukan berpikir tingkat tinggi atau higher order thinking skills (HOTS) (Roy & Roth, 2016).”

“Example mechine comprehension tasks include reading comprehensions [11, 44, 46, 50, 52–54, inter alia], science question answering [12, 49, 55, inter alia], algebra word problems [36, 48, inter alia], geometry problems [51, 57] and pre-university entrance exams [5, 23].”

“It at-tracts much attention recently in the field of natural language processing [21,28,29,36,39,40,47].”

“For AI2 and CC, we simply split the data randomly and for IL we opted to maintain the clusters described in Roy and Roth (2015).”

“Recently, utilizing LLMs as evaluators becomes a natural idea for their remarkable performance across various tasks (Kushman et al., 2014; Roy and Roth, 2016; Bubeck et al., 2023).”

“To evaluate LLMs’ reasoning ability and general ability, we utilize four mathematical datasets, namely GSM8K (Cobbe et al., 2021), Multi-Arith (Roy and Roth, 2015), ASDiv (Miao et al., 2020), and SVAMP (Patel et al., 2021).”

“…et al., 2021; Han et al., 2024; Zhong et al., 2024; Lin et al., 2025a; Yu et al., 2020; Clark et al., 2020; Cobbe et al., 2021; Patel et al., 2021; Roy and Roth, 2015; Miao et al., 2020; Ling et al., 2017; Koncel-Kedziorski et al., 2016; Dua et al., 2019; Mirzaee et al., 2021; Shi et al., 2022;…”

“To test generalization and difficulty scalability, we further evaluate on out-of-domain math datasets, including GSM-Hard [6], SVAMP [53], MultiArith [54], AQUA-RAT [55], and MATH [56].”

“45% ), and also leads on GSM8K and MultiArith.”

“We use four test sets: GSM8K test split which is in-domain, and out-of-domain arithmetic datasets SVAMP (Patel et al., 2021), GSM-Hard (Gao et al., 2023), and MultiArith (Roy and Roth, 2015) to test for transferability under distribution shift.”

“( 3) MultiArith(Roy & Roth, 2016): A collection of 600 problems for multi-step arithmetic math word problems designed to evaluate a model’s ability to perform compositional numerical reasoning over natural language.”

“We evaluate across multiple task domains: general reasoning using MMLU [7]; mathematical reasoning using GSM8k [5], MultiArith [17], AQuA [13], and SVAMP [15]; and code generation using HumanEval [4].”

“…popular mathematical reasoning benchmarks: (1) GSM8K (Cobbe et al., 2021), a challenging dataset of grade school math problems; (2) MultiArith (MA) (Roy and Roth, 2016); (3) SVAMP (Patel et al., 2021); (4) MAWPS (Koncel-Kedziorski et al., 2016); and a logical reasoning benchmark: (5) CoinFlips…”

“At the same time, we used the MultiArith dataset [22] as the test set to systematically verify the effectiveness and generalization performance of the proposed method.”

“For mathematical reasoning, we evaluate performance on six test datasets: GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), Multi-Arith (Roy and Roth, 2016), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), and Sin-gleEq (Koncel-Kedziorski et al., 2015).”

“It contains a total of 1000 test questions. e) MultiArith [22] is a dataset that includes a collection of polynomial arithmetic problems, consisting of a series of questions and corresponding answers.”

“In the experimental section, we assessed the training efficacy using four reasoning datasets: SVAMP[18], GSM8K[6], MultiArith[38], and MATH[22].”

“Our method is evaluated on six English mathematical reasoning benchmarks: AddSub (Hosseini et al., 2014), SingleEQ (Koncel-Kedziorski et al., 2015), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), one Chinese math reasoning benchmark, Math23K (Wang et…”

“We focus on three benchmark datasets: converting natural language to structure queries (NL2Code) – HumanEval (Chen et al., 2021), and reasoning over mathematical problems – Multi-Arith (Roy and Roth, 2015) and GSM8K (Cobbe et al., 2021).”

“To validate the effectiveness of our system, we conduct experiments on six datasets including two arithmetic reasoning tasks (MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2015)), two commonsense reasoning tasks (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021)),and…”

“( Roy and Roth, 2016) used expression trees to solve word problems where the whole solving process is decomposed into multiple classification tasks involving quantities and operations.”

“Tree-based meth-ods (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Roy et al., 2016; Roy and Roth, 2017, 2018) replaced the process of deriving an equation by constructing an equivalent tree structure, step by step, in a bottom-up manner.”

“Roy and Roth [35] developed a theory for expression trees that can be used to represent and evaluate the target arithmetic expressions; they use it to uniquely decompose the target arithmetic problem to multiple classification problems; they then compose an expression tree, combining these with world knowledge through…”

“We evaluate EvoHyper on three categories of benchmarks: general reasoning (MMLU (Hendrycks et al., 2021)), mathematical reasoning (GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017)), and code generation (HumanEval (Chen et al., 2021)).”

“Here, the MAWPS dataset [21] is an extension of [22], [23], [24], [25].”

“Experiments are conducted using the MultiArith dataset [32] to assess challenging mathematical tasks and using the SST-2 test set to assess traditional sentiment analysis tasks.”

“In the datasets used in this paper, the first three arithmetic datasets, i.e. SingleEq, AddSub, and MultiArith, contain relatively simple problem(Chu et al., 2023).”

“…reasoning; HotpotQA (Yang et al., 2018), BeerQA (Qi et al., 2020), 4 and MuSiQue (Trivedi et al., 2021) for multihop reasoning; GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015) for mathematical reasoning; CommonsenseQA (CSQA) (Talmor et al., 2019), CSQA2.”

“Roy et al. [15] proposed the first tree based 50 algorithm for solving arithmetic word problem.”

“…3 main task categories: (1) General Reasoning, using MMLU (Hendrycks et al., 2021); (2) Mathematical Reasoning, using GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), SVAMP (Patel et al., 2021), and AQuA (Ling et al., 2017); and (3) Code Generation, using HumanEval (Chen et al., 2021).”

“On in-distribution tasks (GSM8k-Test, GSM-Hard), C-MTP’s advantage is partic-5 ularly clear, whereas on out-of-distribution tasks (SVAMP, MultiArith) the gap is smaller and more variable.”

“For the base configuration, we evaluate zero-shot performance on four standard arithmetic benchmarks: GSM8K ( 1 , 319 ) [6], GSM-Hard ( 1 , 319 ) [12], SVAMP ( 1 , 000 ) [31], and MultiArith ( 600 ) [33].”

“Neutralization recovers 70% of emotion-broken problems over-all (MultiArith 81%, ARC 70%, GSM8K 66%).”

“Benchmarks We evaluate ST-EVO on 9 benchmarks: MMLU (Hendrycks et al.) for general reasoning, GSM8K (Cobbe et al., 2021), Mul-tiArith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), and AQuA (Ling et al., 2017) for math reasoning, HumanEval (Chen, 2021) and DS-1000 (Lai et al., 2023) for code…”

“Each head is implemented as a two-layer feed-forward network with hidden dimension D , SiLU activation, and output dimension D .”

“MetaMathQA is an augmented combination of the MATH and GSM datasets, and we therefore evaluate the resulting models on out-of-domain benchmarks, including Gaokao-Math-2023 (Liao et al., 2024), Svamp (Patel et al., 2021), MultiArith (Roy & Roth, 2015) and SingleEq(Koncel-Kedziorski et al., 2015).”

“In the realm of arithmetic reasoning, we scrutinize six benchmarks: MultiArith [22], GSM8K [21], AddSub [25], AQuA [23], SingleEq [26], and SVAMP [27].”

“Roy et al. contributed the IL dataset with 1,404 elementary-level arithmetic problems and the CommonCore dataset [92] containing 600 multi-step problems.”

“For OOD setting, we primarly trained GSM8K and eval on other arithmetic datasets, including AddSub [Hosseini et al., 2014], SingleEQ [Koncel-Kedziorski et al., 2015], MultiArith [Roy and Roth, 2016], AQuA [Ling et al., 2017], MAWPS [Koncel-Kedziorski et al., 2016], and SVAMP [Patel et al., 2021].”

“Scheduled Sampling vs. Performance We analyze how different scheduled sampling strategies affect accuracy on the MultiArith dataset (Roy and Roth, 2016).”

“MultiArith inherits the causal relationships extracted the LLM’s explanations as a class, and resulting accuracy is 94.3%, shown in Figure 5 (b).”

“As shown in Table IV, Task 4 is a Fill in the blank dataset, which retrieves questions from an arithmetic question bank [9] [10] [11],and converts them into the format of fill in the blank questions.”

“For instance, in the arithmetic datasets [56]–[58], although much effort has been devoted to discover the relationships between different steps, the non-uniqueness of the reasoning process makes it difficult to transcribe the chains of thought (CoT) into a set of causal variables.”

“, 2010), common sense reasoning (Roy and Roth, 2015; Goldwasser and Zhang, 2016), and many others.”

“We choose multiple datasets across different tasks including arithmetic reasoning (GSM8K Cobbe et al. (2021), Multi-Arith Roy & Roth (2015)), commonsense reasoning (StrategyQA Geva et al. (2021)), and symbolic reasoning (Letter Wei et al. (2022b)).”

“Approach [15] leverages expression trees for arithmetic word problem-solving.”

“Roy and Roth (2015) develop a theory for expression trees to represent and evaluate the target arithmetic expressions of AWPs, and they use it to decompose the target AWP into multiple clas-sification problems uniquely.”

“Roy et al. [29] Proposed the first method using the expression tree structure.”

“Multi-step arithmetic problems are usually addressed to primary students (Powell & Fuchs, 2018; Roy & Roth, 2015).”

“In the second stage, semantic parsing [11, 18, 21, 29, 33] attracted a lot of research interest.”

“[21] proposed the first method using a formula tree.”

“Tree-based approaches[1, 2, 4, 7] leverage the idea of transforming an arithmetic expression to an equivalent tree-structure whereas template-based approach[6] focuses on predicting the appropriate template for the given AWP.”

“in the domain of arithmetic problems (Hosseini et al., 2014; Roy and Roth, 2015; Ling et al., 2017) that can be treated as the test-beds for text editing.”

“, 2017), solving algebra word problems (Hopkins et al., 2017; Hosseini et al., 2014; Kushman et al., 2014; Matsuzaki et al., 2017; Roy and Roth, 2015; Roy et al., 2015) and SAT style geometry problems (Seo et al.”

“Models proposed use both feature engineered machine learning (Hosseini et al., 2014; Kushman et al., 2014; Roy and Roth, 2015; Shi et al., 2015; Zhou et al., 2015; Huang et al., 2016; Huang et al., 2017; Matsuzaki et al., 2017) and deep learning (Ling et al., 2017; Wang et al., 2017; Huang et al.,…”

“RELATED WORK Roy and Roth (2016) introduced a web based tool that solved math reasoning.”

“Four more challenging datasets require multi-step reasoning: MultiArith (Roy and Roth, 2015) (600 math problems), SVAMP (Patel et al., 2021) (1000 varied math problems), GSM8K (Cobbe et al., 2021) (1319 grade-school level problems), and AQuA (Ling et al., 2017) (254 algebraic word problems).”

“Following Li et al. (2026b) we evaluate on MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), MultiArith (Roy & Roth, 2016), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), and HumanEval (Chen et al., 2021), reporting Pass@1 / accuracy.”

“Table 1 presents the main results on one in-domain benchmark (GSM8K-Aug) and three out-of-domain benchmarks (GSM-Hard, SVAMP, MultiArith), with three backbone models from different model families: GPT-2, Qwen2.”

“We evaluate five textual implicit reasoning models— CODI (Shen et al., 2025), SIM-CoT (Wei et al., 2025), PCCoT (Wu et al., 2025), CoLaR (Tan et al., 2025), and RoT (Wang et al., 2026)—across three mathematical reasoning datasets: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), and SVAMP (Patel et al., 20…”

“We evaluate on six arithmetic-reasoning benchmarks: AddSub [Hosseini et al., 2014], Multi-Arith [Roy and Roth, 2016], SingleEq [Koncel-Kedziorski et al., 2015], GSM8K [Cobbe et al., 2021], AQuA [Ling et al., 2017], and SVAMP [Patel et al., 2021].”

“Usage Vol. GSM8K-aug(SFT)[6] 385620 GSM8K-hard(GRPO)[10] 1319 SVAMP(GRPO)[19] 700 OpenMathInstruct-2(GRPO)[27] 69981 GSM8K(Testing) 1319 MultiArith(Testing)[20] 600 SVAMP(Testing) 300”

“[105] built on [101] by introducing Unit De-pendency Graphs (UDGs) that capture relationships between the units of quantities.”

“For targeted arithmetic capabilities, AddSub[14] and MultiArith [15] address multi-step arithmetic reasoning, ASDiv [16] emphasizes linguistic diversity across elementary problem types, and AQuA-RAT [17] extends to algebraic reasoning with over 100K problems paired with natural language rationales.”

“…(Suzgun et al., 2023), with a mean base accuracy 0.70), 2 Mathematical reasoning (including GSM8K (Cobbe et al., 2021), MultiArith datasets (Roy and Roth, 2015), with a mean base accuracy 0.97), 3 Logical reasoning (including boolean expressions, coinflip (Suzgun et al., 2023) datasets,…”

“We evaluate SIGMA against single-agent and multi-agent baselines across six benchmarks: three general reasoning datasets ( MMLU , MMLU-Pro , and GPQA ) and three domain-specific datasets ( GSM8K , MultiArith , and HumanEval ), as shown in Section 4.1.”

“Specifically, we use MMLU [Hendrycks et al.] to evaluate broad-domain reasoning; GSM8K [Cobbe et al., 2021], MultiArith [Roy and Roth, 2015], SVAMP [Patel et al., 2021], and AQuA [Ling et al., 2017] for mathematical reasoning; HumanEval [Chen, 2021] and DS-1000 [Lai et al., 2023] for coding…”

“While MultiArith contains questions need more calculations.”

“MultiArith [13] : comprising multiple-operation arithmetic problems emphasizing combinatorial numerical reasoning.”

“• MultiArith Roy & Roth (2016), a dataset of multi-step arithmetic word problems requiring combination of basic operations such as addition, subtraction, multiplication, and division.”

“…with the math dataset collected by Hu et al. (2023), and subsequently evaluated on six arithmetic reasoning benchmarks, including Mul-tiArith (Roy and Roth, 2015), AddSub (Hos-seini et al., 2014), GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al.,…”

“OOD transfer benchmarks (evaluated on SimCoT without retraining): SVAMP (Patel et al., 2021): arithmetic word problems with varied structure (1K examples); MultiArith (Roy & Roth, 2015): multi-step arithmetic (180 examples); GSM-Hard (Gao et al., 2023): harder variants of GSM8K with larger numbers (1.3K examples).”

“We conduct experiments on seven datasets spanning diverse arithmetic, algebraic, and symbolic reasoning skills: SAT [38], MATH [39], GSM8K [40], MultiA [41], MMLU [42], MAWPS [43], and GPQA [44].”