Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“• MultiArith (Roy and Roth, 2015): A set of math word problems requiring multiple reasoning steps and operations.”

“For example, a widely-used instruction for reasoning tasks is chain-of-thoughts (CoT) prompting (Kojima et al., 2022), which can improve GPT-3’s math reasoning accuracy from 17.7% to 78.7% (on MultiArith dataset (Roy and Roth, 2016)) by simply appending the instruction “Let’s think step-by-step” to each query.”

“For instance, certain solvers, like the one in (Roy and Roth, 2015), train a relevance classifier to identify which textual numbers are relevant to solving the word problem.”

“With the advancements in natural language processing, math word problem solving has received growing attention in recent years (Roy and Roth, 2015; Mitra and Baral, 2016; Ling et al., 2017; Huang et al., 2018).”

“7% ), MultiArith ( 99 .”

“…et al., 2021], SVAMP [Patel et al., 2021], ASDiv [Miao et al., 2020], AQuA [Ling et al., 2017], MAWPS [Koncel-Kedziorski et al., 2016], MultiArith [Roy and Roth, 2016]) and commonsense reasoning tasks (e.g., CSQA [Talmor et al., 2018], StrategyQA [Geva et al., 2021], HotpotQA [Yang et al., 2018]).”

“…Model for Reasoning Reasoning ability is a critical skill to solve complex problems, such as arithmetic reasoning (Koncel-Kedziorski et al., 2016; Roy & Roth, 2016; Miao et al., 2020; Cobbe et al., 2021b; Dua et al., 2019), logical reasoning (Yu et al., 2020), commonsense reasoning (Bhagavatula…”

“(Roy et al., 2015), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2015), and SimulEq (Kushman et al., 2014).”

“CIFAR-10 and Tiny ImageNet are standard datasets for in-distribution and OOD evaluation in this context.”

“There are two categories of word problems (Roy and Roth 2015).”

“We evaluate models’ performance using eleven benchmarks, including (i) commonsense reasoning (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021), Date Understanding, Sports Understanding (Srivastava et al., 2022)); (ii) arithmetic reasoning (AQUA-RAT (Ling et al., 2017), GSM8K (Cobbe et al., 2021), SVAMP (Pate…”

“Table 10: The few-shot prompt used with LLaMA 7B for MultiArith.”

“Addressing such limitations necessitates an approach that goes beyond conventional methodologies [Rajani et al., 2019] [Ling et al., 2017] [Cobbe et al., 2021a] [Chiang and Chen, 2018] [Amini et al., 2019] [Chen et al., 2019] [Roy and Roth, 2016] [Lu et al., 2023] [Yao et al., 2022].”

“…reasoning; HotpotQA (Yang et al., 2018), BeerQA (Qi et al., 2020), 4 and MuSiQue (Trivedi et al., 2021) for multihop reasoning; GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MultiArith (Roy and Roth, 2015) for mathematical reasoning; CommonsenseQA (CSQA) (Talmor et al., 2019), CSQA2.”

“Recently, Kushman et al. (2014) and Roth (2015) focused on how to solve arithmetic problems without using any pre-defined template.”

“5-turbo, LLaMA-2-7B-Chat, and TigerBot-13B-Chat, across 3 arithmetic datasets: GSM8K (Cobbe et al., 2021a), MAWPS (Roy & Roth, 2016), and SVAMP(Patel et al., 2021) 2 T HE Q UERY -”

“Roy et al. (2015) proposes three types of classifiers to identify different elements of problems.”

“…reasoning, out-of-distribution (OOD) evaluation GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), GSM-Hard (Gao et al., 2023), Multi-Arith (Roy & Roth, 2015), CommonsenseQA (Tal-mor et al., 2019) GitHub SynAdapt (Wang et al., 2025a) Adaptive reasoning, synthetic continuous CoT,…”

“There has been a lot of recent work in automatic math word problem solving (Kushman et al., 2014; Roy et al., 2015; Hosseini et al., 2014; Roy and Roth, 2015).”

“…a comprehensive analysis of UAG’s versatility and effectiveness: • Arithmetic Reasoning: This category includes GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SingleEq (Koncel-Kedziorski et al., 2016), AddSub (Hosseini et al., 2014), SVAMP (Patel et al., 2021), and AQUA (Ling et al.,…”

“…≥ 2 ⋆ GSM-hard (Gao et al., 2022) 1,313 3.25 MATH (Hendrycks et al., 2021) 5,000 ≥ 3 ⋆ AddSub (Hosseini et al., 2014) 395 1 SingleOP (Roy et al., 2015) 562 1 SingleEQ (Koncel-Kedziorski et al., 2015) 508 1.31 MultiArith (Roy and Roth, 2015) 600 2 results in a failure in the verification process.”

“After that, Roy and Roth (2015) proposed an algorithmic approach which could handle arithmetic word problems with multiple steps and operations.”

“For the few-shot prompting, we consider two real-world math tasks that require solving multi-step elementary-level arithmetic word problems: MultiArith [18] and GSM8K [19].”

“…out-of-distribution (OOD) mathematical reasoning benchmarks to assess the OOD generalization capability, including SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), and Multi-Arith (Roy and Roth, 2015) from the Math World Problem Repository (Koncel-Kedziorski et al., 2016).”

“…of an operand token, researchers built hand-crafted features that capture the semantic content of a word, such as the unit of a given number (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Zhou et al., 2015; Upadhyay et al., 2016; Roy and Roth, 2017) or dependency relationship between…”

“Datasets Two math word problems benchmark datasets GSM8K (Cobbe et al. 2021) and MultiArith (Roy and Roth 2015) are adopted in the experiments.”

“algebraic word problems (Ling et al., 2017; Roy and Roth, 2016; Zhang et al., 2019), building novel neural modules to directly learn numerical oper-”

“MultiArith (Roy and Roth, 2015) covers a set of multi-step arithmetic reasoning tasks, while GSM8k (Cobbe et al., 2021) covers primary school-level mathematical problems.”

“For tasks MultiArith, AddSub, SingleEq, and SVAMP, we define the set of operation tokens based on a glossary from Wikipedia, as the operations involved are relatively straight-forward.”

“In the past, the emphasis in mathematical reasoning tasks was primarily on fundamental arithmetic reasoning datasets [44; 45; 46], addressing basic mathematical problems such as AddSub [47].”

“…learning methods (Kushman et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018; Zou and Lu, 2019), semantic parsing meth-ods (Shi et al., 2015; Roy and Roth, 2015; Huang et al., 2017) and deep learning methods (Ling et al., 2017a; Wang et al., 2018b; Liu et al., 2019b; Wang et al., 2017;…”

“…across eight datasets: GSM8K, CommonSenseQA, AddSub, MultiArith, SVAMP, BigBench-Date, StrategyQA, and Coin Flip (Hosseini et al., 2014; Talmor et al., 2018; Arkil et al., 2021; Cobbe et al., 2021; Suzgun et al., 2022; Roy & Roth, 2016; Wei et al., 2022b; Kojima et al., 2022; Weng et al., 2024b).”

“We select Flan-T5 over the original T5 models (Raffel et al., 2019) as our backbone models based on the evidence from previous research (Chung et al., 2024; Fu et al., 2023), which demonstrates that instruction-tuned models like Flan-T5 outperform their pre-trained counterparts in mathematical…”

“Following prior works (Roy and Roth, 2015; Liang et al., 2018), we focus on two commonly-used benchmark datasets for arithmetic word problems, AI2 (Hosseini et al.”

“We include five math reasoning datasets: AddSub (Hosseini et al., 2014), Sin-glEq (Koncel-Kedziorski et al., 2015), MultiArith (Roy & Roth, 2015), SVAMP (Patel et al., 2021), and GSM8K (Cobbe et al., 2021), which are commonly used to evaluate the math reasoning ability of LLMs (Wei et al., 2022;…”

“Sun [115] Arithmetic reasoning (MultiArith [108], GSM8K [33], SVAM [93]) Perez et al. [96] Question-Answering 1.”

“The studies about mathematical AI (math word problems) are divided into three parts: statistical learning-based methodologies [34, 56], traditional machine learning techniques [23, 41, 42] and deep learning-based methods [14, 47].”

“There exist several NLP datasets on math word problems (Amini et al., 2019; Saxton et al., 2019; Roy and Roth, 2015; Hendrycks et al., 2021).”

“Most prior works focus on automatically answering MWPs, e.g., (Li et al., 2019, 2020; Qin et al., 2020; Shi et al., 2015; Wang et al., 2018a; Roy and Roth, 2015; Wu et al., 2020a) instead of generating them (Nandhini and Balasundaram, 2011; Williams, 2011; Polozov et al., 2015; Deane and Sheehan,…”

“MultiArith (Roy and Roth, 2015) includes problems with multiple steps which we listed all the seven types in Table 1.”

“…learning abilities (Brown et al., 2020; Wei et al., 2022; Min et al., 2022) to guide LLMs to generate desired task responses, marking the advent of the prompting era and surpassing the age of the intermediate steps in algorithmic and structured reasoning (Roy and Roth, 2015; Ling et al., 2017).”

“SingleEq and AddSub feature relatively straightforward problems that can be solved without multi-step reasoning, whereas MultiArith, AQUA-RAT, GSM8K, and SVAMP require more intricate, sequential problem-solving.”

“Existing datasets are either scraped from the web and filtered (Kushman et al., 2014; Hosseini et al., 2014; Roy and Roth, 2015; Shi et al., 2015) or crowdsourced (Ling et al., 2017).”

“, 2014], MultiArith [Roy and Roth, 2016], SingleEq [Koncel-Kedziorski et al.”

“The reasoning performance of these SLMs is assessed on the GSM8K test dataset, as well as ASDiv, SVAMP, and MultiArith.”

“tomatically solving math word problems. Specific topics include number word problems (Shi et al., 2015), logic puzzle problems (Mitra and Baral, 2015), arithmetic word problems (Hosseini et al., 2014; Roy and Roth, 2015), algebra word problems (Kushman et al., 2014; Zhou et al., 2015; Koncel-Kedziorski et al., 2015a; R…”

“Arithmetic Reasoning : GSM8K (Cobbe et al., 2021a), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015) and Multi-Arith (Roy & Roth, 2015); Commonsense Reasoning : CSQA (Talmor et al., 2019); Symbolic Reasoning : Last Letter Concatenation (LLC…”

“2) Datasets: Early datasets of MWP solving were designed for rule-based methods and are relatively small [11], [26], [27].”

“These include arithmetic (Cobbe et al., 2021; Hendrycks et al., 2021; Amini et al., 2019; Patel et al., 2021; Miao et al., 2020; Ling et al., 2017; Roy and Roth, 2016), common-sense (Talmor et al., 2019; Geva et al., 2021; Clark et al., 2018), symbolic (Wei et al., 2022b), and table reasoning (Nan…”

“• Math reasoning (GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015)): Arithmetic or word-problem solving.”

“…of research has been done trying to solve math problems, using either manually designed features and rules (Roy et al., 2015; Mitra & Baral, 2016; Roy & Roth, 2016; Upadhyay et al., 2016) or sequence-to-sequence neural networks Wang et al. (2017), but the quantity of numerals is not important in…”

“We also select Multi-Arith [30], SVAMP [26], and AQuA [24], which are diverse in terms of language patterns and problem types, and is used to evaluate the model’s ability to solve mathematical word problems; (3) For code generation , we evaluate on HumanEval [6], which consists of 164 original…”

“Then we do experiments on 5 reasoning datasets 1-Digit-Composite (1DC, 1,640 cases) (Brown, 2020), SVAMP (1,000 cases) (Patel et al., 2021), MultiArith (MA, 600 cases) (Roy and Roth, 2016), TwoHop (3,484 cases) (Biran et al., 2024), and StrategyQA (SQA, 2,290 cases) (Geva et al., 2021).”

“…we also evaluate it on several reasoning tasks in the other domains, Dataset Domain # Samples Answer Type GSM8K [12] Math 1319 Number MultiArith [44] Math 600 Number AddSub [31] Math 395 Number SVAMP [29] Math 1000 Number SingleEq [45] Math 508 Number AQuA [30] Math 254 Option Last Letters [7]…”

“− → − so L.H.S not equal to R.H.S with option B. p ( a ) = a/ 2 and p ( b ) = b/ 2 L.H.S = a/ 2 b/ 2 1 / 2( a b ) Each document in this dataset contains multiple tables and longer unstructured by fine-grained annotations of reasoning processes and supporting facts numerical reasoning.”

“The evaluation is performed on AsDiv (Miao, Liang, and Su 2020), GSM8K (Cobbe et al. 2021), MultiArith (Roy and Roth 2015) and SVAMP (Patel, Bhattamishra, and Goyal 2021), with a selection of 500 instances randomly chosen from each original testset to form the new testsets.”

“Along this line, research works have attempted many methods, such as statistical learning [7], and deep learning models [13].”

“Roy and Roth (2015); Roy et al. (2016) leverage the tree structure of equations.”

“…include a diverse set of benchmarks (Hosseini et al., 2014; Talmor et al., 2018; Arkil et al., 2021; Cobbe et al., 2021; Suzgun et al., 2022; Roy & Roth, 2016; Wei et al., 2022b; Kojima et al., 2022; Weng et al., 2024; Zheng et al., 2023; Dubois et al., 2023): AddSub, AQUA, CommonSenseQA,…”

“Following Li et al. (2023) and Ling et al. (2023), we choose GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), Sin-gleEq (Koncel-Kedziorski et al., 2015), and MultiArith (Roy & Roth, 2016) for evaluation.”

“We evaluate our models on 13 reasoning benchmarks across three different categories: (1) arithmetic reasoning: MultiArith (Roy & Roth, 2015), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQUA-RAT (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021); (2) commonse…”

“Works such has Roy and Roth (2015) and Koncel-Kedziorski et al. (2015) treat arithmetic word problem templates as equation trees and perform efficient tree-search by learning how to combine quantities using textual information.”

“Mathematical reasoning has become a pivotal area within contemporary AI research, posing significant challenges for LLMs and MLLMs. Initially, datasets in this domain targeted elementary algebra[15] and arithmetic word problems[33], which were relatively limited in scope and number.”

“…complex reasoning datasets divided into four classes are used to evaluate the proposed method: arithmetic (SingleEq [63], AddSub [64], MultiArith [65], GSM8K [31], Aqua [66], Svamp [67]), symbolic [68] (Last Letter Concatenation, Coin Flip), common sense (Common-SenseQA [69], StrategyQA [70]),…”

“Numeric expressions play an important role in various tasks, including textual entailment (Lev et al., 2004; Dagan et al., 2013), solving arithmetic problems (Roy and Roth, 2015), numeric reasoning (Roy et al., 2015; Trask et al., 2018) and language modeling (Spithourakis and Riedel, 2018).”

“This category can be further divided into two subtypes: (a) Without understanding (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Huang et al., 2017; Shrivastava et al., 2017), which does not check the entity-attribute consistency between each quantity and the target of the given question; and (b) With understand…”

“, 2016), c: UNITDEP (Roy and Roth, 2017), d: Relevance and LCA operation classifier (Roy and Roth, 2015).”

“…used comprehensive datasets, including GSM8K (Cobbe et al., 2021), GSM-Hard (Gao et al., 2023), Al-gebra (He-Yueya et al., 2023), MultiArith (Roy and Roth, 2015), SingleEQ (Koncel-Kedziorski et al., 2015), SingleOP (Roy et al., 2015), AddSub (Hos-seini et al., 2014), and SVAMP (Patel et al., 2021).”

“Initial datasets addressed basic algebraic (Hendrycks et al., 2021b) and arithmetic (Roy & Roth, 2016) word problems with limited scope.”

“RoyAlg is reported in [23], being a representative algorithm among the multiple algorithms in this approach [20–23].”

“The Contextual Reasoning category contains everyday word problems requiring interpretation of short narratives and arithmetic reasoning, further divided into single-step problems from AddSub (Hosseini et al., 2014; Mishra et al., 2022) and Sin-gleOp (Roy & Roth, 2016) datasets and multi-step problems from GSM8K (Cobbe…”

“For this point, strategies such as simulated annealing can be employed to avoid local optima (see Appendix H). metic (SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2015), Svamp (Patel et al., 2021)), Symbolic (Last Letter Concatenation (Kojima et al., 2022)), Comm…”

“Since the answer format of Multiarith is identical to that of GSM8K (Cobbe et al., 2021), consisting of integer numbers, we directly utilize the generators and reward/value models trained on GSM8K to evaluate this dataset.”

“No Task Zero-Shot Others 1 TOMI1 Among A through C, the answer is Therefore, among A through C, the answer is 2 TOMI2 Among A through C, the answer is Therefore, among A through C, the answer is 3 SocialIQA Among A through C, the answer is Therefore, among A through C, the answer is 4 Coin Flip The answer (Yes or No)…”

“…finetuing SOTA baselines are obtained from: a : GPT-3 175B finetuned plus an additional 175B verifier (Cobbe et al., 2021), b : Relevance and LCA operation classifier (Roy and Roth, 2015), c : (Pi et al., 2022), d : DeBERTaV3-large + KEAR (Xu et al., 2021), e : UnifiedQA-FT (Khashabi et al., 2020).”

“Prominent benchmarks in mathematical reasoning include the grade school math problem GSM8K (Cobbe et al., 2021), math word problems SVAMP (Patel et al., 2021), and two algebraic word problems AQuA (Ling et al., 2017), MultiArith (Roy and Roth, 2016 Commonsense reasoning emerges as a pivotal area.”

“A question example from an arithmetic reasoning dataset [9] can illustrate this: Q : A chef needs to cook 15 potatoes.”

“These approaches have been evaluated on relatively simple one-step tasks, such as mathematical calculations (Roy & Roth, 2016; Cobbe et al., 2021), instruction induction (Honovich et al., 2022), and factual analysis (Wu et al., 2023b).”

“For the ID dataset, we use the MultiArith [44], which consists of Math Word Problems on arithmetic reasoning.”

“Besides, we adopt Byte-Pair Encoding (BPE) (Sennrich, Haddow, and Birch 2016) to tokenize the regular text.”

“Past studies have mainly targeted at primary school level arithmetic word problems (Bobrow, 1964; Charniak, 1969; Kushman et al., 2014; Hos-seini et al., 2014; Shi et al., 2015; Roy and Roth, 2015; Zhou et al., 2015; Koncel-Kedziorski et al., 2015; Mitra and Baral, 2016; Upadhyay et al., 2016).”

“Using language models to evaluate model generations has been a long standing idea [22, 36, 40, 4].”

“…the robustness of our model on three Out-of-Distribution (OOD) datasets: (1) GSM-Hard (Gao et al., 2023), which is a difficult variant of GSM8k with more than 1k test samples, as well as (2) SVAMP (Patel et al., 2021) and (3) MultiArith (Roy and Roth, 2015), which are simpler reasoning datasets.”

“We evaluate CE-Graph on six public benchmarks across three domains: (1) math reasoning (GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), Multi-Arith (Roy & Roth, 2015)); (2) code generation (HumanEval, MBPP) (Chen et al., 2021; Austin et al., 2021); and (3) tool use (GAIA) (Mialon et al., 2023).”

“…arithmetic word problems that require multiple operation steps like addition, subtraction, multiplication, and division to reach the solution (Roy and Roth, 2015); 3) the SVAMP dataset consists of arithmetic word problems up to grade 4 level, which is obtained by modifying existing datasets…”