Solving General Arithmetic Word Problems

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“A recent work (Sadeghi et al., 2015) tries to answer science questions by visually verifying relations from images.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“• Math Word Problem Repository (Koncel-Kedziorski et al., 2016): AddSub (Hosseini et al., 2014): https://www.cs.washington.edu/nlp/arithmetic ; MultiArith (Roy and Roth, 2015), license: CC BY 4.0.”

“Known as Zero-Shot-CoT, this prompting strategy improves the zero-shot performance of InstructGPT on MultiArith (Roy & Roth, 2016) from 17.7 to 78.7 and improves performance on GSM8K(Cobbe et al., 2021) from 10.4 to 40.7.”

“Mathematical Reasoning MATH [284], GSM8k [244], SVAMP [245], MultiArith [328], ASDiv [246], MathQA [329], AQUA-RAT [330], MAWPS [331], DROP [332], NaturalProofs [333], PISA [334], miniF2F [335], ProofNet [336]”

“The proposed method is evaluated on the ten benchmark datasets from three categories of reasoning problems: Arithmetic Reasoning: (1) the GSM8K (Cobbe et al., 2021) dataset of high quality linguistically diverse grade school math word problems created by human problem writers, (2) the SVAMP (Patel et al., 2021) benchm…”

“5 with 20 different datasets covering 7 representative task categories: reasoning (MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AddSub (Hos-seini et al., 2014), AQUA-RAT (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al., 2015), SVAMP (Patel et al., 2021), CSQA (Talmor et al.,…”

“SQuAD [336], BoolQ [337], SQUADv2 [338], DROP [339], RTE [340], WebQA [341], CMRC2017 [342], CMRC2018 [343], CMRC2019 [344], COTE-BD [345], COTE-DP [345], COTE-MFW [345], MultiRC [346], Natural Questions [347], CNSE [301], DRCD [348], DuReader [349], Dureader robust [350], DuReader-QG [349], SciQ [351], Sogou-log [352…”

“As in their work, we consider three tasks: addition, sorting [26] and word algebra problems (WAPs) [27].”

“…have emerged as a significant area of focus, posing considerable challenges for large foundational models, e.g., LLMs and MLLMs. Initially, datasets in this realm are designed to address basic algebraic [26] and arithmetic [50] word problems, which are relatively limited in scope and volume.”

“While our Zero-shot-CoT underperforms Few-shot-CoT with carefully-crafted and task-specific step-by-step examples, Zero-shot-CoT achieves enormous score gains compared to the zero-shot baseline, e.g. from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K with large-scale InstructGPT model (text-davinci-002).”

“It establishes new state-of-the-art results on three math reasoning datasets (GSM8K; Cobbe et al., 2021; MultiArith; Roy & Roth, 2015; MathQA; Amini et al., 2019), a temporal reasoning task (Date Understanding; Suzgun et al., 2022), and the referential game task (Penguins; Suzgun et al., 2022).”

“We can surmise that the heavily patterned nature of the tasks in MultiArith helps the student model to understand what is asked of it, eliciting correct reasoning.”

“, 2015), logic puzzle problems (Mitra and Baral, 2015), arithmetic word problems (Hosseini et al., 2014; Roy and Roth, 2015), algebra word problems (Kushman et al.”

“5 on MultiArith, AddSub, and SingleEq.”

“datasets: MultiArith (Roy and Roth 2015), GSM8K (Cobbe et al.”

“Early works on this task (Hosseini et al., 2014; Kushman et al., 2014; Roy et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015) focus on relatively small datasets consist-”

“In addition, Roy and Roth Roy et al. (2015); Roy and Roth (2016) tries to handle arithmetic problems with multiple steps and operations without depending on additional annotations or predefined templates.”

“M ATH WORD PROBLEMS (MWP) are mathematical exercises or scenarios presented in the form of written or verbal descriptions rather than straight-forward equations in A RITHMETIC .”

“, 2015), logic puzzle problems (Mitra and Baral, 2015), arithmetic word problems (Hosseini et al., 2014; Roy and Roth, 2015), algebra word problems (Kushman et al.”

“Extensive efforts have been made in the domain of math problem solving (Hosseini et al., 2014; Kushman et al., 2014; Roy and Roth, 2015), which aim at obtaining the correct answer to a given math problem.”

“We use the following question ‘Q’ from the MultiArith dataset to demonstrate the problem solving process followed by MathPrompter.”

“Early benchmarks contain simple arithmetic operations (Hos-seini et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Koncel-Kedziorski et al., 2016).”

“In the past, studies have explored the ability of NLP models to solve arithmetic and algebraic problems (Hosseini et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Ling et al., 2017).”

“, 2019), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2016; Miao et al., 2020; Cobbe et al., 2021), logical reasoning (Yu et al.”

“The MultiArith [Roy and Roth, 2015] dataset, which contains multi-step arithmetic problems.”

“Inspired by the concept of Quantity Schema (Roy and Roth, 2015) and Qset (Koncel-Kedziorski et al.”

“Early math word problem datasets are relatively small or limited to a small number of operation steps (Hosseini et al., 2014; Kushman et al., 2014; Roy et al., 2015).”

“…Zhou et al., 2015; Mitra and Baral, 2016; Roy and Roth, 2018) and semantic parsing-based methods (Shi et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Huang et al., 2017) are proposed to tackle this problem, yet they still require considerable manual ∗ The work was done when Lei…”

“Operationally, it combines the published word problem datasets used in (Hosseini et al. 2014; Kushman et al. 2014; Koncel-Kedziorski et al. 2015; Roy and Roth 2015).”

“There are 600 examples in the MultiArith dataset [57], which we divided into [35], we use the standard test set with 254 samples, 300 samples from the standard training set for our training, and 100 samples for the standard validation set for our validation.”

“We evaluate PHP on six datasets (AddSub [26], MultiArith [27], SingleEQ [28], SVAMP [11], GSM8K [12], AQuA [13]) and MATH [14].”

“Difficulty of questions were increased in subsequent datasets (Roy and Roth, 2016), (Upadhyay et al.”

“7% ), MultiArith ( 99 .”

“In previous work, local classifiers (Roy & Roth, 2015; Roy et al., 2015), hand-crafted grammars (Koncel-Kedziorski et al.”

“The Chain-of-thought idea itself is inspired by earlier work where natural language steps are generated for arithmetic reasoning [30, 85], and the use of formal languages for reasoning [3, 20, 22, 121].”

“Koncel-Kedziorski et al. (2015) and Roy and Roth (2015) target math problems that can be solved by one single linear equation.”

“MultiArith (Roy & Roth, 2015), SVAMP (Patel et al., 2021), AddSub (Hosseini et al., 2014) and ASDiv (Miao et al., 2020) are not chosen in this paper because they are relatively simple.”

“In this case, PE2 is able to discover a high quality prompt from scratch that matches with “Let’s think step by step” on MultiArith.”

“…datasets across commonsense question answering, arithmetic reasoning, and natural language understanding/inference: CommonsenseQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021 (Roy and Roth, 2015), RTE (Bentivogli et al., 2009), MNLI (Williams et al., 2018), and SST-5 (Socher et al., 2013).”

“To examine the transferability of the optimized instructions, we also evaluate the instructions optimized for GSM8K on two other mathematical reasoning datasets, i.e., MultiArith (Roy & Roth, 2016) and AQuA (Ling et al., 2017).”

“The idea of tree-based approaches [26], [27], [28], [15] is to transform the derivation of the arithmetic expression to constructing an equivalent tree structure step by step in a bottom-up manner.”

“Similarly, datasets like AddSub (Hosseini et al., 2014) and SingleOp (Roy et al., 2015) MultiArith (Roy and Roth, 2016) are exclusively dedicated to fundamental arithmetic operations: addition, subtraction, multiplication, and division.”

“Ling et al. (2017); Cobbe et al. (2021); Nye et al. (2021) use natural language to generate reasoning steps while Roy & Roth (2016); Reed & De Freitas (2015); Chen et al. (2017); Cai et al. (2017) show that symbolic reasoning may suffice.”

“, 2017], MultiArith [Roy and Roth, 2015] for math problem solving tasks.”

“…G , G-Designer topology G 𝑐𝑜𝑚 auto-encoder (VGAE) framework [34, 84], G-Designer employs a 𝑣 where 𝑓 𝑣 is the encoder-decoder architecture with parameters Θ 𝑣 𝑞 (·) is the encoder module, 𝑝 (·) is the decoder module. low-dimensional latent vector representations H…”

“We start by collecting sample math word problems from existing open-source datasets, namely NumGLUE [26], AddSub [13], ALGES [17], ASDiv [24], DRAW [35], GSM8k [7], MATHQA [2], MultiArith [32], SingeOP [33], SingleEQ [16], and SVAMP [30].”

“The The results are consistent with those hand-craft features for solving math word problems proposed by the prior research (Hos-seini et al., 2014; Roy and Roth, 2015; Roy et al., 2015).”

“Recently, UnitDep (Roy and Roth 2017) augments the scoring function of the expression tree in (Roy and Roth 2015) by further considering the consistence of rate unit associated with the quantities.”

“Broadly speaking, common to the existing approaches (Kushman et al., 2014; Hosseini et al., 2014; Zhou et al., 2015; Shi et al., 2015; Roy and Roth, 2015) is the task of grounding, that takes as input a word problem in the natural language and represents it in a formal language, such as, a system of equations, express…”

“Key datasets for mathematical reasoning are GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and SingleEQ (Koncel-Kedziorski et al., 2015) while code generation evaluations employ HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and DS1…”

“, 2016), c: UNITDEP (Roy and Roth, 2017), d: Relevance and LCA operation classifier (Roy and Roth, 2015).”

“Consequently, there has been a growing interest in developing automated methods to solve math word problems (Kushman et al., 2014; Hosseini et al., 2014; Roy and Roth, 2015).”

“…methods on three out-of-domain math reasoning datasets: (1) GSM-Hard [9], a modified version of GSM8K with approximately 1k test samples featuring larger magnitude numbers, (2) SVAMP [20] and (3) MultiArith [21], two simpler math reasoning datasets with 1k and 600 test samples, respectively.”

“…and Baral, 2016; Roy and Roth, 2018; Zhou et al., 2015; Huang et al., 2016) and semantic parsing-based methods (Shi et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Huang et al., 2017), these methods do not need hand-crafted features and achieve high performance on large datasets.”

“As in their work, we consider three tasks: addition, sorting [26] and word algebra problems (WAPs) [27].”

“The evaluation of (Roy and Roth 2015) was done separately on different types of arithmetic problems.”

“The first research line maps a problem text into logical forms, and then uses the logical forms to obtain the equation (Shi et al., 2015; Roy and Roth, 2015; Huang et al., 2017; Roy and Roth, 2018).”

“Tables 11, 17, and 18 illustrate examples from the skill exemplar repositories for the GSM-8k and MATH datasets respectively created using GPT-4-0613 .”

“Much work has been done in solving arithmetic (Mitra and Baral, 2016; Hosseini et al., 2014; Roy and Roth, 2016), geometric (Seo et al., 2015), and algebraic problems (Zhou et al., 2015; Koncel-Kedziorski et al., 2015; Upadhyay et al., 2016; Upadhyay and Chang, 2016; Shi et al., 2015; Kush-man et…”

“Although these models exhibit surprisingly impressive capabilities in complex arithmetic reasoning tasks, such as MultiArith (Roy and Roth, 2015) and GSM8k (Cobbe et al.”

“I n s t r u c t G P T , P a L M S i n g l e E q [ 49 ] , A dd S u b [ 38 ] , M u l t i A r i t h [ 82 ] , A Q U A R A T [ 58 ] , G S M 8 K [ 17 ] , S V A M P [ 71 ] , L a s t L e tt e r C o n c a t e n a t i o n [ 103 ] , C o i n F l i p [ 103 ] , C o mm o n s e n s e Q A [ 91 ] , S t r a t e g y Q…”

“As a result we also consider the “reverse operation” for division and subtraction (Roy and Roth, 2015).”

“This significantly improves the performance on arithmetic [26], commonsense [21, 27], and symbolic reasoning [5] benchmarks.”

“In the future, more sophisticated trial-and-error processes and feedback mechanisms will be necessary to teach the LMs to better utilize tools and even create new tools.”

“In this section, we share AlexaTM 20B performance on MultiArith (Roy & Roth, 2016) dataset.”

“Roy and Roth (2015); Koncel-Kedziorski et al. (2015) assume there is only one unknown variable in the problem and cannot derive math concepts involving constants or more than one unknown variables, such as “ the product of two unknown numbers ”.”

“Table 7 – Continued from previous page Dataset Question Self-Verification MultiArith Q: Luke was organizing his baseball cards in a binder with 3 on each page.”

“We demonstrate the feasibility and effectiveness of DTV by reaching state-of-the-art performance on GSM8K, subsets of MATH and MultiArith datasets.”

“MultiArith (Roy and Roth, 2015), SVAMP (Patel et al., 2021), AddSub (Hosseini et al., 2014) and ASDiv (Miao et al., 2020) are not chosen in this paper because they are relatively simple.”

“This dataset of this task contains 300 training instances and was introduced in [32].”

“Our work also relates to solving arithmetic word problems (Hoss-eini et al., 2014; Mitra and Baral, 2016; Zhou et al., 2015; Upadhyay et al., 2016; Huang et al., 2017; Kushman et al., 2014a; Koncel-Kedziorski et al., 2015; Roy and Roth, 2016; Roy, 2017; Ling et al., 2017).”

“We evaluate the effectiveness of Corex across four types of reasoning tasks: (1) Arithmetic reasoning over eight mathematical problems, which includes GSM8K (Cobbe et al., 2021), MultiArith (Roy & Roth, 2015), SingleOP/SingleEQ (Koncel-Kedziorski et al., 2016), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017),…”

“We consider the following arithmetic and logical reasoning tasks: AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2016), GSM-8K (Cobbe et al., 2021), SingleEq (Koncel-Kedziorski et al., 2015), CSQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021).”

“, 2017; Ughade and Kumbhar, 2019) Most existing MWP datasets have focused on returning the solutions of elementary arithmetic problems (Roy and Roth, 2015; KoncelKedziorski et al., 2016) and algebra problems (Kushman et al.”

“Early statistical learning methods (Feigenbaum et al., 1963; Fletcher, 1985; Bakman, 2007; Roy and Roth, 2016) extracted templates or features from problems and generated corresponding math expressions based on these templates or features.”

“The technique was found to be particularly beneficial in solving arithmetic problems (Roy & Roth, 2016).”

“Current MWP corpora can be classified into four categories: (1) the Number Word Problem corpus (Shi et al., 2015), which contains number word problems only; (2) the Arithmetic Word Problem corpora (Hosseini et al., 2014; Roy et al., 2015), which involve the four basic arithmetic operations”

“Early benchmarks contain simple arithmetic operations (Hos-seini et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Koncel-Kedziorski et al., 2016).”

“…(Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDiv (Miao et al., 2020), AQuA (Ling et al., 2017), AddSub (Hosseini et al., 2014), Multi-Arith (Roy and Roth, 2015), SingelEQ (Roy et al., 2015), three commonsense reasoning benchmarks (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021),…”

“…ranging from rule-based meth-ods (Bakman, 2007; Yuhui et al., 2010), statistical machine learning methods (Kushman et al., 2014; Zhou et al., 2015; Roy and Roth, 2015, 2016; Mitra and Baral, 2016; Huang et al., 2016; Roy and Roth, 2018), semantic parsing methods (Shi et al., 2015;…”

“In recent years, mathematical reasoning benchmarks have closely evolved alongside LLMs, covering almost every domain from basic math word problems (Roy & Roth, 2015; Cobbe et al., 2021) to complex multidisciplinary calculations and proofs (Hendrycks et al., 2021; Chen et al., 2023), and even to…”

“To further examine generalization beyond the training domain, we include three out-of-domain benchmarks: GSM-Hard, which contains more challenging arithmetic problems with subtle variations in reasoning steps; MultiArith, which evaluates performance on multi-step arithmetic operations requiring careful sequencing of a…”

“The math reasoning tasks includes a total of 6 benchmarks: AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), gsm8k (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), SingleEQ (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021).”

“The datasets include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GSM-Hard (Gao et al., 2023), SVAMP (Patel et al., 2021), Asdiv (Miao et al., 2020), AddSub (Hos-seini et al., 2014), SingleEQ (Roy et al., 2015), SingleOP (Roy et al., 2015) and MultiArith (Roy and Roth, 2015).”

“8% on MultiArith (Roy and Roth, 2015).”

“Specifically, they are SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), Mul-tiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AQUA-RAT (Ling et al., 2017), and SVAMP (Patel et al., 2021), which are standard datasets widely used in the community.”

“…are commonly used in evaluating CoT reasoning ability (Zheng et al., 2023; Wang et al., 2023): (i) AddSub (Hosseini et al., 2014), (ii) Multi-Arith (Roy and Roth, 2015), (iii) SingleEQ (Kon-cel-Kedziorski et al., 2015), (iv) SVAMP (Patel et al., 2021), (v) GSM8K (Cobbe et al., 2021), (vi) AQuA…”

“When trained on the more challenging AQuA dataset, it achieves an average performance improvement of 1.59 over the SOTA method, while for the simpler MultiArith dataset, the average improvement is 0.81.”