Solving General Arithmetic Word Problems

“(Hos-seini et al., 2014) mentions the result of running the system of (Kushman et al., 2014) on AI2 dataset, and we report their result here.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“(Hosseini et al., 2014) mentions the result of running the system of (Kushman et al.”

“This has been successfully used to incorporate world knowledge at inference time, as well as getting around the need for large amounts of jointly annotated data for structured prediction (Roth and Yih, 2005; Punyakanok et al., 2005; Punyakanok et al., 2008; Clarke and Lapata, 2006; Barzilay and Lapata, 2006; Roy et al…”

“The system described in (Berant et al., 2014) attempts to automatically answer biology questions, by extracting the structure of biological processes from text.”

“For the IL dataset, we acquired the system of (Roy et al., 2015) from the authors, and ran it with the same fold information.”

“In particular, we use the L + I scheme of CCMs, which predicts structured output by independently learning several simple components, combining them at inference time.”

“There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”

“Our approach is related to work in understanding and solving elementary school standardized tests (Clark, 2015).”

“We evaluate the performance of EoP on 8 datasets: AddSub[34], MultiArith [35], SingleEQ [36], SVAMP [37], GSM8K [38], AQuA [39] and Math [40], OlympiadBench [52].”

“…in research have led to the gradual integration of semantic analysis techniques [3] [4], as well as statistical machine learning approaches [5] [6], aimed at more effectively tackling the challenges *Corresponding author associated with resolving MWPs.Table I illustrates a typical example of…”

“∂ MultiArith: This dataset is a benchmark for evaluating mathematical reasoning abilities, specifically focusing on solving multi-step arithmetic problems [10].”

“A variety of techniques have been employed for solving visual math problems including programmed symbolic methods, large language models with and without tooling[1, 6], and specifically trained models including seq2seq[19], seq2tree[21], RNNs[20], and other deep learning methods[10, 14, 18].”

“The introduction of an Artificial Neuron aims to revolutionize this landscape by providing a mechanism through which LLMs can enhance their reasoning skills [16] [17] [18] [19] [20] [21] [22] [23] [24].”

“[2]; (3) End-to-end deep learning model-based: Wang et al. (2017) developed a technique that uses Sequence-to-Sequence (Seq2Seq) models to convert problems into answer expressions involving numbers *Corresponding author and operators for solving [3]; (4) Relation-set-based: Yu et al. (2017)…”

“In reference [10], the Expression Tree method was proposed, which dissects math problems into various classification problems and constructs a monotone expression tree.”

“Roy et al. [20] proposed the first approach that leverages expression trees to represent expressions.”

“Siswa sekolah dasar pada jenjang kelas © 2022 JNPM (Jurnal Nasional Pendidikan Matematika) p-ISSN 2549-8495, e-ISSN 2549-4937 rendah umumnya masih berada pada tahap berpikir operasional konkrit sehingga pemecahan soal cerita diharapkan terampil pada jenjang ini (Ojose, 2008; Roy & Roth, 2016).”

“BiLSTM over text This results in N hidden state representations In order to obtain the BiLSTM representation, we run two LSTMs in different directions and concatenate the respective hidden states.”

“Students in primary school strengthen the problem-solving skills of mathematical words as a basis for studying algebra in high school [3].”

“If the problem could not be solved by the first module, the problem was solved by decomposition of the arithmetic problem into a series of classification problems [15].”

“On MultiArith with Llama-8B, where the Best-of-N persona baseline is already a high 83.98%.”

“5 illustrates the reasoning performance of the T2DRL algorithm tested on real-world datasets, including MultiArith [36], ARC [37], Strategy QA [38], Commonsense QA [39], SVAMP [40], ASDiv [41], and GSM8K [42], showing that as the number of reasoning paths increases from 5 to 20 in Fig.”

“A RITH (Roy and Roth, 2015).”

“Subsequently, statistical learning and feature engineering were gradually introduced in MWPs. Goldwasser and Roth proposed feedback-based learning algorithms to enhance nat-ural language interpretation [31].”

“Early research in this area focused on mathematical word problems (MWPs), particularly on equation parsing and symbolic logic in arithmetic and algebraic contexts (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015), laying the foundation for logic-based reasoning frameworks.”

“MultiArith (Roy and Roth, 2016) 600 Number GSM8K (Cobbe et al., 2021) 1319 Number AddSub (Hosseini et al., 2014) 395 Number AQuA (Ling et al., 2017) 254 Option SingleEq (Koncel-Kedziorski et al., 2015) 508 Number SVAMP (Patel et al., 2021) 1000 Number MAWPS (Koncel-Kedziorski et al., 2016) 238…”

“This is also related to the field of Arithmetic Reasoning, which is being explored for text-to-text models and is based on math word problems, mostly at grade school level, (Cobbe et al., 2021; Roy & Roth, 2016; Shen et al., 2021) or proofing theorems (Hendrycks et al., 2021).”

“Following Deb et al. (2024) we use GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015); and Jiang et al. (2024) we use AddSub (Hosseini et al., 2014) AQuA (Ling et al., 2017), GAIA (Mialon et al., 2023).”

“It contains a total of 1000 test questions. e) MultiArith [22] is a dataset that includes a collection of polynomial arithmetic problems, consisting of a series of questions and corresponding answers.”

“…(2) GSM8K [6] dataset of high quality linguistically diverse grade school math word problems created by human problem writers, (3) Mul-tiArith [13] dataset consists high-quality question description and formulas, (4) SVAMP [11] a challenge set for elementary-level Math Word Problems and (5)…”

“We conduct a series of experiments on various reasoning benchmarks: arithmetic reasoning: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2016), AddSub (Hosseini et al., 2014), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017) and SingleOp (Wei et al., 2022). common-sense reasoning: CSQA…”

“Early math word problem datasets (Kushman et al., 2014; Roy and Roth, 2015) are relatively small and are not well suited for testing the limits of modern language models.”

“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”

“Earlier, there have been studies in understanding NLP models’ capabilities to solve arithmetic/algebraic questions (Hosseini et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Ling et al., 2017; Roy and Roth, 2018).”

“Compared to the commonly-used datasets, e.g., MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), our dataset presents two distinct challenges: • Resistance to Intuition.”

“Figure 7 compares the accuracy on MultiArith at each batch (m = 30) in this streaming setting (extended version: Figure 11 in the Appendix).”

“Representative benchmarks for arithmetic reasoning include GSM8K (Cobbe et al., 2021), Math (Hendrycks et al., 2021), MathQA (Amini et al., 2019), SVAMP (Patel et al., 2021), AS-Div (Miao et al., 2020), AQuA (Ling et al., 2017), and MAWPS (Roy and Roth, 2015).”

“Models are tested on various MWP benchmarks (Roy & Roth, 2015; Hendrycks et al., 2021).”

“Early on, mathematical reasoning is mostly focused on solving synthetic basic math problems like AddSub (Hosseini et al., 2014) and other arithmetic reasoning datasets (Koncel-Kedziorski et al., 2015; Roy & Roth, 2015; Patel et al., 2021).”

“Notably, Zero-Shot + RCI outperforms Zero-Shot CoT and Few-Shot CoT without any CoT prompting in four tasks except MultiArith.”

“Previous studies typically assess LLM-powered language agents in focused domains: arithmetic reasoning targeting correct solutions (Roy & Roth, 2015; Cobbe et al., 2021; Patel et al., 2021); tool-use evaluating agents’ proficiency in employing tools and reporting results (Li et al., 2023; Xu et…”

“We follow Wei et al. (2022) and consider the same five MWP benchmarks: GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015), ASDiv (Miao et al., 2020), and AQuA (Ling et al., 2017).”

“The major drawback of MP and PP is the significant amount of communication time required between different devices, which can be regarded as memory-efficient but compute-inefficient methods.”

“For mathematical reasoning, we use the Grade-school math problems (GSM8K; Cobbe et al., 2021) and the multi-step arithmetic dataset from (MultiArith; Roy and Roth, 2015).”

“Sur-5 prisingly, CODI surpasses CoT-SFT when using GPT-2 and achieves a higher relative score improvement on LLaMA1b compared to models trained on GSM8k-Aug.”

“So we have chosen two datasets and two CoT methods (auto-CoT and few-shot-CoT) for this investigation: MultiArith (Roy and Roth, 2015) and GSM8K (Cobbe et al., 2021) in GPT-3.”

“In our experiments, we test the performance of AgentPrune on three types of reasoning tasks and the corresponding logically challenging benchmarks: (1) General Reasoning : We opt for MMLU (Hendrycks et al., 2021) dataset; (2) Mathematical Reasoning : We select GSM8K (Cobbe et al., 2021), MultiArith (Roy & Roth, 2016),…”

“In our experiments, we evaluated the performance of EoT across three complex reasoning tasks: (1) Mathematical Reasoning : This involves six datasets, which includes GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SingleEQ (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), AQuA (Ling et al.…”

“Large-scale pre-trained LMs have achieved revolutionary performance on many tasks, such as solving math word problems (Roy & Roth, 2015; Ling et al., 2017; Cobbe et al., 2021) and commonsense reasoning (Talmor et al., 2022; Geva et al., 2021).”

“Math Word Problem: We select two similar benchmarks: GSM8K (Cobbe et al., 2021), Multi-Arith (Roy and Roth, 2015).”

“• GSM8K (Cobbe et al., 2021): GSM8K is a dataset containing graduate school math questions, where each question is collected from the Math World Problem Repository (Roy & Roth, 2015).”

“Table 7 – Continued from previous page Dataset Question Self-Verification MultiArith Q: Luke was organizing his baseball cards in a binder with 3 on each page.”

“Each question is collected from the Math World Problem Repository (Roy and Roth, 2015) with a numerical answer.”

“…twelve datasets used in prior research (Kong et al., 2024; Kojima et al., 2022) categorized in 4 categories: (1) Arithmetic , including MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQUA-RAT (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al.,…”

“As one of the most focused areas in natural language processing, automatic solution of Math word problem (MWP) reflects the capability of artificial intelligence (Roy and Roth, 2015; Zhang et al., 2019; Patel et al., 2021).”

“We evaluate numerical reasoning tasks from (Wei et al., 2023) for a fair comparison between the methods including, GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), AQUA-RAT (Ling et al., 2017), Sin-gleOp and MultiArith subsets from (Roy and Roth, 2016) .”

“Specifically, we selected three mathematical reasoning datasets—SVAMP (Patel et al., 2021), GSM8K (Cobbe et al., 2021), and Multi_Arith (Roy and Roth, 2016)—and one machine translation dataset, CommonMT (He et al., 2020).”

“…complex reasoning datasets divided into four classes are used to evaluate the proposed method: arithmetic (SingleEq [63], AddSub [64], MultiArith [65], GSM8K [31], Aqua [66], Svamp [67]), symbolic [68] (Last Letter Concatenation, Coin Flip), common sense (Common-SenseQA [69], StrategyQA [70]),…”