FinQA: A Dataset of Numerical Reasoning over Financial Data

“For all models, we use the Adam optimizer (Kingma and Ba, 2015).”

“Retriever The full page of the financial report can go beyond 2,000 tokens, which cannot be coped with the current popular QA models (Devlin et al., 2019).”

“, 2019) and RoBERTa (Liu et al., 2019), our proposed approach outperforms all other baselines and achieves an execution accuracy of 65.”

“3703 scales linearly with sequence length, like the Longformer (Beltagy et al., 2020).”

“Financial QA is more challenging than classic QA (Rajpurkar et al., 2018; Yang et al., 2018) because it requires the system to spot relevant information across heterogeneous sources, such as tables and unstructured texts, and then create a numerical reasoning path to connect all the information.”

“Financial QA is more challenging than classic QA (Rajpurkar et al., 2018; Yang et al., 2018) because it requires the system to spot relevant information across heterogeneous sources, such as tables and unstructured texts, and then create a numerical reasoning path to connect all the information.”

“This way the model is able to learn the program structures as patterns from very large-scale data (~40k for MathQA), but may fail on learning the reasoning paths.”

“The performance of using FinBert is no better than BERT-large, mostly because its pre-training corpus is limited (~30M words from news articles).”

“Another kind of QA datasets related to ours is the math word problem datasets, like MaWPS (Koncel-Kedziorski et al., 2016), MathQA (Amini et al.”

“TabMWP (Lu et al., 2023) 22,729 23,059 1,948,166 2.49% TAT-QA (Zhu et al., 2021) 2,199 13,215 283,776 2.18% HiTab (Cheng et al., 2022) 2,500 7,782 351,299 1.80% MultiHiertt (Zhao et al., 2022) 7,619 7,830 267,615 1.37% FinQA (Chen et al., 2021) 5,276 6,251 242,561 0.99% WikiSQL (Zhong et al., 2017) 74,989 86,202 9,680…”

“Financial NLP tasks (Shah et al., 2022) include sentiment analysis (Araci, 2019), named entity recognition (Salinas Alvarado et al., 2015), news classification (Sinha and Khandait, 2020), and question answering (Chen et al., 2021b, 2022).”

“ConvFinQA has multi-turn conversations with questions and answering.”

“Other noteworthy datasets include FinRED [Sharma et al. , 2022], instrumental for information extraction tasks, FINQA [Chen et al. , 2021] for numerical reasoning assessments, and FinRAD [Ghosh et al. , 2021], which is crucial for understanding and identifying financial terms.”

“To construct the dataset, we de-sign a framework to simulate the conversation flow by decomposition and concatenation of the multi-hop questions from the FinQA (Chen et al., 2021) dataset.”

“5 FinQA (Chen et al., 2021) 2,776 Financial reports (PDF) Post-created - - - Yes - 8,281 16.”

“(Zhu et al., 2021; Chen et al., 2021, 2022b) require reasoning based on the table content.”

“In our benchmark, we present 8 basic math operations, including 4 complex operations as [7]: MINUS , ADDITION , DIVISION , MULTIPLY .”

“…University GMAT/GRE NO MathQA (Amini et al., 2019) Arithm/Algebra University AQuA NO MathQA-Python (Austin et al., 2021) Arithm/Algebra University AQuA NO FinQA (Chen et al., 2021c) Finance University CrowdSource NO TAT-QA (Zhu et al., 2021) Finance University CrowdSource NO on our dataset.”

“It is a metric used to evaluate how well the retrieved context aligns with the ground truth answer, which is considered the definitive correct response 4 .”

“, 2022), but also datasets sourced from diverse domains such as FinQA (Chen et al., 2022) which evaluates the numerical reasoning over financial data and Haiku Generation dataset (Scialom et al.”

“We also report results for Codex (Chen et al., 2021a) (Davinci-code-002) on some datasets.”

“ConvFinQA “Inthecontextofthisseriesofinterconnectedfinance-relatedqueriesandtheadditionalinformation providedbythepretext,tabledata,andposttextfromacompany’sfinancialfilings, pleaseprovidearesponsetothefinalquestion.”

“Specifically, HybridQA (Chen et al., 2020), TAT-QA (Zhu et al., 2021), and FinQA (Chen et al., 2021) focus on both textual and tabular data, while MMQA (Talmor et al., 2021) focus on QA over text, tables, and images.”

“…this gap, datasets annotated with intermediate rationales such as logic forms (Tafjord et 2019; Lu et al., 2021a), programs (Amini et al., 2019; Chen et al., 2021c,a; Cao and Xiao, 2022; Chen et al., 2022a), and reasoning graphs (Zhang et al., 2021) have been proposed to train models for…”

“FinQA [56] and Tat-QA [328] present the challenge of numerical reasoning over financial documents (also containing tables).”

“TabMWP (Lu et al., 2023) 22,729 23,059 1,948,166 1.60% TAT-QA (Zhu et al., 2021) 2,199 13,215 283,776 1.40% HiTab (Cheng et al., 2022) 2,500 7,782 351,299 1.15% MultiHiertt (Zhao et al., 2022) 7,619 7,830 267,615 0.88% FinQA (Chen et al., 2021) 5,276 6,251 242,561 0.64% WikiSQL (Zhong et al., 2017) 74,989 86,202 9,680…”

“Some datasets introduced for NER and QA tasks are also designed for numerical reasoning, including: FiNER-139 [Loukas et al. , 2022], FinQA [Chen et al. , 2021], ConvFinQA [Chen et al. , 2022].”

“• FinQA [304]: A dataset designed for numerical reasoning over financial data.”

“For this, two datasets are employed: FinQA [20] and ConvFinQA [21].”

“We leverage the FinQA dataset (Chen et al., 2021b) to improve numerical reasoning capability, we use human annotated data from (Liu et al., 2024) to boost accuracy on contextualized QA, and the wikitablequestions dataset (Pasupat and Liang, 2015) to strengthen the model’s understanding of…”

“We report both the strengths and limitations of ChatGPT and GPT-4 by comparing them with the state-of-the-art domain-specific fine-tuned models in finance, e.g., FinBert (Araci, 2019) and FinQANet (Chen et al., 2022a), as well as the recently pretrained model such as BloombergGPT (Wu et al., 2023).”

“However, for tables with surrounding text, some methods directly extract [13,18,52] or generate answers [7,32] from the tables and their surrounding text without generating a structured query.”

“This document type is informative and very pervasive in real-world businesses, however with only a few prior efforts [7, 50] on auto-understanding them.”

“For more involved tasks, trained contractors can be an alternative to hiring domain experts (Chen et al. 2021).”

“Datasets like FinQA (Chen et al., 2021) and T AB MWP (Lu et al., 2022b) require numerical reasoning to figure out the answer, but they mainly focus on basic arithmetic, while our benchmark requires more advanced quantitative reasoning skills.”

“Existing work on automating financial numerical reasoning focuses on unrealistically specific document snippets (Chen et al., 2021; Zhu et al., 2021).”

“In particular, LLMs are often not evaluated in specific domains, and available domain-specific QA datasets, such as medical (Pal et al., 2022; Jin et al., 2019), financial (Chen et al., 2021), and legal domains (Zheng et al., 2021), typically include questions, answers, and relevant paragraphs.”

“• FinQA (Chen et al., 2021) : Focuses on reasoning tasks for financial data, such as numerical reasoning in financial business.”

“More advance datasets in the financial domain include FinQA [6], LLMWare [27], ConFIRM [8] and TAT-QA [46] among others [7,38,19] that have been prepared for retrieval and or Questions and Answering (Q&A) tasks over snippets of fi-nancial data that includes tabular data, which has allowed methods on…”

“However, Corex enables LLMs to engage in more exhaustive discussions for problems that are challenging to reach a consensus on (e.g., over 10% of ConvFinQA problems requiring more than 3 rounds), a small proportion of problems require more interactions.”

“Formula prediction is to predict a formula for the target cell in a table [ Chen et al. , 2021a ] .”

“• FinQANet 9 (Chen et al., 2021b) A retriever using BERT first retrieves the supporting facts from the document, then a generator combining RoBERTa and LSTM generates the response, which can be either a schema-based program for CQA or a natural language question for CQG. • T5+T AG O P A…”

“This clearly demonstrates the exceptional quality and effectiveness of the G URU dataset in promoting a wide scope of reasoning ability, highlighting its value as a strong foundation for building general-purpose reasoning models.”

“While these datasets like FinQA (Chen et al., 2021) and TAT-QA (Zhu et al., 2021) address specific scopes such as numerical reasoning, they do not touch upon general LFQA tasks.”

“…into multi-choice question answering system and limited to common sense reasoning tasks, lacking more exploration in other reasoning tasks, e.g. arithmetic reasoning (Cobbe et al., 2021; Chen et al., 2021), conversational reasoning (Chen et al., 2022) and symbolic reasoning (Wei et al., 2022).”

“In comparison to the mentioned works, our work offers the following advantages: (1) Comprehensive financial domain benchmark : Our benchmark is highly comprehensive and emphasizes general financial knowledge.”

“(Zhu et al., 2021; Chen et al., 2021, 2022b) require reasoning based on the table content.”

“Earlier work on finance QA centered on static datasets such as FinQA [3] and TAT-QA [34], which require numerical and hybrid reasoning over financial reports.”

“We follow the evaluation metrics in Spreadsheet-Coder (Chen et al. , 2021a): (1) Formula Accuracy, (2) Sketch Accuracy, (3) Range Accuracy measuring the percentage of correctly predicted formulas, formula sketches (formula using place-holder [RANGE] as referenced cells), and formula ranges (only…”

“We use FinQA [20] to evaluate Amazon Nova’s ability to understand financial data.”

“Additionally, datasets like FinQA [12] and VisualMRC [52] present well-structured or segmented content directly, thus sidestepping the intricacies of parsing and retrieval.”

“Avg. rows Tabular QA HybridQA (Chen et al., 2020b) 15.7 TAT-QA (Zhu et al., 2021) 9.5 FinQA (Chen et al., 2021b) 6.26 WikiTableQA (Zhu et al., 2021) 30 Table NLI TabFact (Chen et al., 2020a) 14.5 ToTTo-TNLI (Jena et al., 2022) 35 RAG technique RAG is the most common approach to augment the…”

“DS-1000 (Lai et al., 2023) is a code generation benchmark containing 1000 data science problems spanning seven Python libraries such as NumPy, Pandas, Matplotlib, etc. TableQA Benchmarks are a series of question-answering benchmarks based on structured tables, including WikiTQ (Pasupat & Liang, 2015), HybridQA (Chen e…”

“We employed FinQA introduced by the current paper and Con-vFinQA (Chen et al., 2021, 2022) for numerical understanding task.”

“This annotation, applied across all four datasets, reveals their ability in handling intricate arithmetic tasks.”

“SearchQA (Dunn et al., 2017), NarrativeQA (Koˇcisk`y et al., 2018), QuALITY (Zhu et al., 2020) are reading comprehension datasets over purely textual data, while FinQA (Chen et al., 2021), DocFinQA (Reddy et al., 2024) are introduced in the financial domain.”

“2 NUPA test: a benchmark for number understanding and processing ability In this section, we will introduce our NUPA benchmark from the following four aspects: number representations, tasks, metrics, and result analysis of current LLMs.”

“The licenses permit us to compose, modify, publish, and distribute additional annotations upon the original content.”

“Meanwhile, various domain-specific QA datasets can be adopted to evaluate the model-specific unknown knowledge for each specialized applications, such as medical domain ( e.g. , PubMedQA (Jin et al., 2019) and MIRAGE (Xiong et al., 2024a)), finance domain ( e.g. , TAT-QA (Zhu et al., 2021) and FinQA (Chen et al., 2021…”

“We report both the strengths and limitations of ChatGPT and GPT-4 by comparing them with the state-of-the-art domain-specific fine-tuned models in finance, e.g., FinBert (Araci, 2019) and FinQANet (Chen et al., 2022a), as well as the recently pretrained model such as BloombergGPT (Wu et al., 2023).”

“as they have demonstrated outstanding performance in a 483 wide range of tasks such as machine translation [66], 484 [67], [68], question answering [69], [70], [71], [72], sen485 timent analysis [73], [74], [75], [76], name entity recog486 nition [77], [78], [79], [80], extractive and abstractive 487 document summariz…”

“…al., 2023, 2024; Li et al., 2024; Chen et al., 2023) collected existing financial QA datasets (Thakur et al., 2021; Sinha and Khandait, 2020; Salinas Alvarado et al., 2015; Chen et al., 2021, 2022; Soun et al., 2022) to build benchmarks, thereby assessing LLMs’ understanding of financial knowledge.”

“We note that our zero-shot prompting method for the FinQA (Chen et al., 2021) dataset requires the model to generate custom code and is focused only on the financial domain, and the input to Infotabs (Gupta et al., 2020a) is a different type of table structure unseen in the input data, with different labels for the bo…”

“Tabular mathematical reasoning task aims at answering math questions based on heterogeneous tabular and textual data, which can provide users with insights from tables containing valuable figures (Lu et al., 2023b; Zhu et al., 2021; Chen et al., 2021b).”

“Note that 77.6% of questions in TAT-QA can be solvable without mathematical reasoning and 50.0% of questions in FinQA are not table-must to be answered.”

“Existing datasets such as FinQA [4], TAT-QA [28], and FiQA [14] address specific financial reasoning needs, while more recent corpora like FinanceBench [8] and FinDER [6] support open-domain and retrieval-augmented generation tasks.”

“…large language models (LLMs) have exhibited impressive performance across various domains, such as general, finance, and medical, demonstrating their mathematical, knowledge-based, and common-sense reasoning capabilities (Chen et al., 2021d; Aly et al., 2021; Wang et al., 2021; Lu et al., 2023).”

“Our datasets cover domains including mathematics [27, 51, 52, 53, 54, 55], coding [56, 57, 58], science [36, 59, 60, 61], medicine [62, 63, 64], and finance [65, 66, 67], where each dataset is randomly sampled up to 500 examples without replacement; see more details in Section C.”

“…systems provide access to information (Kwiatkowski et al., 2019) and increase accessibility in a range of domains, from healthcare and health emergencies such as COVID-19 (M¨oller et al., 2020; Morales et al., 2021) to legal queries (Martinez-Gil, 2021) and financial questions (Chen et al., 2021).”

“Following the zero-shot results (Table 8), QA-TNLI performs well as expected in the fine-tuned setting.”

“…categories: Mathematics (AIME, Math500 [38], and LiveMathBench [39]), Code (MBPP [40] and HumanEval [41]), Logic (KO-RBench [42], Knights and Knaves [43], and BBH [44]), Knowledge (ARC Challenge [45], MMLUPro [46], GPQA [47], FinQA [48], and MedQA [49]), and Affective (EmoryNLP [50] and MELD [51]).”

“On the other hand, Text2SQL and tabular data reasoning benchmarks assess the ability of models to parse queries, extract information, and synthesize the retrieved data to formulate responses (Katsogiannis-Meimarakis & Koutrika, 2021; Zhong et al., 2018; Chen et al., 2021b).”

“FinQA FinQA (Chen et al., 2021) is a QA dataset in the financial domain.”

“As to extending the context, several studies try to enable the NDR model to operate on context with semi-structured tabular data and hybrid data (Chen et al., 2020c; Herzig et al., 2020; Chen et al., 2021).”

“The former is based on TAT-QA (Zhu et al., 2021) and involves tabular data, whereas the latter is based on FinQA (Chen et al., 2021b) and involves raw text with tabular data.”

“Existing financial question-answering datasets ( e.g., CodeFinQA, CodeTAT-QA (Krumdick et al., 2024)) focus primarily on evaluating models’ basic concept understanding, precise numerical extraction, and simple calculation abilities within given contexts.”

“This complexity facilitates the structured presentation of detailed information in a flexible manner, making them a common feature in financial reports (Zhu et al., 2021a; Chen et al., 2021) and other professional documents (Wang et al., 2021; Zhong et al., 2020).”

“Recent years have witnessed growing interest in developing QA models for the intelligent understanding of financial tabular and textual data, where discrete reasoning is indispensable as massive numerical content is usually included in the tables or text [3, 4, 7, 11, 34, 37, 38].”

“An example of this is the FiQA [56] dataset, a substantial new dataset featuring Question-Answering pairs focused on financial reports crafted by experts in finance.”

“As LLM capabilities advanced, more domain-specific benchmarks emerged, extending evaluation to fields like science reasoning (ScienceQA) (Lu et al., 2022), financial reasoning (FinQA) (Chen et al., 2022b), and theorem comprehension (The-oremQA) (Chen et al., 2023b).”

“Question-Answering(QA) FiQA-QA (Maia et al., 2018), FinQA (Chen et al., 2021), Con-vFinQA (Chen et al., 2022), FinDER(Choi etal.,2025) Text (financialnews,social media posts, earn-ingsstatements); Tables (S&P 500 mar-kettables) Normalized Dis-counted Cumulative Gain(nDCG), Mean Reciprocal Rank (MRR), Execution Accurac…”

“( ii ) Financial QA-style benchmarks (e.g., FinQA [5], FinanceQA [15]) pre-collect relevant context and bypass open-domain search and tool use, thereby under-assessing agents’ search competence and diverging from analyst workflows [22, 25] (see Table 1).”

“Common benchmarks include FiQA_SA [Cheng et al. , 2023] and FinQA [Chen et al. , 2021b], with evaluation based on prediction accuracy or financial calculation correctness.”

“…various narrowly defined tasks (Liu et al., 2023; Qin et al., 2023; Zhou et al., 2023; Yao et al., 2023); limited economic settings (Shah et al., 2022; Chen et al., 2021; Sinha and Khandait, 2020; Araci, 2019; Akata et al., 2023); and open-world video games (Wang et al., 2023a; Zhu et al., 2023).”

“Recently, intensive studies have been performed on different aspects or types of reasoning such as mathematical reasoning (Patel et al., 2021; Chen et al., 2021b; Cobbe et al., 2021), various kinds of logical reasoning Figure 1: Code prompting converts a natural language problem into a code prompt…”