DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

“For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points.”

“We also add context-specific rules to produce strings occurring in both question and paragraph, and those paragraph strings that are neighbors of question tokens in the GloVe embedding space (Pennington et al., 2014), up to a cosine distance of d .”

“We test four different SQuAD-style reading comprehension models on DROP: (1 (Peters et al., 2018) to the original embeddings (78.7% EM); (4) BERT (Devlin et al., 2018), which recently achieved improvements on many NLP tasks with a novel pre-training technique (84.7% EM).”

“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”

“…et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that learn to pick up these…”

“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”

“…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”

“…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”

“…to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018),…”

“, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”

“Common QA tasks, such as SQuAD, [28] and DROP [9], require models to answer reading comprehension queries such as When were the Normans in Normandy? and Which kicker had the most field goals? by extracting a contiguous span of text or performing reasoning over a given passage of text.”

“, 2019; HudsonandManning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al.”

“The R 3 dataset is fully based on DROP with a focus on reasoning.”

“or a table) have been recently gaining increased attention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain (Johnson et al., 2016; Suhr et al., 2017; Talmor and Berant, 2018; Yang et al., 2018a; Suhr et al., 2019; Hudson and Manning, 2019; Dua et al., 2019).”

“Another line of work (Gururangan et al., 2018; Kaushik and Lipton, 2018; Dua et al., 2019; Mc-Coy et al., 2019) shows that much of PLMs’ performance on reasoning tasks is due to statistical artifacts in datasets and does not exhibit true reasoning and generalization capabilities.”

“We fine tune our Pre-traind for Reasoning Model, PReasM, on three RC datasets that require reasoning: DROP (Dua et al., 2019), IIRC (Ferguson et al.”

“The motivation for this task is to demonstrate understanding of a passage of text, using various kinds of questions (Rajpurkar et al., 2016; Joshi et al., 2017; Dua et al., 2019; Amini et al., 2019).”

“DROP (Dua et al., 2019) questions require discrete reasoning over the context paragraphs.”

“To evaluate the fine-tuned LLMs, we follow the benchmark proposed in [9], which consists of MMLU [23], BBH [48], DROP [17], and HumanEval [7] datasets.”

“Dua et al. (2019) exclude questions an existing MRQA system could answer to focus on challenging quantitative reasoning.”

“Numerically Augmented QANet (NAQANet) (Dua et al., 2019) was built by”

“These benchmarks are designed to address challenging features, such as evidence checking in multi-document inference (Yang et al., 2018), co-reference resolution (Dasigi et al., 2019), dialog understanding (Reddy et al., 2019), symbolic reasoning (Dua et al., 2019), and so on.”

“…been used to create benchmarks of various capabilities, including basic reading comprehension (Rajpurkar et al., 2016; Kwiatkowski et al., 2019), assembling information across multiple contexts (Yang et al., 2018), and coreference resolution and numerical reasoning (Dua et al., 2019), among others.”

“M o s t M D L s y m b o l i c m a n i p u l a t i o n , c o m p o s i t i o n a l g e n e r a l i z a - t i o n , m a t h r e a s o n i n g G P T - 3 L a s t L e tt e r C o n c a t e n a t i o n [ 103 ] , S C A N [ 53 ] , G S M 8 K [ 1 7 ] , D R O P [ 21 ] 2022 I C L R 672 F u e t a l .”

“98 DROP (Dua et al., 2019) 48.”

“…2019; Liu et al., 2020; Zhong et al., 2020b; Soleimani et al., 2020) focus on the understanding of natural language, namely, linguistic reasoning, but fail to consider symbolic information, which plays an important role in complex reasoning (Liang et al., 2017; Dua et al., 2019; Chen et al., 2019).”

“Lastly, multi-evidence questions require reasoning over multiple facts such as multi-hop questions (Yang et al., 2018; Dua et al., 2019).”

“The ability to understand and induce numerical knowledge in texts has been studied in several recent works (Dua et al., 2019; Ravichander et al., 2019), including numerical commonsense (Lin et al.”

“Then several widely studied complex reasoning tasks should be integrated, such as reasoning for commonsense knowledge [27, 29, 30], multi-hop relationships [35, 36] and numerical calculation [44, 45] described in § II-A.”

“MASH-QA (Dua et al., 2019) extends the answer space to texts that span across a longer document, but this dataset is highly domain-specific, in the healthcare domain.”

“…work have focused on complementing knowledge-based explanation with operational interpretability, introducing explicit annotation for the decomposition of multi-hop and discrete reasoning questions (Dua et al., 2019) into a sequence of atomic operations (Wang et al., 2020; Wolfson et al., 2020).”

“Popular machine reading datasets such as QuAC (Choi et al., 2018), SQuAD 2 (Rajpurkar et al., 2018) or DROP (Dua et al., 2019) are flagged for >90% contamination.”

“Examples include tasks requiring discrete and arithmetic reasoning (Dua et al., 2019), textbook question answering which requires understanding various forms of knowledge (Clark et al., 2018; Kembhavi et al., 2017) and multi-hop question answering which requires reading multiple distinct pieces of…”

“More challenging evaluation benchmarks have recently been proposed: Dua et al. (2019) released the DROP dataset, for which the annotators were encouraged to provide adversarial questions; Burchell et al. (2020) released the MSQ dataset, consisting of multi-sentence questions.”

“(2019) and Dua et al. (2019), respectively.”

“In contrast to DROP (Dua et al., 2019) where queries are answered over single passages, and bAbI (We-ston et al., 2015), where each query is based on a context of less than 20 facts, our dataset scales from databases of 25 instances to 1000, and could be extended further.”

“Drop [85] tests discrete numerical reasoning over the context.”

“Both the models employ a similar arithmetic block introduced in the baseline model (Dua et al., 2019) on top of contextual representations from BERT (De-vlin et al., 2019).”

“3) All of the adversarial attacks in our AdvRACE benchmark incur a significant drop of models’ performance, which is much more effective than previous attempts (Dua et al., 2019a) in revealing weaknesses of the most competitive MRC models.”

“Khot et al. (2020) learn a generative model that decomposes the task in the language of existing QA models for HotpotQA (Yang et al., 2018) and DROP (Dua et al., 2019).”

“More recently, reading comprehension tasks requiring multi-hop reasoning (Welbl et al., 2018; Yang et al., 2018b), and discrete reasoning (Dua et al., 2019; Amini et al., 2019) are proposed and widely studied.”

“Dua et al. (2019), Nie et al. (2020a), and Bartolo et al. (2020) use models in the loop during data collection, where crowdworkers can only submit examples that cannot be solved by the models.”

“Other datasets focused on specific reasoning in NLU, including numerical reasoning (Amini et al., 2019; Dua et al., 2019; Tafjord et al., 2019; Ravichander et al., 2019), corefer-ential reasoning (Dasigi et al., 2019; Sakaguchi et al., 2020), abductive reasoning based on commonsense (Bhagavatula et…”

“Results from both a simulation study and a 10-person evaluation show ScatterShot improves in-context function performance, as well as annotator’s awareness and handling of diverse patterns.”

“While there are some recent efforts on question answering with more programmatic structures [29] for tasks like DROP [21], these systems are highly specialized to applications like answering numerical questions.”

“…this stronger form of generalization, an emerging line of research has investigated adversarial data collection (ADC), a scheme in which a worker interacts with a model (in real time), attempting to produce examples that elicit incorrect predictions (e.g., Dua et al., 2019; Nie et al., 2020).”

“To answer our questions, one needs numerical reasoning (Dua et al., 2019).”

“…reasoning: AQuA (Ling et al., 2017): A multi-choice dataset of arithmetic questions covering various topics and difficulty levels, and DROP (Dua et al., 2019): A reading comprehension dataset that needs discrete reasoning; • Natural Language Inference: Adversarial NLI subsets (Nie et…”

“There are two main types of MRC task: 1) extractive/abstractive question answering (QA) such as SQuAD (Rajpurkar et al., 2018) and DROP (Dua et al., 2019); 2) multiple-choice QA (MCQA) such as MultiRC (Khashabi et al., 2018) and DREAM (Sun et al., 2019a).”

“…training data was used to train the question generation model and teacher, and 6 outof-domain datasets—BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al.,…”

“So, further efforts were made to create more complex datasets, like HotpotQA[15], DROP[16], and RACE[17], that required more linguistic inferences and a deeper understanding of the text.”

“S OLVING Math Word Problems (MWPs) is the task that infers a mathematical expression and the final answer from the natural language description of a math problem, which has been crucial due to its importance of numerically reasoning in natural language (NLP) processing [1], [2].”

“The out-of-domain set we evaluate on has examples from BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017).”

“Substantial progress in NLP has been made on “closed” tasks, where queries are paired with relevant documents (Rajpurkar et al., 2016; Dua et al., 2019).”

“The dataset has attracted various methods [44, 28, 10, 46, 36, 27, 3], each of which however follow the dataset’s task and treat this as a classification problem.”

“Transformer-based language models have shown promising results on multi-hop or single-hop question answering benchmarks such as HotpotQA (Yang et al., 2018), SQuAD (Rajpurkar et al., 2016), and Drop (Dua et al., 2019).”

“Discrete reasoning over paragraphs (DROP) [79] examples are collected similar to SQuAD, where the question–answer pairs are constructed from Wikipedia paragraphs by crowd workers.”

“…reasoning (MultiRC (Khashabi et al., 2018)), multi-hop reasoning (HotpotQA (Yang et al., 2018)), mathematical and set reasoning (DROP (Dua et al., 2019)), commonsense reasoning (CosmosQA (Huang et al., 2019)), coreference resolution (QuoRef (Dasigi et al., 2019)), and logical…”

“Several other QA datasets have also been recently published accounting for characteristic such as requiring multi-passage or discrete reasoning (Yang et al., 2018; Dua et al., 2019); further, conversational QA datasets have been made available: CoQA (Reddy et al.”

“Following NAQANet (Dua et al., 2019), we use an answer type prediction layer to decide the type of the answer.”

“However, the modern formulation of QA usually assumes that each question has only one answer, e.g. , SQ u AD (Rajpurkar et al., 2016), H otpot QA (Yang et al., 2018), DROP (Dua et al., 2019).”

“(d) A recent dataset, DROP (Dua et al. 2019), requires discrete reasoning over text (such as counting or addition).”

“Knowledge Base Question Answering Comparing to machine reading comprehension (Rajpurkar et al., 2016; Joshi et al., 2017; Shao et al., 2018; Dua et al., 2019; d’Hoffschmidt et al., 2020), KBQA is less diverse in terms of datasets.”

“There is a rich body of recent work showing the benefits of dynamic adversarial data collection in model evaluation (Yang et al., 2017; Dua et al., 2019; Dinan et al., 2019; Nie et al., 2020; Bar-tolo et al., 2020; Kiela et al., 2021; Wallace et al., 2021), although the approach has been challenged…”

“The general trend has been to create large scale datasets for various tasks such as Abduc-tive NLI (Bhagavatula et al., 2019), DROP (Dua et al., 2019), and SWAG (Zellers et al., 2018).”

“SQuAD and DROP use Wikipedia pages as context passages whereas SearchQA (Dunn et al., 2017) uses IR approaches to collect context passages.”

“Setup The eight reading comprehension tasks are from the ORB benchmark (Dua et al., 2019b): DROP (Dua et al., 2019a), DuoRC (Saha et al., 2018), NarrativeQA (Koˇcisky et al., 2017), NewsQA (Trischler et al., 2017), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), SQuAD (Rajpurkar et al.,…”

“…in reading comprehension, the task of answering questions about a given passage of text, has seen a huge surge of interest in recent years, with many large datasets introduced targeting various aspects of reading (Rajpurkar et al., 2016; Dua et al., 2019; Dasigi et al., 2019; Lin et al., 2019).”

“Those focus on prediction consistency (Ribeiro et al., 2019), acquired knowledge (Richardson and Sabharwal, 2019), unanswer-ability (Nakanishi et al., 2018) or multi-dataset evaluation (Dua et al., 2019a) rather than performing an analysis of acquired linguistic or reasoning capabilities.”

“We have put more use cases into the appendix, including DROP (Dua et al., 2019), MATRES (Ning et al., 2018), TORQUE (Ning et al., 2020), VQA-E (Li et al., 2018), and two ongoing projects.”

“This limits them in tasks that require numerical reasoning and possibly even beyond (Dua et al., 2019; Thawani et al., 2021a).”

“Size Domain WebQuestions (Berant et al., 2013) 6.6K Freebase WebQSP (Yih et al., 2016) 4.7K Freebase WebQComplex (Talmor and Berant, 2018) 34K Freebase MuSiQue (Trivedi et al., 2021) 25K Wiki DROP (Dua et al., 2019) 96K Wiki HotpotQA (Yang et al., 2018) 112K Wiki 2Wiki2MultiHopQA (Ho et al., 2020) 193K Wiki Natural-QA…”

“DROP 124 (Dua et al., 2019) is a benchmark MRC dataset 125 requiring mathematical reasoning.”

“The evaluations were performed on three typical data sets, including HotpotQA (Yang et al., 2018), ComplexWebQuestions (Talmor and Berant, 2018), and DROP (Dua et al., 2019).”

“Wallace et al. (2019) has investigated that how will different types of embedding, such as BERT (Devlin et al., 2019) and GloVe (Pennington et al., 2014), affect the performance of the same NAQANet model (Dua et al., 2019) on the same tasks including List Maximum, Decoding and Addition.”

“We experiment on a subset of DROP (Dua et al., 2019) with numeric answers (67% of the entire dataset) focusing on mathematical reasoning.”

“We use NAQANet (Dua et al., 2019) and BiDAF models (Seo et al., 2017).”

“3) All of the adversarial attacks in our AdvRACE benchmark incur a significant drop of models’ performance, which is much more effective than previous attempts (Dua et al., 2019a) in revealing weaknesses of the most competitive MRC models.”

“Adversarial data collection has been widely adopted in recent work, especially for building evaluation datasets (Dua et al., 2019; Nie et al., 2020a; Dinan et al., 2019; Bartolo et al., 2020; Potts et al., 2021; Liu et al., 2021; Kaushik et al., 2021; Xu et al., 2020, 2021).”

“And, generative learning has a great potential for an evaluation [12].”

“…pre-trained models on natural language understanding challenges, including Grade 8 New York Regents science exam (Clark et al., 2019), discrete reasoning over natural language (Dua et al., 2019), reasoning over rules in natural language (Clark et al., 2020), and logical reasoning (Yu et al., 2020).”

“In 2019, once the DROP [21] dataset was released, studies on multi-span extraction emerged one after another.”

“DROP ( Dua et al., 2019 ) (Discrete Reasoning Over the content of Paragraphs) is a large reading comprehension benchmark introduced by Dua et al. (2019) that contains over 96,000 questions extracted from Wikipedia.”

“EM and F1 scores were used as the evaluation metrics. b: DROP DROP [48], which stands for ‘‘Discrete Reasoning Over the Content of Paragraphs,’’ is a crowdsourced dataset covering a variety of categories of Wikipedia pages, particularly those with a high proportion of numbers (sports game summaries…”

“…reaching human performance in standard QA benchmarks (Devlin et al., 2018; Rajpurkar et al., 2016), more challenging datasets have been proposed (Dua et al., 2019) that require background knowledge, commonsense reasoning (Talmor et al., 2019) and visual reasoning (Antol et al., 2015; Zellers et…”

“…text-to-SQL dataset, Spider (Yu et al., 2018), has 10K questions, which is smaller compared to question answering datasets of other types: the DROP dataset with text paragraphs has 97K questions (Dua et al., 2019) and the GQA dataset with images has 22M questions (Hudson and Manning, 2019).”

“This skill is taught by datasets such as DROP (Dua et al., 2019) and until recently, neural module networks have needed a stronger form of supervision to learn it (Gupta et al., 2020).”

“The disentangled reasoning model is built on RoBERTa ∗ , which is a fine-tuned RoBERTa [37] model on DROP [51], 20Q 8 , and BoolQ [52] by Geva et al. [8].”