The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Open accessAbstract
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our agents improve F1 scores on reading comprehension tasks in DROP (Dua et al., 2019) by 13.6 /100”
From Automated Design of Agentic Systems (2024) · cites this paper as Dua et al., 2019
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points.”
MethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“We also add context-specific rules to produce strings occurring in both question and paragraph, and those paragraph strings that are neighbors of question tokens in the GloVe embedding space (Pennington et al., 2014), up to a cosine distance of d .”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“We test four different SQuAD-style reading comprehension models on DROP: (1 (Peters et al., 2018) to the original embeddings (78.7% EM); (4) BERT (Devlin et al., 2018), which recently achieved improvements on many NLP tasks with a novel pre-training technique (84.7% EM).”
MethodInfluentialSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”
MethodA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“…et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that learn to pick up these…”
BackgroundInfluentialKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 2×
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2×
“…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”
BackgroundBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 3×
“…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”
MethodInfluentialTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2×
“…to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018),…”
BackgroundCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”
Background
Cited by
Sorted by citations
Items 201–300 of 427
Ekram, S. M. S., Rahman, A. A., Altaf, M. S., Islam, M. S., Rahman, M. M., Rahman, M. M., Hossain, M. A., & Kamal, A. R. M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.186 · Cited by 13
“To calculate the F1 of multiple span type answers, we followed the DROP (Dua et al., 2019) paper.”
MethodMulti-Step Inference for Reasoning Over Paragraphs
Liu, J., Gardner, M., Cohen, S. B., & Lapata, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.245 · Cited by 18
“Neural Module Networks were originally proposed for visual question answering tasks (Andreas et al., 2016b,a), and recently have been used on several reading comprehension tasks (Jiang et al., 2019; Jiang and Bansal, 2019; Gupta et al., 2020), where they specialize the module functions such as FIND and COMPARE to retr…”
MethodInfluentialSelf-citationImproving QA Generalization by Concurrent Modeling of Multiple Biases
Wu, M., Moosavi, N. S., Rücklé, A., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.74 · Cited by 14
“This includes BioASQ (Wiese et al., 2017), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017).”
MethodNumerical reasoning in machine reading comprehension tasks: are we there yet?
Al-Negheimish, H., Madhyastha, P., & Russo, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.759 · Cited by 12
“In this paper, we focus on numerical reasoning based MRC and investigate DROP (Dua et al., 2019), a recent benchmark designed to measure 1for eg., https://leaderboard.allenai.org/ complex multi-hop and discrete reasoning, including numerical reasoning2.”
BackgroundMethodInfluentialHave You Seen That Number? Investigating Extrapolation in Question Answering Models
Kim, J.-H., Hong, G., Kim, K.-M., Kang, J., & Myaeng, S.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.563 · Cited by 10
“(2019) reveals that NAQANet (Dua et al., 2019) struggles to deal with numbers outside the training range, showing a drop in performance in the extrapolation setting, they simply treat it as one of the failure modes in NAQANet and provide no further analysis on this alarming issue on model reliability.”
BackgroundAssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., & Berant, J. (2024) · arXiv · doi:10.48550/arxiv.2407.15711 · Cited by 11
Retrieval augmented generation for building datasets from scientific literature
Maharana, P. R., Verma, A., & Joshi, K. (2025) · Journal of Physics Materials · doi:10.1088/2515-7639/ade1fa · Cited by 12
Robust Question Answering Through Sub-part Alignment
Chen, J., & Durrett, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.98 · Cited by 10
“, 2018) and DROP (Dua et al., 2019), since these are so far out-of-domain from the perspective of SQuAD that we do not see them as a realistic crossdomain target.”
BackgroundTruth-Conditional Captions for Time Series Data
Jhamtani, H., & Berg-Kirkpatrick, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.55 · Cited by 10
“Prior work has investigated handling of numeracy in question answering datasets (Dua et al., 2019; Andor et al., 2019; Gupta et al., 2020), typically using a predefined set of executable operations or using specific distributions for number prediction (Spokoyny and Berg-Kirkpatrick, 2020; Thawani et al.”
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Ivison, H., Smith, N. A., Hajishirzi, H., & Dasigi, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.576 · Cited by 23
Self-citationSynonym Knowledge Enhanced Reader for Chinese Idiom Reading Comprehension
Long, S., Wang, R., Tao, K., Zeng, J., & Dai, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.329 · Cited by 12
A Survey of Document Grounded Dialogue Systems (DGDS)Preprint
Ma, L., Zhang, W., Li, M., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13818 · Cited by 15
“In MRC, Dua et al. [21] propose a task for discrete reasoning over the content of paragraphs.”
BackgroundExploring and Predicting Transferability across NLP TasksPreprint
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., & Iyyer, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00770 · Cited by 10
“Task SQuAD-1 SQuAD-2 NewsQA HotpotQA BoolQ DROP WikiHop DuoRC-p DuoRC-s CQ ComQA Baseline 26.8 ± 6.0 50.1 ± 0.1 28.8 ± 4.9 23.3 ± 4.0 62.2 ± 0.0 19.4 ± 11.8 25.5 ± 4.7 41.6 ± 1.1 38.9 ± 3.3 30.7 ± 2.5 39.1 ± 1.2 CoLA 27.3 ± 6.0 50.0 ± 0.2 29.3 ± 4.6 23.8 ± 4.2 62.2 ± 0.0 17.0 ± 10.4 25.0 ± 4.5 41.8 ± 1.1 40.1 ± 3.1 31…”
MethodInfluentialKQA Pro: A Large Diagnostic Dataset for Complex Question Answering over Knowledge Base.Preprint
Shi, J., Cao, S., Pan, L., Xiang, Y., Hou, L., Li, J., Zhang, H., & He, B. (2020) · arXiv (Cornell University) · Cited by 10
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Miller, E. (2024) · arXiv · doi:10.48550/arxiv.2411.00640 · Cited by 9
“For instance, DROP[6], QuAC[5], RACE[13], and SQuAD[17] are reading-comprehension evals having multiple related questions about independently selected passages of text, and multilingual evals such as MGSM[18] consist of the same question translated into many languages.”
BackgroundNeural module networks: A review
Fashandi, H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126518 · Cited by 11
User Centric Evaluation of Code Generation Tools (Invited Paper)
Miah, T., & Zhu, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00022 · Cited by 18
“DROP [20] Reading comprehension and arithmetic problems.”
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models
Geva, M., Katz, U., Ben-Arie, A., & Berant, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.646 · Cited by 9
“Data We fine-tune MSEGENBERT on DROP (Dua et al., 2019), a dataset for numerical reasoning over paragraphs, consisting of passage-questionanswer triplets where answers are either spans from the input or numbers that are not in the input.”
MethodInfluentialOPERA: Operation-Pivoted Discrete Reasoning over Text
Zhou, Y., Bao, J., Duan, C., Sun, H., Liang, J., Wang, Y., Zhao, J., Wu, Y., He, X., & Zhao, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.119 · Cited by 9
Self-citationChaining Simultaneous Thoughts for Numerical Reasoning
Shao, Z., Huang, F., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.187 · Cited by 16
Causal Reasoning of Entities and Events in Procedural Texts
Zhang, L., Xu, H., Yang, Y., Zhou, S., You, W., Arora, M., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.31 · Cited by 14
Coverage-based Example Selection for In-Context Learning
Gupta, S., Gardner, M., & Singh, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.930 · Cited by 37
Self-citationOverestimation of Syntactic Representation in Neural Language Models
Kodner, J., & Gupta, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.160 · Cited by 13
“…is already known in relation to these higher-level tasks (e.g. (Poliak et al., 2018; Geva et al., 2019)), and various challenge data sets have been constructed to mitigate it in different ways (Levesque et al., 2011; Chao et al., 2017; Dua et al., 2019; Lin et al., 2019; Dasigi et al., 2019).”
BackgroundContrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification
Mozes, M., Bartolo, M., Stenetorp, P., Kleinberg, B., & Griffin, L. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.651 · Cited by 11
“NLP tasks such as question answering (Wallace et al., 2019; Dua et al., 2019; Bartolo et al., 2020; Khashabi et al., 2020), natural language inference, (Nie et al.”
BackgroundTracing Origins: Coreference-aware Machine Reading Comprehension
Huang, B., Zhang, Z., & Zhao, H. Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.91 · Cited by 10
“, 2018), DROP (Dua et al., 2019), and BeerQA (Qi et al.”
BackgroundMethodInfluentialWeakly-Supervised Questions for Zero-Shot Relation Extraction
Najafi, S., & Fyshe, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.224 · Cited by 9
Weakly Supervised Neuro-Symbolic Module Networks for Numerical ReasoningPreprint
Saha, A., Joty, S., & Hoi, S. C. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11802 · Cited by 9
“Second kind of models are the reasoning-free hybrid models like NumNet (Ran et al. 2019), NAQANet (Dua et al. 2019), NABERT+ (Kinley and Lin 2019) and MTMSN (Hu et al. 2019), NeRd (Chen et al. 2020).”
BackgroundMethodInfluentialFact-driven Logical Reasoning.Preprint
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · Cited by 11
Automated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10
“## An example question from DROP: You will be asked to read a passage and answer a question.”
InfluentialREPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
Jiao, F., Guo, Y., Niu, Y., Ji, F., Li, F.-L., & Nie, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.13 · Cited by 8
“Many challenging benchmarks have been established to examine various forms of reasoning abilities, e.g., multi-hop (Yang et al., 2018), discrete (Dua et al., 2019), and logic reasoning (Yu et al., 2020).”
BackgroundEnsemble ALBERT and RoBERTa for Span Prediction in Question Answering
Bachina, S., Balumuri, S., & Kamath, S. S. (2021) · doi:10.18653/v1/2021.dialdoc-1.9 · Cited by 8
“Dua et al. (2019) experimented on the scenario of multiple answer spans.”
BackgroundEvaluation Paradigms in Question Answering
Rodríguez, P., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.758 · Cited by 12
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 11
Interpretable modular knowledge reasoning for machine reading comprehension
Ren, M., Huang, H., & Gao, Y. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06975-2 · Cited by 8
“Gupta et al. [23] extended the neural module networks to answer compositional questions based on paragraphs of text as the context and performed symbolic reasoning on the self-pruned subset of DROPS [14].”
BackgroundMethodInfluentialUnifying Structure Reasoning and Language Pre-Training for Complex Reasoning Tasks
Wang, S., Wei, Z., Xu, J., Li, T., & Fan, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3325973 · Cited by 19
“However, complex reasoning tasks are far more challenging and diverse, involving further foundation reasoning skills.”
BackgroundCLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding
Takahashi, T., Taniguchi, M., Taniguchi, T., & Ohkuma, T. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5824 · Cited by 9
“…al., 2018; Kwiatkowski et al., 2019) to train and vali-184 date the model as in-domain settings, and unveiled six out of the twelve test datasets 1 (Dua et al., 2019; Lai et al., 2017; Kembhavi et al., 2017; Levy et al., 2017; Saha et al., 2018) to validate the trained model as out-of-domain…”
BackgroundA Few More Examples May Be Worth Billions of Parameters
Kirstain, Y., Lewis, P., Riedel, S., & Levy, O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.72 · Cited by 21
“Each of these datasets contains an additional “quirk” that makes it more challenging than the original SQuAD dataset (Rajpurkar et al., 2016), which popularized the extractive QA format: SQuAD 2 has unanswerable questions, HotpotQA provides multiple passages per question, and DROP contains many arithmetic questions wh…”
MethodA Vietnamese Dataset for Evaluating Machine Reading ComprehensionPreprint
Nguyen, K. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.14725 · Cited by 8
“• QANet : QANet was proposed by Yu et al. (2018) and this model also demonstrated good performance on multiple MRC datasets (Rajpurkar et al., 2016; Dua et al., 2019).”
MethodChallenges in Information-Seeking QA: Unanswerable Questions and Paragraph RetrievalPreprint
Asai, A., & Choi, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11915 · Cited by 8
“Lastly, multi-evidence questions require reasoning over multiple facts such as multi-hop questions (Yang et al., 2018; Dua et al., 2019).”
BackgroundAn MRC Framework for Semantic Role LabelingPreprint
Wang, N., Li, J., Meng, Y., Sun, X., Qiu, H., Wang, Z., Wang, G., & He, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06660 · Cited by 10
“Some datasets such as DROP (Dua et al., 2019) have answers that include multiple spans, and the answers can be obtained by using BIO tagging (Segal et al.”
MethodA Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
AFlow: Automating Agentic Workflow Generation
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10762 · Cited by 359
“For HotpotQA and DROP, we report the F1 Score.”
InfluentialComprehensive Multi-Dataset Evaluation of Reading Comprehension
Dua, D., Gottumukkala, A., Talmor, A., Gardner, M., & Singh, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5820 · Cited by 6
“…in reading comprehension, the task of answering questions about a given passage of text, has seen a huge surge of interest in recent years, with many large datasets introduced targeting various aspects of reading (Rajpurkar et al., 2016; Dua et al., 2019; Dasigi et al., 2019; Lin et al., 2019).”
BackgroundSelf-citationSemantics Altering Modifications for Evaluating Comprehension in Machine Reading
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17622 · Cited by 8
“We highlight that models optimised on H OTPOT QA and DROP’ perform slightly better than models optimised on SQ U AD and N EWS QA’ (on average 20% vs 18% and 17% , respectively).”
MethodInfluentialMTQA: Text‐Based Multitype Question and Answer Reading Comprehension Model
Chen, D., Ma, Z., Lin, W., Ma, J., & Zhu, Y. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/8810366 · Cited by 7
“NAQANet is the first model designed for the DROP corpus.”
BackgroundMethodInfluentialHow Fast can BERT Learn Simple Natural Language Inference?
Lin, Y., & Su, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.51 · Cited by 7
“Corpora such as HOTPOTQA (Yang et al., 2018), Breaking-NLI (Glockner et al., 2018), CommonSense QA (Talmor et al., 2019), DROP (Dua et al., 2019) and ROPES (Lin et al., 2019) have also been created recently to evaluate more diverse and difficult NLI cases.”
MethodUKP-SQUARE: An Online Platform for Question Answering Research
Baumgärtner, T., Wang, K., Sachdeva, R. S., Geigle, G., Eichler, M., Poth, C., Sterz, H., Puerto, H., Ribeiro, L. F. R., Pfeiffer, J., et al. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.2 · Cited by 7
Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks
Cui, R., Hershcovich, D., & Søgaard, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.359 · Cited by 9
Interpretable Proof Generation via Iterative Backward Reasoning
Qu, H., Cao, Y. H., Gao, J., Ding, L., & Xu, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.216 · Cited by 7
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete ReasoningPreprint
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05514 · Cited by 7
“Following Dua et al. (2019), we design a multi-type answer predictor to selectively produce different kinds of answers such as span, count number, and arithmetic expression.”
BackgroundMethodInfluentialExploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.441 · Cited by 20
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 31
Meta-learning for compressed language model: A multiple choice question answering study
Yan, M., & Pan, Y. F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.148 · Cited by 6
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.756 · Cited by 8
“This could improve the reader to answer numerical reasoning (Dua et al., 2019), temporal reasoning (Ning et al.”
BackgroundMAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
Nathani, D., Wang, D., Pan, L., & Wang, W. Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.407 · Cited by 13
Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition
Geva, M., Wolfson, T., & Berant, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00450 · Cited by 11
“We therefore implement rules based on the reasoning skills required in current RC datasets (Yang et al., 2018; Dua et al., 2019).”
MethodInfluentialTowards Interpretable Reasoning over Paragraph Effects in Situation
Ren, M., Geng, X., Qin, T., Huang, H., & Jiang, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.548 · Cited by 6
Reverse Operation based Data Augmentation for Solving Math Word ProblemsPreprint
Liu, Q., Guan, W., Li, S., Cheng, F., Kawahara, D., & Kurohashi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01556 · Cited by 6
“S OLVING Math Word Problems (MWPs) is the task that infers a mathematical expression and the final answer from the natural language description of a math problem, which has been crucial due to its importance of numerically reasoning in natural language (NLP) processing [1], [2].”
BackgroundDAGN: Discourse-Aware Graph Network for Logical ReasoningPreprint
Huang, Y., Fang, M., Cao, Y. H., Wang, L., & Liang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14349 · Cited by 6
“DROP (Dua et al., 2019) and MA-TACO (Zhou et al., 2019) need the models to have numerical reasoning.”
BackgroundTWEAC: Transformer with Extendable QA Agent ClassifiersPreprint
Geigle, G., Reimers, N., Rücklé, A., & Gurevych, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07081 · Cited by 26
“Reasoning RC covers a range of questions that require higher reasoning like arithmetic or comparisons as proposed by Dua et al. (2019).”
BackgroundTeaching Autoregressive Language Models Complex Tasks By DemonstrationPreprint
Recchia, G. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02102 · Cited by 6
“…or reasoning tasks expressed in natural language by including components explicitly designed to execute numerical or logical operations [27]–[32]; this is not the approach we take here, but presumably could be combined with more dataset-focused approaches to achieve improved…”
BackgroundLessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“Alternatively, when a reference answer is known, one can perform string-matching approaches heuristically to determine whether the model’s answer matches the ground truth (Dua et al., 2019; Joshi et al., 2017; Hendrycks et al., 2021).”
EviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading Comprehension
Zhou, Y., Bao, J., Sun, H., Liang, J., Wu, Y., He, X., Zhou, B., & Zhao, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_35 · Cited by 5
Discourse-Aware Graph Networks for Textual Logical Reasoning
Huang, Y., Liu, L., Xu, K., Fang, M., Lin, L., & Liang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3280178 · Cited by 7
“Textual reasoning tasks such as reasoning QA [15], [63], [64], [65], Fact-Checking [66], and natural language inference (NLI) [67], [68] validate systems’ reasoning with multiple schemes and granularity.”
BackgroundImproving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task
Chen, C.-C., Takamura, H., Kobayashi, I., & Miyao, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.4 · Cited by 5
“4 To evaluate the aggregate performance, we average all results as in previous studies (Dua et al., 2019; Mishra et al., 2022), and named this score the Quantitative-101 Score.”
ResultAnalyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
Ho, X., Nguyen, A.-K. D., Sugawara, S., & Aizawa, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.87 · Cited by 5
SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA
Tonglet, J., Reusens, M., Borchert, P., & Baesens, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.837 · Cited by 8
Tan, S. W., Lee, C. P., Lim, K. M., Connie, T., & Alqahtani, A. (2024) · IEEE Access · doi:10.1109/access.2024.3487581 · Cited by 5
“This study leverages the Discrete Reasoning Over the Content of Paragraphs (DROP) dataset [18] as the cornerstone for pretraining.”
MethodShim, H., Lowet, D., Luca, S., & Vanrumste, B. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.29 · Cited by 5
Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems
Xu, J., Zhou, M., He, X., Shi, H., & Zhang, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.542 · Cited by 7
“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility
Gupta, H., Varshney, N., Mishra, S. R., Pal, K. K., Sawant, S. A., Scaria, K. J., Goyal, S., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.30 · Cited by 13
Shao, Z., Shang, L., Liu, Q., & Huang, M. (2021) · doi:10.18653/v1/2021.acl-long.318 · Cited by 5
“Fig 6 compares NeRd predictions on four types of questions from DROP when using different learning methods.”
BackgroundInfluentialCoreference Reasoning in Machine Reading Comprehension
Wu, M., Moosavi, N. S., Roth, D., & Gurevych, I. (2021) · doi:10.18653/v1/2021.acl-long.448 · Cited by 7
“In the first approach, potentially biased examples are discarded from the dataset, either after the dataset creation (Zellers et al., 2018; Yang et al., 2018a; Le Bras et al., 2020; Bartolo et al., 2020), or while creating the dataset (Dua et al., 2019; Chen et al., 2019; Nie et al., 2020).”
BackgroundMethodInfluentialAuto-Grading Comprehension on Reference-Student Answer Pairs using the Siamese-based Transformer
Sayeed, M. A., Gupta, D., & Kanjirangat, V. (2024) · IEEE International Conference for Convergence in Technology (I2CT) · doi:10.1109/i2ct61223.2024.10543346 · Cited by 4
“Span QA type (answers are contiguous spans of text from a given context): DROP [24].”
BackgroundMining Numbers in Text: A Survey
Yoshida, M. W., & Kita, K. (2021) · IntechOpen eBooks · doi:10.5772/intechopen.98540 · Cited by 6
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Chen, Z., & Gao, Q. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.234 · Cited by 4
How Many Answers Should I Give? An Empirical Study of Multi-Answer Reading Comprehension
Zhang, C., Lin, J., Liu, X., Lai, Y., Feng, Y., & Zhao, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.359 · Cited by 6
Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Varshney, N., & Baral, C. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.55 · Cited by 15
Lee, S., Park, E., Seo, D., Jeon, D., Kang, I.-H., & Na, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.177 · Cited by 4
“…passages are provided as a source of evidence for an answer (Joshi et al., 2017; Yang et al., 2018; Rajpurkar et al., 2018; Welbl et al., 2018; Dua et al., 2019; Karpukhin et al., 2020; Zhu et al., 2021b; Pang et al., 2022), realistic questions often need to refer to “heterogeneous” evidences…”
BackgroundEnhancing Quantitative Reasoning Skills of Large Language Models through Dimension Perception
Huang, Y., He, Q., Liang, J., Jiang, S., Xiao, Y., & Chen, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00066 · Cited by 6
“DROP [16] proposes a benchmark that contains a large number of reading comprehension problems requiring numerical symbolic calculation to test the numerical understanding and reasoning ability of models.”
BackgroundSakamoto, T., & Aizawa, A. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.14 · Cited by 4
“In our experiments, we used four datasets, DROP (Wikipedia) (Dua et al., 2019), arXiv (Science Pa-pers) (Spithourakis and Riedel, 2018), FinNum (Financial Tweets) (Chen et al., 2018), and Numeracy-600K (Article Titles) (Chen et al., 2019).”
BackgroundWhat Can a Generative Language Model Answer About a Passage?
Summers-Stay, D. A., Bonial, C., & Voss, C. R. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.7 · Cited by 6
“Though significant caution is needed in interpreting LM results, we see diagnostic value in constructing challenge datasets to evaluate systems’ passage understanding , as carefully constructed by Dua et al. (2019).”
BackgroundQasar: Self-Supervised Learning Framework for Extractive Question Answering
Assem, H., Sarkar, R., & Dutta, S. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671570 · Cited by 5
“• DROP Dataset [35]: Discrete Reasoning over Para-graphs (DROP) dataset is introduced in an attempt to focus more on questions that must perform some reasoning on texts to obtain the correct answer.”
BackgroundSurvey of Machine Reading Comprehension Models and its Evaluation Metrics
Therasa, M., & Mathivanan, G. (2022) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc53470.2022.9754070 · Cited by 4
Xia, Y., Li, F., Liu, Q., Jin, L., Zhang, Z., Sun, X., & Shao, L. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.046 · Cited by 5
Numeric Magnitude Comparison Effects in Large Language Models
Shah, R. C., Marupudi, V., Koenen, R., Bhardwaj, K., & Varma, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.383 · Cited by 9
Tab-CQA: A Tabular Conversational Question Answering Dataset on Financial Reports
Liu, C., Li, J., & Xiong, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.20 · Cited by 7
“Dua et al. (2019) propose a QA dataset DROP with numerical inference-type questions.”
BackgroundDo Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?
Kudo, K., Aoki, Y., Kuribayashi, T., Brassard, A., Yoshikawa, M., Sakaguchi, K., & Inui, K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.98 · Cited by 5
Paired Examples as Indirect Supervision in Latent Decision Models
Gupta, N., Singh, S., Gardner, M., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.466 · Cited by 6
“2020a) designed to answer complex compositional questions on DROP (Dua et al., 2019), a dataset requiring natural language and symbolic reasoning against a paragraph of text.”
InfluentialSelf-citationKey-Elements Graph Constructed with Evidence Sentence Extraction for Gaokao Chinese
Wang, X., Ji, Y., & Li, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60457-8_33 · Cited by 4
Let's Ask Again: Refine Network for Automatic Question GenerationPreprint
Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05355 · Cited by 7
“Our proposed RefNet model consisting of a Preliminary Decoder and a Refinement Decoder with Dual Attention Network outperforms the existing state-of-the-art models on the SQuAD, HOTPOT-QA and DROP datasets.”
InfluentialWhy Machine Reading Comprehension Models Learn Shortcuts?Preprint
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01024 · Cited by 4
“These benchmarks are designed to address challenging features, such as evidence checking in multi-document inference (Yang et al., 2018), co-reference resolution (Dasigi et al., 2019), dialog understanding (Reddy et al., 2019), symbolic reasoning (Dua et al., 2019), and so on.”
BackgroundFQuAD2.0: French Question Answering and knowing that you know nothingPreprint
Heinrich, Q., Viaud, G., & Belblidia, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13209 · Cited by 7
“Others take interest in more complex forms of reasoning: HotpotQA (Yang et al., 2018) contains questions that require reasoning among multiple documents, DROP (Dua et al., 2019) introduces questions requiring discrete reasoning types such as addition, comparison or counting.”
BackgroundLearning to Solve Complex Tasks by Talking to Agents.Preprint
Khot, T., Richardson, K., Khashabi, D., & Sabharwal, A. (2021) · arXiv (Cornell University) · Cited by 4
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.19348 · Cited by 4
Self-citationFinal: Combining First-Order Logic With Natural Logic for Question Answering
Shi, J., Ding, X., Hui, S. C., Yan, Y., Zhao, H., Liu, T., & Qin, B. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3551231 · Cited by 3
“Furthermore, answering a question often requires composing multiple pieces of knowledge to form a conclusive hypothesis [2], [3], [4], [5].”
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features
Hlavnova, E., & Ruder, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.396 · Cited by 3
Discrete Reasoning Templates for Natural Language Understanding
Al-Negheimish, H., Madhyastha, P., & Russo, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2021.eacl-srw.12 · Cited by 3
“The data for this was gathered from the DROP training set in the same way we curated the clean evaluation set, for this step we simply identify the compared entities and delimit them with ‘#’.”
BackgroundMethodInfluentialLong-Tailed Question Answering in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Huang, F., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.351 · Cited by 6
Metadata
- Authors
- Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 2368-2378
- OpenAlex
- W2919420119
- Cited by
- 393
- References
- 53
- Citations by year
- 2026: 20 · 2025: 8 · 2024: 20 · 2023: 60 · 2022: 62 · 2021: 113 · 2020: 78 · 2019: 32
- FWCI
- 27.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- reading comprehension, benchmark, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
