The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Open accessAbstract
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our agents improve F1 scores on reading comprehension tasks in DROP (Dua et al., 2019) by 13.6 /100”
From Automated Design of Agentic Systems (2024) · cites this paper as Dua et al., 2019
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points.”
MethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“We also add context-specific rules to produce strings occurring in both question and paragraph, and those paragraph strings that are neighbors of question tokens in the GloVe embedding space (Pennington et al., 2014), up to a cosine distance of d .”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“We test four different SQuAD-style reading comprehension models on DROP: (1 (Peters et al., 2018) to the original embeddings (78.7% EM); (4) BERT (Devlin et al., 2018), which recently achieved improvements on many NLP tasks with a novel pre-training technique (84.7% EM).”
MethodInfluentialSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”
MethodA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“…et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that learn to pick up these…”
BackgroundInfluentialKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 2×
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2×
“…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”
BackgroundBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 3×
“…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”
MethodInfluentialTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2×
“…to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018),…”
BackgroundCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”
Background
Cited by
Sorted by citations
Items 301–400 of 427
Learning from Limited Labels for Long Legal Dialogue
Hong, J., Chong, D., & Manning, C. D. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.20 · Cited by 3
“Some passages require numerical abilities which smaller language models tend to find difficult (Dua et al., 2019).”
BackgroundKovatchev, V., Chatterjee, T., Govindarajan, V. S., Chen, J., Choi, E., Chronis, G., Das, A., Erk, K., Lease, M., Li, J. J., et al. (2022) · doi:10.18653/v1/2022.dadc-1.5 · Cited by 7
Easy, Reproducible and Quality-Controlled Data Collection with CrowdaqPreprint
Ning, Q., Wu, H., Dasigi, P., Dua, D., Gardner, M., Logan, R. L., Marasović, A., & Nie, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06694 · Cited by 3
“We have put more use cases into the appendix, including DROP (Dua et al., 2019), MATRES (Ning et al., 2018), TORQUE (Ning et al., 2020), VQA-E (Li et al., 2018), and two ongoing projects.”
BackgroundMethodInfluentialSelf-citationEncoding Explanatory Knowledge for Zero-shot Science Question AnsweringPreprint
Zhou, Z., Valentino, M., Landers, D., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05737 · Cited by 3
“Contemporary Question Answering (QA) is evolving in the direction of addressing more abstractive reasoning tasks (Thayaparan et al., 2020; Dua et al., 2019; Clark et al., 2018; Mihaylov et al., 2018), supported by multi-hop inference (Khot et al., 2020; Yang et al., 2018) and explanatory scientific…”
BackgroundGödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Yin, X., Wang, X., Pan, L., Lin, L., Wan, X., & Wang, W. Y. (2024) · arXiv · doi:10.48550/arxiv.2410.04444 · Cited by 2
More with Less: ZeroQA and Relevant Subset Selection for AI2 Reasoning Challenge
Pătras, C. C.-B., Pîrtoacă, G.-S., Rebedea, T. E., & Ruşeţi, Ş. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.09.334 · Cited by 2
A survey on mathematical reasoning and optimization with large language models
Forootani, A., Aliabadi, D. E., & Thrän, D. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200712 · Cited by 29
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. D., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 19
Locke’s Holiday: Belief Bias in Machine Reading
Søgaard, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.649 · Cited by 2
“Numerically Aware QANet Our last model is an extension of the QANet architecture (Yu et al., 2018), presented in Dua et al. (2019).”
MethodInfluentialStudy of a lifelong learning scenario for question answering
Echegoyen, G., Rodrigo, Á., & Peñas, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118271 · Cited by 4
Help Transformer Improve Performance in Automatic Mathematics Word Problem-Solving
Liu, D., Wang, G., & Yang, J. (2022) · IEEE Access · doi:10.1109/access.2022.3220777 · Cited by 2
“Some more challenging datasets and models have been proposed [6], [14], and it is quite natural for researchers to apply semantic parsing techniques to these numerical reasoning problems over numbers [15], [16], [17].”
BackgroundComplex Reasoning in Natural Languag
Zhao, W., Geva, M., Lin, B. Y., Yasunaga, M., Madaan, A., & Yu, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.2 · Cited by 5
Best-k Search Algorithm for Neural Text Generation
Xu, J., Xiong, C., Savarese, S., & Zhou, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.692 · Cited by 4
Vulnerability detection based on transformer and high‐quality number embedding
Cao, Y., Dong, Y., & Peng, J. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8292 · Cited by 5
Dataset for Evaluation of Mathematical Reasoning Abilities in Russian
Nefedov, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-59082-6_10 · Cited by 2
Understanding Unnatural Questions Improves Reasoning over Text
Guo, X., Li, Y.-F., & Haffari, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.434 · Cited by 2
“To further evaluate the projection model, we provide NMNs with classifier-based projection results to evaluate the performance on the downstream CQA task on the DROP dasatet (Dua et al., 2019).”
BackgroundMethodInfluentialImproving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text
Guo, X., Li, Y.-F., & Haffari, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.231 · Cited by 2
NQuAD: 70,000+ Questions for Machine Comprehension of the Numerals in Text
Chen, C.-C., Huang, H., & Chen, H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482155 · Cited by 3
“Some explorations have been done in current works [11, 20].”
BackgroundVarghese, N., & Punithavalli, M. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-16-9113-3_1 · Cited by 2
Domain Incremental Lifelong Learning in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Yu, B., Huang, F., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.361 · Cited by 2
FERMAT: An Alternative to Accuracy for Numerical Reasoning
Sivakumar, J., & Moosavi, N. S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.838 · Cited by 2
20Q: Overlap-Free World Knowledge Benchmark for Language Models
Bruyn, M. D., Lotfi, E., Buhmann, J., & Daelemans, W. M. P. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.46 · Cited by 2
“…Huang et al., 2019; Ning et al., 2020), physical and social intelligence (Bisk et al., 2020; Sap et al., 2019), temporal reasoning (Zhou et al., 2019), numerical knowledge (Dua et al., 2019; Ravichander et al., 2019), and global commonsense reasoning (Singh et al., 2021; Talmor et al., 2022, 2019).”
BackgroundArithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models
Petrak, D., Moosavi, N. S., & Gurevych, I. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.42 · Cited by 2
“We focus on BART (Lewis et al., 2020) for this experiment and perform Arithmetic-Based Pre-training on a different dataset before finetuning on DROP (Dua et al., 2019) and InfoTabs (Gupta et al., 2020).”
BackgroundMethodInfluentialOperation-Augmented Numerical Reasoning for Question Answering
Zhou, Y., Bao, J., Wu, Y., He, X., & Zhao, T. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3316448 · Cited by 2
“Two examples from the DROP [1] and TAT-QA datasets [2], respectively, consist of question-answer pairs and background knowledge.”
BackgroundMethodInfluentialChain-of-Questions Training with Latent Answers for Robust Multistep Question Answering
Wang, Z., Thomason, J., & Jia, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.547 · Cited by 6
Weak Supervision for Question and Answering Sentiment Analysis
Tomita, V. A. K., Lobato, F. M. F., & Marcacini, R. M. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00287 · Cited by 3
“…the annotation process; IV); QED [31] designed to enhance the interpretability of questions and answers; V) DuoRC [32] generated by extracting questions and answers from different contexts discussing the same subject; and VI) DROP [33] that seeks to establish models capable of discrete operations.”
A guide to the dataset explosion in QA, NLI, and commonsense reasoning
Rogers, A., & Rumshisky, A. (2020) · doi:10.18653/v1/2020.coling-tutorials.5 · Cited by 2
“Among the recent improvements in the dataset collection methodology are complex queries that require aggregating information from several sources (Dua et al., 2019; Kocisky et al., 2018; Yang et al., 2018).”
BackgroundNumNet: Machine Reading Comprehension with Numerical ReasoningPreprint
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06701 · Cited by 6
“In this part, we investigate the effect of different GNN structures on the DROP development set.”
InfluentialR3: A Reading Comprehension Benchmark Requiring Reasoning ProcessesPreprint
Wang, R., Tao, K., Song, D., Zhang, Z., Ma, X., Su, X. H., & Dai, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01251 · Cited by 4
“To this end, many high-quality and large-scale reading comprehension datasets have been released (Rajpurkar et al., 2016; Yang et al., 2018; Reddy et al., 2019; Dua et al., 2019), which lay a good data foundation for QA systems in different scenarios.”
BackgroundMethodInfluentialQuestion Directed Graph Attention Network for Numerical Reasoning over TextPreprint
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07448 · Cited by 2
Competency Problems: On Finding and Removing Artifacts in Language DataPreprint
Gardner, M., Merrill, W., Dodge, J., Peters, M. E., Ross, A., Singh, S., & Smith, N. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08646 · Cited by 25
“, common patterns in DROP (Dua et al., 2019) or ROPES (Lin et al.”
BackgroundSelf-citationConstrained Language Models Yield Few-Shot Semantic ParsersPreprint
Shin, R., Lin, C. H., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J. M., & Durme, B. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08768 · Cited by 38
“, 2018), reading comprehension (Talmor and Berant, 2018; Yang et al., 2018; Dua et al., 2019; Abujabal et al., 2019), and visual question answering (Johnson et al.”
BackgroundOLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00656 · Cited by 2
Self-citationMiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“On DROP, a reading comprehension benchmark, MiMo-7B-Base outperforms compared models, showing advanced language understanding capability.”
Self-citationMamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“…( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐵 , U ) C = contract ( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐶 , U ) , Intermediate output Y ′ is obtained by the residual function 𝜙 , Y ′ ← 𝜙 ( Y 𝑡 , Z ) , where 𝜙 ( Y 𝑡 , Z ) : = Y ⊙ SiLU ( Z 𝑡 ) in Finally, the layer output O ∈ R is computed via the following down projections:”
DGR: Decomposition Graph Reconstruction for Question Understanding
Han, W., Huang, J., Xie, Q., & Peng, M. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892120 · Cited by 1
“1, this divide-and-conquer design is consistent with the process of humans [2] and can provide a better understanding of complex questions, which is the bottleneck of existing methods in question understanding that implicitly model the understanding process [3]–[5].”
BackgroundUnsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model
Wang, Z., Wang, Y., Zhu, Z., & Caverlee, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.379 · Cited by 1
Self-citationLiu, Z., Huang, P., Xu, Z., Li, X., Liu, S., Peng, C., Xin, H., Yan, Y., Wang, S., Han, X., et al. (2026) · AI Open · doi:10.1016/j.aiopen.2026.02.002 · Cited by 1
Spectral Sparsity: A Unifying Framework for Scalable Model Interpretability Using Codes
Kang, J., Tsui, D., Erginbas, Y. E., Butler, L., Aghazadeh, A., & Ramchandran, K. (2025) · IEEE BITS the Information Theory Magazine · doi:10.1109/mbits.2026.3692444 · Cited by 1
“Similarly, on the discrete reasoning over paragraphs dataset [56], which involves numerical reasoning, SPEX outperforms all other methods at identifying the most in fl uential input features.”
Mala, C. S., Gezi̇ci, G., & Giannotti, F. (2026) · Machine Learning · doi:10.1007/s10994-026-07121-y · Cited by 1
“HaluBench , released in 2024, may appear minimally in the training data of recently released LLMs, i.e. Qwen 3-4B ; however, its examples are derived from earlier public QA datasets (e.g., PubMedQA , DROP, RAGTruth ) and augmented with hallucination annotations and perturbed answers that are absent from the pre-traini…”
InfluentialText-based Q&A: Automated Question Generation and Answering for Enhanced Data Processing
Rhythm, E. R., Hosain, A. H., Raya, N. Z., Pranta, K. A. R., Trina, T. T., Hossain, M. S., Mehedi, M. H. K., & Rasel, A. A. (2023) · Computer Applications & Technological Solutions (CATS) · doi:10.1109/cats58046.2023.10424125 · Cited by 1
“The EM score of 0.18 calculated by DROP [22] in multiple span answers suggests that the proportion of predicted answers in multi spans that precisely correspond to the true spans in the gold standard data is approximately 20%.”
MethodInfluentialAn Algorithm based on Semantic Similarity to Extract Candidate Answers in Question Answering Systems
Ghaderi, S. H., Ghafouri, A., & Naderi, H. (2024) · International Conference on Web Research (ICWR) · doi:10.1109/icwr61162.2024.10533337 · Cited by 2
“In the work of Lee et al.[9], links in Wikipedia articles have been used.”
MethodModeling Logical Content and Pattern Information for Contextual Reasoning
Guo, P., Jian, P., & Shi, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650051 · Cited by 1
“Many existing models has already mastered the early MRC datasets, for instance, SQuAD [3] [4], DROP [5], HotpotQA [6] and so on.”
MethodEnhancing Financial Reasoning in Large Language Models: The Role of Gold Facts
Zhu, S., Yuan, Z., Wang, K., Zhang, Y., & Wei, W. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825021 · Cited by 2
“Numerous QA datasets, such as TabFact [21], DROP [22], and HybridQA [23], emphasize the need for numerical understanding and reasoning based on structured tables or knowledge bases.”
Advances in reasoning by prompting large language models: A survey
Hong, R., Pang, X., & Zhang, C. (2026) · Cybernetics and Intelligence · doi:10.26599/cai.2024.9390004 · Cited by 3
Joint Translation and Unit Conversion for End-to-end Localization
Dinu, G., Mathur, P., Federico, M., Lauly, S., & Al-Onaizan, Y. (2020) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2020.iwslt-1.32 · Cited by 1
“, 2019) introduces a benchmark for evaluating quantitative reasoning in natural language inference and (Dua et al., 2019) one for symbolic operations such as addition or sorting in reading comprehension.”
BackgroundRAIN: A Relation-based Arithmetic model with Implicit Numbers
Bai, Z., Li, K., Chen, J., Yuan, C., & Wang, X. (2020) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc51575.2020.9345205 · Cited by 1
“However, the above models which directly produce the answers, perform poorly on arithmetic question [12].”
BackgroundMethodInfluentialMulti-span Extractive Reading Comprehension Without Multi-span Supervision
Takahashi, T., Taniguchi, M., Taniguchi, T., & Ohkuma, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72240-1_41 · Cited by 1
BiQuAD: Towards QA based on deeper text understanding
Grimm, F., & Cimiano, P. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.10 · Cited by 1
“The recent DROP dataset (Dua et al., 2019) focuses on complex reasoning tasks in form of both, open and closed domain, questions.”
BackgroundA Framework for Evaluating MRC Approaches with Unanswerable Questions
Du, H., Thudumu, S., Singh, S., Barnett, S., Logothetis, I., Vasa, R., & Mouzakis, K. (2022) · IEEE International Conference on eScience · doi:10.1109/escience55777.2022.00075 · Cited by 1
Self-citationLINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity
Arunkumar, A., Sharma, S., Agrawal, R., Chandrasekaran, S., & Bryan, C. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14840 · Cited by 1
UKP-SQuARE v3: A Platform for Multi-Agent QA Research
Puerto, H., Baumgärtner, T., Sachdeva, R. S., Fang, H., Zhang, H., Tariverdian, S., Wang, K., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.55 · Cited by 1
SharPT: Shared Latent Space Prompt Tuning
Pang, B., Yavuz, S., Xiong, C., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.92 · Cited by 1
Movie Plot Analysis via Turning Point IdentificationPreprint
Papalampidi, P., Keller, F., & Lapata, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10328 · Cited by 14
“…Trischler et al., 2016) to large scale datasets that consist of complex stories (Tapaswi et al., 2016; Frermann et al., 2018; Koˇcisk`y et al., 2018; Joshi et al., 2017) or require reasoning across multiple documents (Welbl et al., 2018; Wang et al., 2018; Dua et al., 2019; Yang et al., 2018).”
Fine-tune BERT for E-commerce Non-Default Search RankingPreprint
Jiang, Y., Shang, Y., Shen, H., Yang, W.-Y., & Xiao, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09689 · Cited by 1
“BERT (Bidirectional Encoder Representations from Transformers)[6], as a pre-training language model, achieved the state-of-the-art results on many tasks, such as sentiment analysis[? ] and question answering[7].”
BackgroundForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text DataPreprint
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00792 · Cited by 24
“To answer our questions, one needs numerical reasoning (Dua et al., 2019).”
BackgroundEvaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPPreprint
Chen, A., Gudipati, P., Longpre, S., Ling, X., & Singh, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06830 · Cited by 1
“Substantial progress in NLP has been made on “closed” tasks, where queries are paired with relevant documents (Rajpurkar et al., 2016; Dua et al., 2019).”
BackgroundSelf-citationSpartQA: : A Textual Question Answering Benchmark for Spatial ReasoningPreprint
Mirzaee, R., Faghihi, H. R., Ning, Q., & Kordjmashidi, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05832 · Cited by 29
“Solving Task 17 of bAbI typically does not require sophisticated reasoning, which is an important capability emphasized by more recent works (e.g., Dua et al. (2019); Khashabi et al. (2018); Yang et al. (2018); Dasigi et al. (2019); Ning et al. (2020)).”
BackgroundMultilingual Compositional Wikidata Questions.Preprint
Cui, R., Aralikatte, R., Lent, H., & Hershcovich, D. (2021) · arXiv (Cornell University) · Cited by 1
Zero-shot Task Transfer for Invoice Extraction via Class-aware QA EnsemblePreprint
Damodaran, P., Singh, P., & Achankuju, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06069 · Cited by 2
“Include QA models trained on other datasets like DROP (Dua et al., 2019) and RACE (Lai and Xie, 2017) 4.)”
MethodShi, J., Cao, S., Hou, L., Li, J., & Zhang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07302 · Cited by 25
“However, multi-hop QA, which requires reasoning with the entity relations at multiple steps, is far from resolved (Yang et al., 2018; Dua et al., 2019; Zhang et al., 2017; Talmor and Berant, 2018).”
BackgroundStudies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
MiMo-V2-Flash Technical Report
Core Team, Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.02780 · Cited by 1
Self-citationGated Delta Networks: Improving Mamba2 with Delta Rule
Yang, S., Kautz, J., & Hatamizadeh, A. (2024) · arXiv · doi:10.48550/arxiv.2412.06464 · Cited by 1
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“The suite includes SWDE [46] for structured relation extraction from HTML, FDA [47] for key-value retrieval from PDFs, and question-answering datasets including SQuAD [48], TriviaQA [49], DROP [50], and Natural Questions [51].”
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“To gauge the ability to recall in-context information, we follow Arora et al. [4] and Yang et al. [111] and evaluate models on SWDE [65], SQUAD [82], FDA [3], TQA [55], NQ [58] and DROP [29].”
ATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
Zhou, H., Wan, X., Sun, R., Palangi, H., Iqbal, S., Vulić, I., Korhonen, A., & Arık, S. Ö. (2025) · arXiv · doi:10.48550/arxiv.2502.02533
“Aside from the common benchmarks used for automating MAS (Hu et al., 2024a; Zhang et al., 2024b), we conduct experiments on an extensive collection of tasks: 1) Hendryck’s MATH (Hendrycks et al., 2021) and DROP (Dua et al., 2019) for reasoning; HotpotQA (Yang et al., 2018), MuSiQue (Trivedi et al., 2022), 2WikiMultiHo…”
InfluentialScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
Wang, Y., Yang, L., Li, G., Wang, M., & Aragam, B. (2025) · arXiv · doi:10.48550/arxiv.2502.04306
Self-citationSignal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
“MMLU [23], ARC [11], BoolQ [10], CSQA [59], OBQA [39], PiQA [4], SocialIQA [53], HellaSwag [69], WinoGrande [52], DROP [16], CoQA [48], Jeopardy [61], NaturalQs [28], SQuAD [47], TriviaQA [26], MedMCQA [43], MMLU Pro [64], AGI Eval [70], GPQA [49] Math.”
VeRO: A Harness for Agents to Optimize Agents
Ursekar, V., Shanker, A., Chatrath, V., Xue, Y., & Denton, S. M. (2026) · arXiv · doi:10.48550/arxiv.2602.22480
“ADAS (Hu et al., 2025) represents agents as functions in code and evaluates the ability of agents to improve these functions on several benchmarks, including DROP (Dua et al., 2019) and GSM8K (Cobbe et al., 2021).”
Contribution of reflection in language emergence with an under-restricted situation
Todo, K., & Yamamura, M. (2020) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse49731.2020.9313061
“Introducing the language system using the current datasets has been difficult in machine learning [13], [14].”
BackgroundOverestimation of Syntactic Representationin Neural Language ModelsPreprint
Kodner, J., & Gupta, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05067
“…is already known in relation to these higher-level tasks (e.g. (Poliak et al., 2018; Geva et al., 2019)), and various challenge data sets have been constructed to mitigate it in different ways (Levesque et al., 2011; Chao et al., 2017; Dua et al., 2019; Lin et al., 2019; Dasigi et al., 2019).”
BackgroundForecastQA: Machine Comprehension of Temporal Text for Answering Forecasting QuestionsPreprint
Jin, W., Kim, S., & Ren, X. (2020) · arXiv (Cornell University)
Chinnappa, D., Palmer, A., & Blanco, E. (2020) · Language Resources and Evaluation
Coreference Reasoning in Machine Reading Comprehension
Wu, M., Moosavi, N. S., Roth, D., & Gurevych, I. (2020) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2012.15573
“In the first approach, potentially biased examples are discarded from the dataset, either after the dataset creation (Zellers et al., 2018; Yang et al., 2018a; Le Bras et al., 2020; Bartolo et al., 2020), or while creating the dataset (Dua et al., 2019; Chen et al., 2019; Nie et al., 2020).”
BackgroundMethodInfluentialPrediction or Comparison: Toward Interpretable Qualitative Reasoning
Ren, M., Huang, H., & Gao, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.59
“Since Question Answering (QA) has been significantly developed in recent years, various challenging datasets have been proposed (Lai et al., 2017; Rajpurkar et al., 2018; Yang et al., 2018; Choi et al., 2018; Dua et al., 2019), and one often encounters the context and questions about qualitative relationships.”
BackgroundEviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading ComprehensionPreprint
Zhou, Y., Bao, J., Sun, H., Liang, J., Wu, Y., He, X., Zhou, B., & Zhao, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07994
Accessing Diverse Web Knowledge with Natural Language Interface
chen, W. (2021) · eScholarship (California Digital Library)
What's in a Name? Answer Equivalence For Open-Domain Question AnsweringPreprint
Si, C., Zhao, C., & Boyd-Graber, J. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05289
“However, the modern formulation of QA usually assumes that each question has only one answer, e.g. , SQ u AD (Rajpurkar et al., 2016), H otpot QA (Yang et al., 2018), DROP (Dua et al., 2019).”
BackgroundTruth-Conditional Captioning of Time Series DataPreprint
Jhamtani, H., & Berg-Kirkpatrick, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01839
“Prior work has investigated handling of numeracy in question answering datasets (Dua et al., 2019; Andor et al., 2019; Gupta et al., 2020), typically using a predefined set of executable operations or using specific distributions for number prediction (Spokoyny and Berg-Kirkpatrick, 2020; Thawani et al.”
Open-Domain Textual Question Answering Systems
Huang, Z., Su, X., Xu, S., & Hu, M. (2021) · Springer eBooks · doi:10.1007/978-3-030-70716-3_4
Investigating Numeracy Learning Ability of a Text-to-Text Transfer ModelPreprint
Pal, K. K., & Baral, C. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04672 · Cited by 2
“Specially Designed Models: NALU (Trask et al., 2018), NAU and NMU (Madsen and Johansen, 2020), numBERT (Zhang et al., 2020), GenBERT (Geva et al., 2020), NT5 (Yang et al., 2021) have emerged in the last few years to incorporate arithmetic skills into models through specially designed architecture or fine-tuning tasks…”
BackgroundICDAR 2021 Competition on Document VisualQuestion AnsweringPreprint
Tito, R., Mathew, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05547
“On the other hand, Reading Comprehension [25,12,7] and Open-domain Question Answering [22,16] on machine readable text is a popular research topic in Natural Language Processing (NLP) and Information Retrieval (IR).”
BackgroundMuppet: Massive Multi-task Representations with Pre-FinetuningPreprint
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11038 · Cited by 65
Diaz, D. Y. R., Anioł, M., & Culnan, J. (2022) · doi:10.18653/v1/2022.dadc-1.6
Han, S., Gao, N., Guo, X., & Shan, Y. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891947
“1) Baselines: For comparison, We choose several models as our baselines: • NAQANet [11], a model that enhances QANet with an answer type prediction component and three answer prediction components for numeric reasoning; • ALBERT-Calculator [19], a model that designs a pre-defined set of executable…”
BackgroundMethodInfluentialMachine Learning Based Machine Reading Comprehension - A Survey
Singh, M., & Yadav, V. (2022) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca54612.2022.9985764
“0 [22] and NewsQA, two benchmark MRC challenge datasets, are used to evaluate the proposed reader.”
MethodAlexandrov, D., Zakharova, A., & Butakov, N. A. (2023) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc56153.2023.00012
“Addressing the lack of complex reasoning, QUOREF [14] and DROP [15] datasets were developed.”
MethodЗАГАЛЬНОДОСТУПНІ НАБОРИ ДАНИХ ТА МЕТРИКИ ДЛЯ СПРИЯННЯ ДОСЛІДЖЕННЯМ СИСТЕМ НАДАННЯ ВІДПОВІДЕЙ
Vyshniak, M. Y., & Pyrozhenko, M. Y. (2023) · Таврійський науковий вісник Серія Технічні науки · doi:10.32851/tnv-tech.2023.1.2
Pre-reading Activity over Question for Machine Reading Comprehension
Yuan, C., Liu, K., & Zhang, X. (2022) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai56018.2022.00213
SADE: A Self-Adaptive Expert for Multi-Dataset Question Answering
Peng, Y.-X., Wang, Q., Mao, Z.-D., & Zhang, Y. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp49357.2023.10097208 · Cited by 2
Predicting Numerals in Text Using Nearest Neighbor Language Models
Sakamoto, T., & Aizawa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.295
“In this study, we used the following four datasets with different domains and passage lengths: • Numeracy-600K (article titles) (Chen et al., 2019), • ACLsent (scientific papers) (Abekawa and Aizawa, 2016), • FinNum (financial tweets) (Chen et al., 2018), • DROP (Wikipedia) (Dua et al., 2019).”
MethodBeinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_4
Few-shot Unified Question Answering: Tuning Models or Prompts?
Bansal, S., Yavuz, S., Pang, B., Bhat, M., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.550 · Cited by 3
When Do Decompositions Help for Machine Reading?
Wei, K., Lawrie, D., Durme, B. V., Chen, Y., & Weller, O. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.219 · Cited by 2
Text-based Q&A: Automated Question Generation and Answering for Enhanced Data Processing
(2023) · Computer Applications & Technological Solutions (CATS) · doi:10.1109/cats58046.2023.10424347
Atanasova, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-51518-7_1
LogSay: An Efficient Comprehension System for Log Numerical Reasoning
Qi, J., Luan, Z., Huang, S., Fung, C., & Yang, H. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3386068 · Cited by 1
“…[12] Video ♣ Medium LifeQA [13] Video ♣ Medium VQuAD [7] Video ♣ Hard SQuAD [1] Wikipedia ♣ Simple HotpotQA [14] Wikipedia ♠♣ Medium Drop [2] Wikipedia ♣ Medium 2WikiMultiHopQA [15] Wikipedia ♠♣ Hard NewsQA [16] News ♣ Simple JEC-QA [8] Legal ♣ Hard Event-QA [9] Knowledge Graphs ♠♣…”
BackgroundMethodInfluentialNumerical reasoning reading comprehension on Vietnamese COVID-19 news: task, corpus, and challenges
Nguyen, K. V., Le, T. V., & Pham-Phuc, T. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09744-5
“Large-scale reading comprehension corpora such as SQuAD [3, 5, 10 – 13] have recently accelerated the development of MRC models [8, 9, 14 – 17].”
Influential
Metadata
- Authors
- Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 2368-2378
- OpenAlex
- W2919420119
- Cited by
- 393
- References
- 53
- Citations by year
- 2026: 20 · 2025: 8 · 2024: 20 · 2023: 60 · 2022: 62 · 2021: 113 · 2020: 78 · 2019: 32
- FWCI
- 27.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- reading comprehension, benchmark, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
