The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Open accessAbstract
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“our agents improve F1 scores on reading comprehension tasks in DROP (Dua et al., 2019) by 13.6 /100”
From Automated Design of Agentic Systems (2024) · cites this paper as Dua et al., 2019
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“For example, BERT, the current state-of-the-art on SQuAD, drops by more than 50 absolute F1 points.”
MethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“We also add context-specific rules to produce strings occurring in both question and paragraph, and those paragraph strings that are neighbors of question tokens in the GloVe embedding space (Pennington et al., 2014), up to a cosine distance of d .”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“We test four different SQuAD-style reading comprehension models on DROP: (1 (Peters et al., 2018) to the original embeddings (78.7% EM); (4) BERT (Devlin et al., 2018), which recently achieved improvements on many NLP tasks with a novel pre-training technique (84.7% EM).”
MethodInfluentialSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Question answering datasets: With systems reaching human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), many follow-on tasks are currently being proposed.”
MethodA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“…et al., 2018; Kaushik and Lipton, 2018) has identified that popular synthetic NLP datasets (such as SQuAD (Ra-jpurkar et al., 2016) or SNLI (Bowman et al., 2015)) are prone to have artifacts and annotation biases which can be exploited by supervised algorithms that learn to pick up these…”
BackgroundInfluentialKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 2×
“, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 2×
“…challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018), integrating knowledge from…”
BackgroundBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 3×
“…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”
MethodInfluentialTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2×
“…to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2018; Choi et al., 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018), mis-matched passages and questions (Saha et al., 2018; Kocisk´y et al., 2018; Rajpurkar et al., 2018),…”
BackgroundCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602 · cited 1×
“, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”
Background
Cited by
Sorted by citations
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
“Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
“Of the datasets where model tuning is better, we see that DROP shares a domain (Wikipedia) with SQuAD and is thus one of the smallest domain transfers.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“…Common Sense Reasoning: PIQA (Bisk et al., 2019), ARC (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018) • In-context Reading Comprehension: DROP (Dua et al., 2019), CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACE (Lai et al., 2017) •…”
BackgroundFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“0 -20 20 40 60 Percent of Data Clean in Dataset Percent change in performance of FLAN (accuracy, F1, or BLEU) eval on only clean data did better eval on all data (including dirty) did better ANLI R2 SQuAD v2 ANLI R1 DROP ReCoRD PIQA 25 500 75 100 Figure 9: Like GPT-3, we also measured performance on cleaned versions o…”
BackgroundInfluentialClimbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
“For example, the DROP reading comprehension benchmark (Dua et al., 2019) seeks to create more stringent tests of understanding by creating questions that require the system to integrate information from different parts of a paragraph via simple arithmetic or similar operations.”
BackgroundHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090
“Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”
BackgroundDeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025) · Nature · doi:10.1038/s41586-025-09422-z · Cited by 1,701
Self-citationMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationTaPas: Weakly Supervised Table Parsing via Pre-training
Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.398 · Cited by 721
“To train the model given this form of supervision one could search offline (Dua et al., 2019; Andor et al., 2019) or online (Berant et al.”
MethodUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514
“The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
BackgroundMethodInfluentialClark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 477
“, 2018), and DROP (Dua et al., 2019).”
MethodGeva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284
“It has been used for various NLU tasks, including Question Answering (Rajpurkar et al., 2018; Mihaylov et al., 2018; Dua et al., 2019), commonsense and visual reasoning (Talmor et al., 2019; Zellers et al., 2019; Suhr et al., 2018), and Natural Language Inference (NLI) (Williams et al., 2018).”
Method♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283
“Our models are Transformer-based (Vaswani et al., 2017) language models (Devlin et al., 2019), implemented using PyTorch (Paszke et al., 2019), HuggingFace Transformers (Wolf et al., 2019) and AllenNLP (Gardner et al., 2017).”
MethodByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00461 · Cited by 330
“Finally, DROP (Dua et al., 2019) is a challenging reading comprehension task that requires numerical reasoning.”
BackgroundInfluentialCross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269
“, 2019), DROP (Dua et al., 2019), EssentialTerms (Khashabi et al.”
MethodEvaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 323
“A similar analysis on DROP shows that MTMSN does substantially worse on event re-ordering (47.3 F 1 ) than on adding compositional reasoning steps (67.5 F 1 ).”
BackgroundMethodInfluentialSelf-citationFinQA: A Dataset of Numerical Reasoning over Financial Data
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258
“As most existing QA datasets were constructed by MTurk workers (Yang et al., 2018; Dua et al., 2019; Chen et al., 2020c), it requires substantial domain-specific knowledge to compose meaningful questions that are hard for computers to answer.”
BackgroundMethodInfluentialDynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
“…QA, and are looking to expand this to include unanswerable questions (Rajpurkar et al., 2018), longer and more complex passages, Yes/No questions and multi-span answers (Kwiatkowski et al., 2019), and numbers, dates and spans from the question (Dua et al., 2019) as model performance progresses.”
BackgroundToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252
“…ability for reasoning and numerical inference (Wiseman et al., 2017) and for building representations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015; Chen et al., 2019; Dua et al., 2019).”
BackgroundVILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P. A., Shoeybi, M., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02520 · Cited by 448
“Therefore, we further evaluate the text-only performance of the models under three benchmarks: MMLU [26], BBH [53], and DROP [21] in Table 6.”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242
“• DROP (Dua et al., 2019): A large dataset for numerical reasoning over paragraphs.”
BackgroundMethodInfluentialDo NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227
Self-citationMRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 254
“DROP (Dua et al., 2019) DROP (Discrete Reasoning Over the content of Paragraphs) examples were collected similarly to SQuAD, where crowd-workers are asked to create question-answer pairs from Wikipedia paragraphs.”
MethodInfluentialHybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., & Wang, W. Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.91 · Cited by 290
“2) Text-only datasets with single passage: like SQuAD (Rajpurkar et al., 2016), DROP (Dua et al., 2019).”
BackgroundMethodMathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., & Jawahar, C. V. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00264 · Cited by 326
“DROP [13] dataset for reading comprehension defines answer types such as question span and passage span and categorizes questions by the kind of discrete operations arithmetic or logical operations required to find the answer.”
BackgroundInfluentialMuppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Vu, T., Lester, B., Constant, N., Al‐Rfou, R., & Cer, D. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.346 · Cited by 267
“, 2021), DROP (Dua et al., 2019), WINOGRANDE (Sakaguchi et al.”
MethodInfluentialRecent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 289
“The Discrete Reasoning Over Paragraphs (DROP) benchmark by Dua, Wang, Dasigi, Stanovsky, Singh, and Gardner (2019) consists of nearly 100,000 questions over about 7,000 passages from Wikipedia.”
BackgroundInfluentialMetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 197
“Such models perform well on existing datasets, such as DROP (Dua et al., 2019), but do not generalize to unsupported computations, which will require modifying the model architecture.”
BackgroundMethodQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 213
“In an RC setting, this corresponds to the so-called multi-hop datasets that necessitate the combination of information across sentences [145], paragraphs [83], and documents [314].”
BackgroundMethodInfluentialSelf-citationLarge language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161
Self-citationCan You Unpack That? Learning to Rewrite Questions-in-Context
Elgohary, A., Peskov, D., & Boyd-Graber, J. L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1605 · Cited by 181
“…vary in difficulty (Sugawara et al., 2018), existing research either introduces new benchmarks of difficult (adversarial) stand-alone questions (Dua et al., 2019; Wallace et al., 2019, inter alia), or models that simplify hard questions through paraphrasing (Dong et al., 2017) or decomposition…”
BackgroundMultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 172
“The best generalization to DROP is from H OTPOT QA, since both require multi-hop reasoning.”
InfluentialQuoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., & Gardner, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1606 · Cited by 173
“We use the same implementation of EM as SQuAD, and we employ the F 1 metric used for DROP (Dua et al., 2019 (3) BERT QA (Devlin et al., 2019), the adversarial baseline used in data construction, and (4) XLNet QA (Yang et al., 2019), another large pretrained language model based on the Trans-former architecture (Vaswan…”
BackgroundMethodInfluentialSelf-citationGoing Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., & Pałka, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_47 · Cited by 174
Retrieving and Reading: A Comprehensive Survey on Open-domain Question AnsweringPreprint
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 343
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningPreprint
Yu, W., Jiang, Z., Dong, Y., & Feng, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04326 · Cited by 245
“…et al., 2018), multi-hop reasoning across multiple sentences (Khashabi et al., 2018) and multiple documents (Welbl et al., 2018), commonsense knowledge reasoning (Mihaylov et al., 2018; Zhang et al., 2018; Huang et al., 2019) and numerical discrete reasoning over paragraphs (Dua et al., 2019).”
BackgroundA Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 144
“We experiment on a subset of DROP (Dua et al., 2019) with numeric answers (67% of the entire dataset) focusing on mathematical reasoning.”
BackgroundMethodInfluentialA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
“Some examples include SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), NarrativeQA (Koˇciský et al., 2018), WikiHop (Welbl et al., 2018), HotpotQA (Yang et al., 2018), CoQA (Reddy et al., 2019), DROP (Dua et al., 2019), Q UOREF (Dasigi et al., 2019).”
BackgroundSelf-citationConstrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139
“, 2018), reading comprehension (Talmor and Berant, 2018; Yang et al., 2018; Dua et al., 2019; Abujabal et al., 2019), and visual question answering (Johnson et al.”
BackgroundLarge Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134
“In addition to GSM-IC, we extend our evaluation to the DROP dataset (Dua et al., 2019), where the task is to answer a question according to a long passage that naturally contains irrelevant context.”
BackgroundMethodTrick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Wallace, E., Rodríguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. L. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00279 · Cited by 139
“For instance, Dua et al. (2019) and Zhang et al. (2018) filter out easy questions using a baseline QA model, while Zellers et al. (2018) use stylistic classifiers to filter language inference examples.”
MethodTowards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 196
“Their model is capable of answering questions involving discrete reasoning in the DROP dataset (Dua et al. 2019), for example, Who kicked the longest field goal in the second quarter?, given a long description of the match.”
BackgroundReframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140
“For example, they are studied in the context of classification (Petroni et al., 2019; Schick and Schütze, 2021) or applied as prompt composition and prompt decomposition for relation extraction (Han et al., 2021) and named entity recognition (Cui et al., 2021) tasks, respectively.”
BackgroundEvaluating Open-Domain Question Answering in the Era of Large Language Models
Kamalloo, E., Dziri, N., Clarke, C. L. A., & Rafiei, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.307 · Cited by 232
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 142
Shi, J., Cao, S., Hou, L., Li, J., & Zhang, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.341 · Cited by 123
“However, multi-hop QA, which requires reasoning with the entity relations at multiple steps, is far from resolved (Yang et al., 2018; Dua et al., 2019; Zhang et al., 2017; Talmor and Berant, 2018).”
BackgroundAllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations · doi:10.18653/v1/d19-3002 · Cited by 116
“We use NAQANet (Dua et al., 2019) and BiDAF models (Seo et al., 2017).”
MethodSelf-citationMuTual: A Dataset for Multi-Turn Dialogue Reasoning
Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.130 · Cited by 110
“DROP (Dua et al., 2019) and COSMOS (Huang et al., 2019) focus on factual understanding and commonsense comprehension, respectively.”
BackgroundTyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesPreprint
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05002 · Cited by 109
“, 2018), and DROP (Dua et al., 2019).”
MethodDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109
Self-citationNumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108
“In this part, we investigate the effect of different GNN structures on the DROP development set.”
InfluentialB reak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 150
“, 2019; HudsonandManning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al.”
BackgroundInfluentialSelf-citationBeat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102
Exploring and Predicting Transferability across NLP Tasks
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., & Iyyer, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.635 · Cited by 146
“Task SQuAD-1 SQuAD-2 NewsQA HotpotQA BoolQ DROP WikiHop DuoRC-p DuoRC-s CQ ComQA Baseline 26.8 ± 6.0 50.1 ± 0.1 28.8 ± 4.9 23.3 ± 4.0 62.2 ± 0.0 19.4 ± 11.8 25.5 ± 4.7 41.6 ± 1.1 38.9 ± 3.3 30.7 ± 2.5 39.1 ± 1.2 CoLA 27.3 ± 6.0 50.0 ± 0.2 29.3 ± 4.6 23.8 ± 4.2 62.2 ± 0.0 17.0 ± 10.4 25.0 ± 4.5 41.8 ± 1.1 40.1 ± 3.1 31…”
MethodInfluentialDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationEvaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105
“Yang et al. (2018) study the effect of using BLEU and ROUGE in evaluating QA, focusing on yes-no and entity questions on the Chinese DuReader dataset (He et al., 2017).”
MethodSelf-citationReasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P. E., & Gardner, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5808 · Cited by 110
“Reading comprehension There are many reading comprehension datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Kwiatkowski et al., 2019; Dua et al., 2019), the majority of which principally require understanding local predicate-argument structure in a passage of text.”
BackgroundSelf-citationImpact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
“…tasks reflects robust reasoning capabilities, especially since numerical reasoning is an essential skill needed to perform other complex reasoning tasks such as question answering through reading comprehension (Dua et al., 2019), and commonsense reasoning (Thawani et al., 2021; Lin et al., 2020a).”
BackgroundSelf-citationZhao, T., Yan, Z., Cao, Y., & Li, Z. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/546 · Cited by 78
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78
“On the DROP Question Answering benchmark, BERT performs five times worse when the answer is a number instead of a span of text (Dua et al., 2019).”
BackgroundA Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1170 · Cited by 78
“Following Dua et al. (2019), we design a multi-type answer predictor to selectively produce different kinds of answers such as span, count number, and arithmetic expression.”
BackgroundMethodInfluentialA Simple and Effective Model for Answering Multi-span Questions
Segal, E., Efrat, A., Shoham, M., Globerson, A., & Berant, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.248 · Cited by 76
Few-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 95
“…datasets in various domains (Trischler et al., 2017; Kembhavi et al., 2017) and languages (Lewis et al., 2020; Clark et al., 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or multiple answer spans (Dua et al., 2019; Dasigi et al., 2019).”
BackgroundConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering
Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., & Wang, W. Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.421 · Cited by 198
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
Andor, D., He, L., Lee, K., & Pitler, E. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1609 · Cited by 72
“The NAQANet model2(Dua et al., 2019) predicts a negative number of people, whereas our model predicts that an operation Diff should be taken and identifies the two arguments.”
BackgroundMethodInfluentialWhat to Pre-Train on? Efficient Intermediate Task Selection
Poth, C., Pfeiffer, J., Rücklé, A., & Gurevych, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.827 · Cited by 92
“CQ (Bao et al., 2016) QA web snippets EM/ F1 DROP (Dua et al., 2019)”
BackgroundCompetency Problems: On Finding and Removing Artifacts in Language Data
Gardner, M., Merrill, W., Dodge, J., Peters, M., Ross, A., Singh, S., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.135 · Cited by 81
“, common patterns in DROP (Dua et al., 2019) or ROPES (Lin et al.”
BackgroundSelf-citationGeneralizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5827 · Cited by 76
“…Trivi-aQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), NaturalQuestions (Kwiatkowski et al., 2019), DROP (Dua et al., 2019), RACE (Lai et al., 2017), DueRC (Saha et al., 2018), BioASQ (Tsat-saronis et al., 2012), TextbookQA (Kembhavi et al., 2017), and…”
BackgroundDid Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning StrategiesPreprint
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02235 · Cited by 70
“• DROP (Dua et al., 2019): A large dataset for numerical reasoning over paragraphs.”
BackgroundMethodInfluentialGraph Neural Networks for Natural Language Processing: A SurveyPreprint
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06090 · Cited by 70
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69
“…systems, including knowledge graphs (Mihaylov and Frank, 2018; Bauer et al., 2018; Kundu et al., 2019; Wang et al., 2019; Lin et al., 2019; Ding et al., 2019; Feng et al., 2020; Wang et al., 2022b), or question taxonomies (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019b; Wang et al., 2022a).”
BackgroundMethodImproving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 94
“Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
BackgroundMethodA Vietnamese Dataset for Evaluating Machine Reading Comprehension
Nguyen, K. V., Nguyen, D.-V., Nguyen, T. H. A., & Nguyen, N. L.-T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.233 · Cited by 77
“• QANet : QANet was proposed by Yu et al. (2018) and this model also demonstrated good performance on multiple MRC datasets (Rajpurkar et al., 2016; Dua et al., 2019).”
MethodChen, X., Chen, L., Yu, A. W., Zhou, D., Song, D. X., & Le, Q. V. (2020) · International Conference on Learning Representations · Cited by 64
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 69
“For example, multi-hop QA tasks, which include multiple evidence inference across documents [104], [107], [131], symbolic reasoning like numeric calculation, count and sort [132], and extraction-based generation [33], [34].”
BackgroundPerturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69
Multi-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 97
“Multi-hop question answering datasets explicitly require aggregating clues from different parts of some given documents (Dua et al., 2019; Welbl et al., 2018; Yang et al., 2018; Jansen et al., 2018; Khashabi et al., 2018).”
BackgroundTo Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 68
“Other forms of reasoning tested include coreference resolution (Quoref, Dasigi et al., 2019; Winograd Schema Challange, Levesque et al., 2012), numerical reasoning (DROP, Dua et al., 2019), and commonsense reasoning (Cosmos QA, Huang et al., 2019).”
BackgroundLet’s Ask Again: Refine Network for Automatic Question Generation
Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1326 · Cited by 60
“Our proposed RefNet model consisting of a Preliminary Decoder and a Refinement Decoder with Dual Attention Network outperforms the existing state-of-the-art models on the SQuAD, HOTPOT-QA and DROP datasets.”
InfluentialDynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56
“However, there are existing adversarial datasets that are large enough to support full-scale training efforts (Zellers et al., 2018, 2019; Chen et al., 2019; Dua et al., 2019; Bartolo et al., 2020).”
BackgroundMatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altün, Y., Collier, N., & Eisenschlos, J. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.714 · Cited by 79
DAGN: Discourse-Aware Graph Network for Logical Reasoning
Huang, Y., Fang, M., Cao, Y. H., Wang, L., & Liang, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.467 · Cited by 63
“DROP (Dua et al., 2019) and MA-TACO (Zhou et al., 2019) need the models to have numerical reasoning.”
BackgroundArabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 64
Look at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 80
“Various question answering datasets have been introduced with diverse challenges including reasoning over multiple sentences (Joshi et al., 2017), answering multi-hop questions (Yang et al., 2018), and more (Trischler et al., 2017; Welbl et al., 2018; Kwiatkowski et al., 2019; Dua et al., 2019).”
BackgroundSPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 81
“Data Source Has training data SQuAD (Rajpurkar et al., 2016) Wikipedia News (Trischler et al., 2016) News Trivia (Joshi et al., 2017) Web NQ (Kwiatkowski et al., 2019) Google Search Hotpot (Yang et al., 2018) Wikipedia Has no training data BioASQ (Tsatsaronis et al., 2015) PubMed Documents DROP (Dua et al., 2019) Wiki…”
BackgroundQuestion Directed Graph Attention Network for Numerical Reasoning over Text
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.549 · Cited by 56
Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering
Yadav, V., Bethard, S. J., & Surdeanu, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.414 · Cited by 57
“Several multi-hop QA datasets have been proposed recently (Yang et al., 2018; Khashabi et al., 2018a; Welbl et al., 2018; Dua et al., 2019; Chen and Durrett, 2019; Khot et al., 2019a; Sun et al., 2019b; Jansen and Ustalov, 2019; Rajpurkar et al., 2018).”
BackgroundMovie Plot Analysis via Turning Point Identification
Papalampidi, P., Keller, F., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1180 · Cited by 54
“…Trischler et al., 2016) to large scale datasets that consist of complex stories (Tapaswi et al., 2016; Frermann et al., 2018; Koˇcisk`y et al., 2018; Joshi et al., 2017) or require reasoning across multiple documents (Welbl et al., 2018; Wang et al., 2018; Dua et al., 2019; Yang et al., 2018).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 56
“, 2020) and numerical reasoning (Dua et al., 2019), would be to decompose them into already solved simpler problems, e.”
BackgroundMethodInfluentialGENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation.Preprint
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021) · arXiv (Cornell University) · Cited by 47
Self-citationSuccessive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 106
Self-citationIIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 61
“It then concatenates the encoded representations of the first and last tokens of each link as input to a scoring function, following the span classification procedure used by Joshi et al. (2013), selecting any links that score above a threshold g.”
MethodSelf-citationDon’t Blame the Annotator: Bias Already Starts in the Annotation Instructions
Parmar, M., Mishra, S. R., Geva, M., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.130 · Cited by 59
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 66
“Solving Task 17 of bAbI typically does not require sophisticated reasoning, which is an important capability emphasized by more recent works (e.g., Dua et al. (2019); Khashabi et al. (2018); Yang et al. (2018); Dasigi et al. (2019); Ning et al. (2020)).”
BackgroundEvaluating Models' Local Decision Boundaries via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 132
“…analysis (Maas et al., 2011) • MATRES Temporal RE (Ning et al., 2018) • English UD parsing (Nivre et al., 2016) • PERSPECTRUM (Chen et al., 2019) • DROP (Dua et al., 2019) • Quoref (Dasigi et al., 2019) • ROPES (Lin et al., 2019) • BoolQ (Clark et al., 2019) • MC-TACO (Zhou et al., 2019) We…”
BackgroundMethodInfluentialSelf-citationPrompt Tuning for Discriminative Pre-trained Language Models
Yao, Y., Dong, B., Zhang, A., Zhang, Z., Xie, R., Liu, Z., Lin, L., Sun, M., & Wang, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.273 · Cited by 44
Metadata
- Authors
- Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 2368-2378
- OpenAlex
- W2919420119
- Cited by
- 393
- References
- 53
- Citations by year
- 2026: 20 · 2025: 8 · 2024: 20 · 2023: 60 · 2022: 62 · 2021: 113 · 2020: 78 · 2019: 32
- FWCI
- 27.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Keywords
- reading comprehension, benchmark, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
